Language Modeling by Language Models
Authors: Junyan Cheng
Peter Clark
Kyle Richardson
Venue: NeurIPS 2025 spotlight ()
Keywords: Autonomous Scientific Discovery
Large Language Model Agents
Genetic Programming
Primary area: applications

Abstract: *Can we leverage LLMs to model the process of discovering novel language model (LM) architectures?* Inspired by real research, we propose a multi-agent LLM approach that simulates the conventional stages of research, from ideation and literature search (proposal stage) to design implementation (code generation), generative pre-training, and downstream evaluation (verification). Using ideas from scaling laws, our system *Genesys* employs a *Ladder of Scales* approach; new designs are proposed, adversarially reviewed, implemented, and selectively verified at increasingly larger model scales (14M$\sim$350M parameters) with a narrowing budget (the number of models we can train at each scale). To help make discovery efficient and factorizable, Genesys uses a novel genetic programming backbone, which we show has empirical advantages over commonly used direct prompt generation workflows (e.g., $\sim$86\% percentage point improvement in successful design generation, a key bottleneck). We report experiments involving 1,162 newly discovered designs (1,062 fully verified) and find the best designs to be competitive with known architectures (e.g., outperform GPT2, Mamba2, etc., on 6/9 common benchmarks).  We couple these results with comprehensive system-level ablations and formal results, which give broader insights into the design of effective autonomous discovery systems.

Language Modeling by Language Models
Junyan Cheng" Peter Clark Kyle Richardson'
Allen Institute for A Dartmouth College"
j jc. th@dartmouth. edu kyler@allenai. org
+ httpss/ggitbb.oomm/lllll//ggggss
Abstract
Can we leverage LLMs to model the process of discovering novel language model architectures? Inspired by real research, we propose multi-agent LLM approach that simulates the conventional stages of research, from ideation and literature search (proposal stage) to design implementation (code generation), generative pre-training, and downstream evaluation (verification). Using ideas from scaling laws, our system Genesys employs a Ladder of Scales approach; new designs are proposed, adversarially reviewed, implemented, and selectively verified at increasingly larger model scales (14M~350M parameters) with a narrowing budget (the number of models efficient and factorizable, Genesys uses a novel genetic programming backbone, which we show has empirical advantages over commonly used direct prompt generation workflows (e.g., ~866 percentage point improvement in successful design generation, key bottleneck). We report experiments involving 1,162 newly discovered designs (1,062 fully verified through pre-training) and find the best designs to be highly competitive with known architectures (e.g., outperform GPT2, Mamba2, etc., on 6/9 common benchmarks). We couple these results with comprehensive system-level ablations and formal results, which give broader insights into the design of effective autonomous LLM-driven discovery systems.
train at each scale). To help make discovery
Introduction
Automated scientific discovery (ASD) (Langley 1987 Wang et al. 2023). which aims to simulate all aspects of the conventional research process from idcation/system design to experiment execution, has the promise of changing the way that research is performed by making it more accessible, efficient, and less error-prone. However, while many large language model (LrvM)-driven ASD systems have been recently proposed, including Al Scientist (Lu et al. 2024a| Yamada et al. 2025) and others (Liu et al.] 2024b. Jansen al., 2025b; Schmidgall et al. 2025b), much of this work focuses open-ended research with unclear goals and where discoveries are hard to verify. This motivates the development of new tasks that address foundational challenges in ASD, tasks that are broad in scope and address impactful research problems, but that have clear goals and criteria for success.
In this paper, focus discovery in machine learning and ask: Can we model the process of dis- covering novel language model architectures that improve on the standard transformer architecture? While transformers (Vaswani et al. 2017) remain the de facto standard architecture for language
models, research into alternative architectures (Dao & Gu 2024 Sun et al. 2024 2023 Peng et al.) 2024) and transformer variants (Tay et al. 2022b) remains active and important
research with connections to the mature field of neural architecture search (NAS) (Elsken et al. 2019). In contrast to open-ended research tasks, architecture research involves clear goal (i.e., producing executable architecture design) and offers many metrics for evaluating success. It also introduces
Work done during an internship at the Allen Institute for Al.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

LM Architecture Discovery Env.
Genesys
Mayne def pwtpy KE, VO3:
Knowledge Engine
Referencs Library
mesporoct
ieni
desiign
"Run distributed discovery Toup"
budget:
agync dest destgn(evoTrree
Designs
asyne veri perify (Ex
a
iv
SeTect
Rapprrs Wir Dode
Quariec
def deviggn KE]: "*"selec parent designs to Improve, produce new deston and add tree"" parents, seTecttevotres) design best (parrrrttttttttttaiaa. asyee Evotree, (ddeeee
Piper Vector 88
Wie Search Agent
Exclutioaay Tree
mathpix
perplexity
Verification Engine
Autorated Trairmer
Reatline Leaderboard
SmolLM
Hugging Face
sayne def ty (EvoTree, vI]:
Weights & Blases
Scate
Se lect
"sasalet a desten and scale to verify, expertments produce report" design, scale Select(evotree) report Veriffer(destge scalle,E)
Design & Boale
Automated Exiluatorr
LM-Eval BLMPP
GLUE
Verifier fier
verify
asyon Evotree, www.teeeeeiig..reeeeoeo. report)
Figure 1: Can we discover novel language model architectures? A high-level illustration of our approach, consisting of discovery environment (Left), LMADE, that provides knowledge access (Knowledge Engine) and automated evaluation (Verification Engine). Right: Genesys, a LLM-driven agent system that proposes, implements, then verifies new designs using design and verifier agents (see algorithmic workflow, far right) and feedback from LMADE.
new challenges for ASD, including requiring deep literature understanding, careful management of resources (e.g., pretraining compute), and the need to write code in an unbounded design space.
Our approach is shown in Figure and factors into discovery environment that provides the foundational tools for ASD and a discovery system that produces discovery artifacts using feedback from the environment. Our Language Model Architecture Discovery Environment (LMADE) specifically consists of two core resources, a general-purposs knowledge engine that provides access to the academic literature and a verification engine that provides tools for performing model pre- training and evaluation. Our system Genesys then consists of LLM-driven designer agents that propose new research ideas and produce executable architecture designs, and verifier agents that select designs and perform on-the-fly generative pre-training. At the core of Genesys is an evolution tree that stores seed designs and new discovery artifacts. These artifacts are implemented using a special code construct called a generalized autoregressive block (GAB) (Figure B) that is capable of expressing wide range of neural architecture types and factorizable into discrete tree representations that allow us to employ efficient genetic programming (GP)-style optimization.
We performed large-scale discovery experiments that resulted in 1,062 new architecture designs fully verified through pre-training (at the 14M-350M parameter scales). To make verification feasible, we employ a Sflder-of-Scaleee approach where new designs are verified on increasingly larger model scales with a controlled budget, closely following the methodology used in research on small LMS (Lu et al. 2024b Hu al. (024). To our knowledge, our work constitutes the largest ASD experiment of its kind, involving > billion tokens, 2.76M lines of code, and 86K agent interactions P We find that our system produces highly competitive designs, e.g., ones that outperform comparable
transformer and mamba2 models (Dao & Gu. 2024) in 6 9 common downstream tasks. These results are significant and show the feasibility of L.LM-driven discovery for competitive MI. research. Through systematic ablations, we also find that our system leads to more stable discovery (e.g.. measurable improvements the fitnss of new designs over time) and effective code generation (e.g.. ~866 percentage point improvement in successful design generation), which give broader insight into how to effectively build large-scale discovery systems.
Related work
AI in Scientific Discovery Al approaches to ASD have recently proliferated, notably in biomedical science (Jumper et al., 2021 Cheng et al.) 2023 Wong et al.) 2024). material science (Park et al.
2 All code and discovery artifacts (e.g.. new designs, agent interactions and dialogues) can be found at https://genessy..lleen (live console) and https://githubbcee wwt.e///aaameaaaaaeeeeeeee aenennsiigggggsa/eeeea (system code).

---

Tensor Cysstrryppeaatch_Saee seg_le,, din] Dict[str, Anyl
Repeat
GPT(GABBase):
Embedding
root GAU the GAU Tree of CPTBLock
LM Head
class GaBBase (odMeee)):
[sel, x3, #e1f. 4)
2]:
def [self, x: X, <: 2) +> (X, 2):
Block
Actulll GAB implerentation""
x2, 2 MHAXL, 2]
raise NotImplenentedEron
x2
x3, 201f.AMSOOMEX, self. GatedHLL
def Forward(self, X: X, 2: 2) -> (6, 2):
X_, x_ forwardz,,, 2) assert k.shape -n shage 2.updato(2_)
Fractal Factorization
return X,
GAU Tree GPTBlock
class rBa(GAUBase):
return X.,
def forlf x: X, z: 2]:
GPT
class GPTBLock(GBBBSE):
def init (self, ...):
"toput
RoPE
self.root GPT(...)
GatedMLP
NHA
2 corpput 4
2[ "output_ R
_forward(self, x: X, 2: Z): return self.root(k, 2)
RMSNOME
RoPE
3
return
6
What are we trying to discover? 1 visualizes standard autoregressive LMs and the blocks that our system aims to discover (implemented via the Pytorch modules in 2 and 4 with function type (x, + (x,2) x. ② shows an implemented block for the GPT and its factorization into a tree 3 that shows the units in that block (e.g., multi-head attention implemented in )))
Figure 3:
2024: Merchant et al. 2023). and other areas (Chen al. 2024 Nearing et al. 2024). As discussed above, recent attempts at fully end-to-end research via L-M--riven systems, such as AI Scientist (Lu et al., 2024a) Yamada al. (025). AgentLab (Schmidgall et al. 2025a), CodeScientist (Jansen iet al., (225a)), and AIGS (Liu et al. 20224)) have focused on open-ended research tasks with unclear goals and evaluation protocols. In contrast, we focus on the challenging task of neural architecture discovery, which offers a clear objective yet involves many new challenges for ASD.
Language Model Architectures Our work relates to research on efficient transformer variants (Xiao et al. 2024
Code sample
Paper churba
Metscatte
Ye et al. 20044), and alternative architec- tures, such as state-space models (Gu et al. 2022 Dao & Gu
2024). mod-
emn RNNs (Peng et al.,
2024 [eck
let al. 2024 Feng al.
2024). and test-
An illustration of our reference library in LMADE - a graph of papers on architecture design (nodes containing details of the original paper, code snippets, and other details) and citation links (edges) - that our system queries when performing background research.
Figure 2:
2024 [Behrouz] et al. 2024). Since our system aims to perform autonomous research in this area, much of this related work is mod- eled directly and stored in a reference
time training (Sun et al.
library shown in Figure 2 that serves as the background work used for system ideation.
Neural Architecture Search (NAS) Lastly, we take inspiration from the NAS literature (Chitty Venkata et al. 2022; White et al. 2023 Hisken et al. 2019) Chen et al. 2023) which has the same aim of discovering improved architectures. Unlike this work, which traditionally searches fixed operation spaces (e.g., attention heads, convolution kernels), we aim for broader space of operations and architectures and, importantly, attempt to model the broader scientific discovery process. We follow many approaches NAS that employ genetic programming techniques (GP) (Koza] 1994) and more recent approaches that mix GP with LLMs (Hemberg et al. 2024 Romera- et al. 2024).
3 Language Model Architecture Discovery
As illustrated in Fig. L.. standard LMs work by embedding input. then applying N layer or block transformations over that input to produce a final representation (e.g., one that can be used for next token prediction as in autoregressive LMs). Central to any layer/block is a block design, concretely a piece of code BLM, that dictates how information flows through network. Our goal is to jointly discover novel autoregressive block designs BLM while also modeling the broader research
3

---

process associated with producing BLM. In this section, we define this problem formally ($ 3.1 and introduce our Language Model Architecture Discovery Environment (LMADE) ($ (.2) that provides the foundational tools used for discovery and for evaluating block designs BLM-
3.1 Problem Definition and Goals
We define architecture discovery a program search problem that involves finding optimal program BLM (in the space of valid programs BcM) that maximizes some fitness function BcM + R. We can define this formally as:
M K M K
argmax BLM EB2M F(BLAO). with F(BLM)
BLM
P Sz) (1)
i=1 1=
where, following standard practice, F defined as the average empirical performance Perf of BLM on set of M downstream tasks (D., DM> across K different model scales {S,, ..., Sk) (i.e., model parameter sizes). Operationally, a valid program will be any syntactically correct instantiation of the GABBase module in Figure 3 ( that involves (via the implementation of .forward) differentiable, causal transformation of input tensor X E IR batch size x len Jenh ,im to an output tensor of the same dimension and type, along with the other semantic constraints (see Table
The role of LMADE to provide input and feedback I to a separate discovery system that produces new block designs, as well as to provide all other tools needed for checking the validity of designs, verifying them through experiments, and computing F. We consider these components next, followed by discussion of our discovery system, Genesys, in $
3.2 Language Model Architecture Discovery Environment (LMADE)
LMADE consists of two core utilities, knowledge engine and verification engine that provide signal and feedback Z to a discovery system. The Knowledge Engine (KE) provides information from the academic literature that needed produce new research ideas. It specifically includes a manually curated reference library (Fig. 2 of 297 LM papers (stored in a searchable vector store) coupled with code, as well as tools for querying ArXiv, Semantic Scholar (Kinney et al. (023). and the web via services such as Perplexity.ai. More details are provided in B.1.3
The Verification Engine (VE) then provides tools for verifying the correctness of designs and executing experiments, In the former case, the VE uses a general code construct, called a Generalized Autoregressive Block (GAB) (operationalized by the GABBase class in Figure 3 see code templates in B.p B.3) to represent all architecture designs BLA and uses the structure of this module to check the syntactic correctness of each design. Semantically, the VE also includes Symbolic checker that performs static (AST-based) and runtime
code analysis to check for differentiability. causality, numerical stability, and the efficiency of a code design detailed in Table (further details in & B.1.2). Finally, VE can perform design verification by automating pretraining on filtered SmolLM corpus (Allal et al., (025) and evaluation on 29 selected LM-Exa benchmarks (Gao et al.) 2024). Standard pretraining protocols (Biderman et al. 2023) are applied (see $ for more details).
4 Genesys: Genetic Discovery System
Using resources from IMADE, our system Genesys employs genetic programming (GP)-style optimization to discover new designs. Importantly, this relies factorized representation of code designs and an evolution tree described in $ 14. Genesys then includes two core sets of agents: LLM-driven designers ($ (.2) that select past designs from the evolution tree, propose unit-wise modifications to those designs based on background research, then implement the proposed designs and add them the evolution tree. Verifiers ($..3) select designs from the evolution tree and verify them through budget-aware pre-training. We consider each component in turn and provide various technical justifications for our design decisions that we further formalize in Appendix
4.1 Evolution Tree and Design Factorization

---

In order to apply GPooptimization, Genesys factorizes each block program BLM into a discrete tree representa- tion called a generalized autoregres- sive unit (GAU) tree. For example, Figure 3 shows GAU tree for the transformer block, where each unit, or GAU, corresponds to a portion of the ex- ecutable code implementation (see exam- ple in Fig. [ This factorization forms the basis of an evolutionary free (Fig- 4 that stores new designs with these details and other artifacts. Importantly, this provides interpretable representation of the discovery search space, where each block in the tree can be compared to another via their atomic units.
Dxoculatie code
CAU tree
Design
Perfermence rectlees
Figure 4: The evolution tree in Genesys (left), where nodes denote block designs and contain each design's 's executable code, a GAU tree representation of the code, design traces, and empirical performance metries.
Based this tree representation, standard GP operations (Koza) 1994) such mutation (i.e., modifying a unit of design) and crossover (i.e., merging the units of multiple designs) can be applied (see examples in Figure [4] In contrast to traditional GP, however, we use a relaxed form of GP that does not rely on a fixed inventory of mutation operators but instead uses an LLM to generate new code units, similar Hemberg et al. (2024): Romera- Pareees et al. (002)). These units implemented using the same GAB class construct described above, which allows for a consistent set of syntactic and semantic checks on the validity of each unit implementation.
Design factorization: formal consider- ations As we discuss later ($ (.22. by representing each code artifact A as a
Ideation and research
Model implementation
GAU tree, consisting of sequence of
GAUs = [1, IN (each implemented as GABBase module in Figure ( we can use such a factorization to not only perform GP-style optimization and effi- cient validity checking, but also devise efficient al algorithms for block generation. While such representations are useful for understanding the discovery space, natural question is: Does such a factor- ization adequately capture the full de- sign space, or does it oversimplify the problem in some limiting way? As noted in Figure 3 using torchtyping- (Kidger) 2021) and Python-style type annotations, blocks and their units are naturally ex- pressible compositions of functions of type (X, Z) + (x, (or more generally Σ + Σ). Through further formalization of the language underlying these structures, in A.2 we show formally that any composition of Σ + Σ functions guarantees a decomposition of the resulting code into the kinds of GAU tree representations we use.
Proposer
Reviewer
Planner
Coder
Observer SC
Save Checkpoint
Input
Outout
Fully implemented or not
Figure 5: How do model the ideation and design stages in LM discovery? A high-level illustration of the agent subsystems, including pair of proposer-reviewer LLM (consisting of proposer and reviewer agent) that draft and score research ideas (left) and hybrid network of couneer-coder-bseee
agents (right) that produce code (consisting of a planner agent, coder agent, and ob- server agent coupled with symbolic checker (SC) tool that performs static and execution-based code analysis).
4.2 Model Designers
As shown in Figure break the design process into two stages, proposal stage and an implementation stage. Further algorithmic details are provided in $ B.2 with prompts in F
Proposal stage The proposal stage starts by selecting past design or pair of designs from the evolution tree (using the strategy $ 4.33) along with background references from the reference library, which queries the knowledge engine in LMADE. Based on this input, an LLM proposal agent comes up with a novel research idea involving a modification of the selected design(s) and writes a research proposal with high-level details of that idea and its implementation. Modifications are limited to either mutating a partieular unit the selected design, mixing units if multiple designs selected (crossover), or designing block from scratch (i.c., special case of mutating the root
5

---

Parentto) from evolitionary tee
Proposal
Restew
Pareddle
Examples from reforonce lbrray
Figure 6: How are new design ideas generated and vetted? An illustration of our proposer-reviews agent architecture using real example design artifacts (right). First, a proposer agent uses parent designs (GAU tree, proposal, verification reports) from the evolutionary tree and selected references (code, text chunks, metadata) from the reference library to generate a research proposal, which is then adversarially reviewed and scored by a reviewer agent before proceeding to implementation.
lerenced L/oor)
Unttmpp eerented
[G.H]
mp lemented Le]
uniri lomented
[t]
untrp ieeened
Q
① subtree B,A. C, E are protected
2 Implement new decompose F to G,
② Implement unit no newly decomposed units
4 Implement unit decompose to
Implement unit implemention done
Figure 7: The abstract code generation process in Genesys, where individual units in code's unit tree (GAU tree) are modified and implemented piece-by- by-piece.. Here shows the implementation of a mutation in ! involving the marked units (B, D) (all white units are protected). A new root F 2 replaces B and consists of sub-units G and H. which each get implemented in turn (i.e., transformed into executable code) 3 until the new design is fully functional.
unit in a tree). Then, a separate LLM reviewer agent reviews and scores this proposal in way analogous to an adversarial peer-reviewer and compares against past proposals to ensure novelty ($ (BB.1.)). An illustration provided in Fig. 6 This loop continues until the proposal is accepted and the score assigned by the reviewer exceeds a certain threshold. (see full algorithm in Alg. I
Implementation stage The accepted research proposals translated to executable designs BLM in this stage. Given that proposals involve unit-wise modifications to existing designs and their GAU trees, this allows for the step-by-step recursive generation shown in Fig. (see also Alg. ( This builds up a block program by incrementally constructing the GAU tree. which implicitly performs the factorization online. maintains an Unimplen emeeted list, initialized with the root of the edit- ing subtree or new tree. In each step: 1) A LLM planner agent selects an unimplemented GAU, and provides a plan for its implementation: 2) A LLM coder agent generates the Python code, potentially decomposing the GAU by declaring new children (via special statements), which will be added to Unimplementer emented with placeholder implementations; 3) Im- plementation is validated by symbolic checker (verifying GAU/GAB compliance for the current unit and the entire tree) and a LLM observer that assesses code quality, proposal adherence, and novelty against prior/sibling implementations, then rates it (threshold: 3/5). If both checks pass, the GAU accepted; otherwise, the tree state reverts for a retry. The implementation finishes when the Unimplemented
fails, retry
(A)
F
I
A
F
IN
F
Figure 8: common direct prompting strategy (A) versus our unit-based genera- tion strategy (B) and how these ap- proaches behave in the presence of failures (F) (i.e., cases when the pro- duced artifacts do not satisfy the de- sired properties).
A visualization of a
emented is empty.

---

Unit-based code generation: algorithmic advantages One motivation for unit- code genera- tion is that a direct prompting approach often fails to produce useful and valid code (i.e., code that not only improves on past designs but also satisfies the constraints in Table 60. Such direct approach,
which is familiar to many code generation systems, is illustrated in Fig. B A) and involves presenting a model with input I and, on failure to produce valid/useful output A, retries (e.g., with modified Z) until success. This difficulty can be understood formally: given the probability p of generating a valid/useful artifact A, the expected number of (i.i.) calls to the model is E[[alls
f. which will be prohibitive for most complex discovery problems with small p. In contrast to direct prompting, our approach (Fig. B) generates umit-by-unit A = Io. ..., IN. where each successful unit L, is frozen in place before the next unit is generated. This operationalizes Viterbi-style search (Viterbil (967). which we show from the first principles in $ AA.1 | expoonnttilly reduces the expected number of model calls. This explains the results in Table 3 and highlights the importance of a factorized search space.
4.3 Verifiers and Efficient Evolution
Distributed approach To allow for efficient exploration, Genesys runs the designer and verifier agents in parallel in PRoeeea-Paredes et al. (2024) (Fig. Right), both of which communicate through the evolutionary tree. The evolutionary tree is initially populated with several sfate-of-the
architecture designs, including the transformer /PPT (Biderman et al. 2023). , Mamba2 (Dao & Gu 2024), RetNet (Sun et al.) (023). RWKV6 (Peng et al., 2024), and TTT (Sun et al., 2024). Designer nodes continuously select parents (per the strategy below), query LMADE for references, and task the designer agent ($(4.2) with generating new designs, Concurrently, verifier nodes select designs/scales
and run verification in the LMADE Verification Engine whenever available. Further analysis of
optimal worker ratios is provided in $
x 350M
-508 (target scale)
-29
Design selection To effectively allocate resources, de- signers and verifiers select designs from the evolution- ary tree by balancing exploitation (i.e., refining promis- ing designs) and exploration (i.e., investigating diverse options). Designs in the evolutionary tree are assessed along two dimensions: fitness F (i.e., aggregate down- stream task performance) and confidence (i.e.. number of model scales where verification was performed). De- signs are then categorized into four quadrants (see Fig- ure 15) by upper quartiles of the two dimensions (e.g.,
40 125M
-2.58 (20x)
-29
150 x 70M
-2.1B (30x)
-29
400 31M
-1.2B (40x)
-29
Good & Confident, Good & Unconfident). Designers primarily exploit 'Good & Confident" designs for fur- ther improvement and explore 'Poor& & Confident' ones. Verifiers primarily exploit 'Good & Unconfident' de- signs (verifying at more scales) and explore 'Poor& & Unconfident" ones. Selection within quadrants is prob- abilistic, favoring higher-ranked designs (by averaging fitness and confidence) but allowing occasional random de-
1000 x 14M
-0.7B (50x)
-29
trials x scale
train. tokens
eval
Figure 9: How do we perform efficient ver- ification? Our Ladder-of-seales strategy involves starting small (training 1,000 14M parameter models on 0.7B tokens; bottom) and allocating progressively fewer trials for larger scales (training 5 350M parameter models on 50B tokens; top).
picks. GP operations (mutation/crrosooerr/ssaa sign) also chosen probabilistically (0.75/0.2/0.05 in our experiments) (see Alg. 4.
Budget management Verifying every design at each scale is prohibitively expensive. Inspired by scaling laws (Kaplan et al. 2020 Tay et al. 2022a)- - which suggest that performance correlates across scales- and the methodology commonly employed for small LMs (Hu al.) (024), we implement the Ladder of Scales strategy shown in Fig. 9 oheee several trials are performed at small parameter/token sizes, then scaled with decreasingly fewer trials. Formally, a total verification budget B,, = .... .N,y across Ns + 1 scales (e.g., 14M-350M parameters) is structured pyramidally: more trials 3, at smaller scales, with 3.+1 2 sr; - Br (sr, < 1 is the target inter scale selection ratio). Higher-scale budgets are released gradually to ensure fairness and to prevent early depletion. A dynamic allocatable budgel B, = (akg .... Ng } is initialized with ag = and a:s0 = Budgets are replenished at the lowest scale upon use, and higher-scale budgets ax+1 are released when used lower-scale budgets 3; exceed 1/8r,- The verifier node selects designs per the above strategy, verifying at the lowest unverified scale with an available budget aq > 0 (see Alg. 5).
7

---

0.034
mean, std lower, std upper, min. max
0.11
0.030
0.032
weo Liz
Wih Exp
Base
0.09
0.10
Full
Exp.
0.026
Basc Mom.
0.08
0.024
0.07
0.022
0.06
0.020
0.018
0.05
00001
0.04
0.014
0.03
0.012
0.010
0.02
0.008
0.01
0.006
0.004
0.00
0.002
-0.01
0.000
-0.02
0.0
6.0
8.0 10.0
0
8
12
16
20
28
32
36
Generation
Generation
Figure 10: Is the discovery process improving with time and benefiting from different components? The mean fitness (i.c., empirical performance) of designs through the different generations of discovery, showing (left) the first 300 designs and (right) the (mean, halved min/max and std bands for better visualization) for 1000 designs. We compare our full system against ablated versions that remove experiment verification (w/o Exp), literature search (w/o Lit.), and direct generations (Base).
Experiments
We empirically test the core components and overall effectiveness of our Genesys system. We aim to demonstrate the advantages of approach and the potential of our system perform advanced LM research. Our investigation is structured around three key research questions: RQ1 ($ 5.Th: Does GP approach lead to better and system component positively impacts the evolution process? RQ2 ($5.2): Does our unit-based code generation approach enhance design generation quality and efficiency compared to direct prompting methods, and how much do the individual model implementation agents (Fig. 5)) affect performance? RQ3 ($ 5.3): Does our system ultimately discover architecture designs that are competitive with standard architectures? Detailed setups are provided in C
stable optimization process, one where each
with additional results in D
5.1 Evolutionary Experiments
Amaz f
f
4
moz 1
MDD
First 300 designs
To assess the effectiveness of our overall system and its components, we compare full Genesys system against systematical ab- lational variants, including w/o Lit. that re- moves access to the background literature, w/o Exp. that removes experiment verifica- tion and fitness information from selection and Base that only search from the five start- ing designs with no evolution (Base w/ Mem extends this by allowing new designs be used solely background references simi- lar to RRomera-Paredes et al. (0224)) These variations allow us to address the following question: How much does literature under- standing, verification feedback, and access acquired knowledge contribute successful discovery?
Full
4.10
4.16
69.0
1.9
61.0
-0.38
who wlo Exp.
2.20
2.33
26.3
60.3
-1.10
w/o
3.37
3.67
56.7
60.3
-0.62
Base
-0.76
w/ Mem.
2.81
4.61
19.6
4.5
60.7
-2.66
First 500 designs
62.5
w/o Exp.
1.69
3.20
12.2
3.3
60.8
-1.13
First 1000 designs
Full
7.13
8.13
26.5
63.3
-1.55
Table 1: Evolution experiments under different config- urations (%). Bold/underlined denotes the best/second.
We sample 300 designs for all configurations and 500 and 1000 designs for selected setups due to computational constraints. Evolutionary progress was evaluated using the following population fitness metrics over time (population size Sp 50, step size k, 25): the End (A) and peak (Amar) fitness improvement. Volatility (v), or the standard deviation (std) of generational differences. We also measure the Sharpe Ratio (SR), or the risk risk-adjusted improvement computed as the mean of generational differences divided by their std and the Maximum Drawdown (MDD) that measures the maximal fitness decrement, which indicates stability. We note that these last metrics originate from financial economics Sharpe 1994): Gu et al. 2020bb with M DD and being the complement

---

PL.
Coder
Obs.
sc
UG
Valid
Attempts
Costs
LFC
Full
V
v
vf
92%
2.6 (+1.1)
15.0 (1:..5)
181 (#:4)
No UG
v
V
73%
3.0 (11.7)
7.9 (#7.1)
75 (+:29)
No Pl.
V
v
91% %
2.6 (+1.1)
16.0 (+:20.8
218 (169)
No Ob.
v
V
89%
(1:1.1)
12.1 (+20.1)
211 (167)
No SC
v
30%
(1:100
2.9 (14.7)
167 (1:33)
Direct
6%
1.1 (40.2)
0.3 (+0.3)
49 (5:
Table 3: Code quality vs. model designer variants w/wo the planner (PI.,, Coder, Observer (Ob.), Symbolic Checker (SC), and Unit-based Generation (UG), and "Direct" prompting strategy. Valid reports the % of valid code generated, Attempts is the avg. number of generation attempts (at most 5
times), Costs the average token cost, and LFC is the average Lines of Function-body Code.
of SR. Recently, SR has been used in reinforcement learning to measure risk-returm balance over time, which also suits our evolutionary search process and the ppploration-exploitaee
trade-off.
The results are reported in Table I For the initial 300 designs, our full system performed the best by having the highest fitness improvements (A = 4.10%, Amaz = 4.16%) and superior stability with the highest SR = 0.69 and lowest MDD = -0.38%. w/o Lit. reduced A by 0.73% and SR to 0.567, underscoring the importance of literature guidance for stable progress. Base showed negligible improvement, while w/ Mem. boosted A (to 2.81 1)) and SR (to 0.19).. confirming the value of experience. For our extended runs (500 & 1000 Designs), we see similar advantages over w/o Exp. with doubled Amor and quadrupled S f, highlighting the role of experimental feedback as selection signals. Full continued to improve up to 1000 designs, reaching a peak fitness of 0.633 and
showing signs of convergence (Fig. I0 Right). Interestingly, its evolutionary tree (Fig. 35 displays "hubness", which is analogous to the long-tail distribution of paper citations (Wu et al. 2009).
Besides impacting fitness and stability, in Table we show how the removal of com- ponents can result in increased errors dur- ing the verification process. For example, removing experiments led to code with a 19% higher error rate compared to our full system, showing how design evaluations can help avoid downstream errors later in discovery.
Full w/o Exp. w/o Lit.
Base
w/ Mem.
Err
8.61%
27.31%
7.67%
21.09%
23.70%
Table 2: The error rate (%) during the design verification and evaluation stages under different system ablations.
5.2 Designer Agent Analysis
As mentioned earlier, a key bottleneck in our discovery system generating valid code that satisfies the conditions in Table To measure the effectiveness of our full system with its different design agents and symbolic checker (see again Fig. [ we directly evaluated the implementation abilities of our designer agents using 100 proposals from our full evolution run as test cases. We systematically compared against variants of our system that removed the code planner (No. Pl), the observer agent (No Ob.), the unit- unit-based generation strategy (No UG), and the semantic checker (No SC). Finally, we compared against the Direct prompting approach discussed in Figure g We measured the rate of successful implementations passing all checkers (Valid ( (%) )), the average attempts (A (Attempts), and token costs (Costs) and Lines of Function-body Code (LFC) as a proxy for code complexity/quality
Table 3 presents the results. Removing UG (No UG) significantly degraded 90.7% in the valid rate and 58.6% in LFC compared to the Full agent, highlighting the benefit of the structured, unit-by-unit approach for generating complex and valid code. Disabling the symbolic checker (No SC) drastic reduced the valid rate by 67.4%, which confirms its importance. Ablating the Planner (No Pl.) or Observer (No Ob.) results in minimal quantitative impact, yet both play a crucial qualitative role, such as guiding the implementation order and assessing novelty/quality. Moreover, the Full agent's code complexity (avg. LFC 181) was comparable to the human- written reference library (avg. LFC 220), suggesting more realistically complex designs. Simpler setups (Direct, No UG) often produce trivial outputs (e.g.. basic ConvNets) with a significantly lower magnitude in LFC (-50).
Few vs. Many Samples: formal considerations One of the design principles underlying our agent system is that designers should produce few but deliberate and interpretable designs, much like in

---

everyday research. This is in contrast to traditional GP approaches where a vast number of simple trials are routinely performed (e.g., see Real al. (2020); Chen et al. (2024)). This raises the natural question: It is more effective to design more complex, and ultimately more expensive, agent systems that are more deliberate (e.g. our system with plamnerdobserver/ceelee, or to rely on simpler, more Table 4: How good are our discovered designs? A comparison of our seed models (top) vs. our discovered models (350M scale, 50B training tokens) based on end-task benchmark accuracy (%).
cost-effective agent systems that can perform more trials? In Appendix AA.3 we provide a formal argument that attempts to justify the former and link this decision with our Viterbi-style search.
Blimp
Wnli
RTE
WG
CoLA
SST2
WSC
IS
Mrpc
Avg.
Random
69.75
43.66
52.71
48.78
50.00
49.08
49.82
50.03
31.62
49.49
GPT
92.70
60.56
62.80
52.17
53.24
54.13
56.76
55.31
68.38
61.78
Mamba2
83.22
63.38
63.88
51.22
55.94
56.58
57.12
53,85
67.89
61.45
RWKV7
88.76
61.97
60.21
49.80
54.25
55.32
54.57
57.00
68.38
61.14
RetNet
85.16
61.97
61.35
50.51
56.29
55.43
56.03
54.95
56.37
59.78
TTT
86.13
63.38
55.23
50.75
55.55
56.35
54.93
55.31
59.80
59.71
VQH
94.37
59.15
59,91
50.28
54.25
53.56
53.83
49.45
56.62
59.05
HMamba
83.74
64.79
61.35
53.59
54.69
57.04
56.40
54.58
59.31
60.61
Geogate 90.95
59.15
61.35
52.72
54.25
55.32
58.96
54.95
68.63
61.81
Hippovq 87.96
50.70
59.91
50.28
54.25
55.73
53.83
55.68
69.88
59.80
SRN
80.83
65.52
59.55
50.75
54.45
52.98
56.03
54.95
61.03
59.57
Bold/onderline/italics denote the best/second/worst.
5.3 Discovered Model Evaluation
To measure the overall performance our new designs, we perform a standard zero-shot end-task evaluation that compares the top 5 designs discovered using the "Full" Genesys system against the five human seed designs shown in Figure 35| (GPT2, TTT, Mamba2, RWKV, and RetNet). We specifically evaluated these designs the scales of T25M and 350M parameters, trained on 25B and 50B tokens, respectively. Following standard protocols Groeneveld et al. (2024), tasks were selected based on their informativeness on smaller scales (see $ B.3.1 and Table 16 for details).
How good are the discovered designs? Tables and 14 D.I) show the results of our evaluation.
Our discovered designs ouiperforrmed/macchee baselines on 7/9 (125M) and 6/9 (350M) benchmarks, with superior averages. Although no single model dominated all tasks, consistent with many other studies on small LM development (Fourrier et al. 2024)), the discovered designs consistently performed competitively with the state-of-the-an human baselines. This shows the feasibility of using I.LMs to automate human-leve LM research, at least at the functional level. We see scaling experiments, as well as analysis on the intelligibility of the Al designs, as promising future work.
6 Discussion, Limitations & Conclusion
We introduce Genesys, autonomous system for discovering novel LM designs, featuring novel unit-hased design agent and cost-effective distributed evolution. We also present LMADE, a resource environment to support further research in this field. Current limitations include integrating efficiency- focused innovations, such as FlashAttention (Dao) 2024), hindered by complex hard hardware-specific evaluations, and the constraints of billion-parann billion-parameter-leee parameter-level discovery due limited computational resources. Future work will aim to enhance the agent's learning from feedback, possibly via reinforcement learning, as well as to develop a more adaptive design selection strategy. Our large- scale experiments yielded 1,062 novel LM architectures (14M-350M parameters), fully verified with pretraining. This is, to our knowledge, the largest automated LM discovery experiment. Genesys produced highly competitive designs; some outperformed human baselines such as the GPT and Mamba2 models in common downstream tasks. These results show the feasibility and lay the groundwork for autonomous evolutionary systems in scientifically complex and costly domains.
10

---

References
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A.,
Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. htLLhttss://are
org/ abs////s////// 14219
Allal, B., Lozhkov, A., and Bakouch, Smollm blazingly fast and remarkably powerful. Hugging Face Blogs, 2024. URI.https: h/tp:://hggggggg ingggiiggacc..cm ca/blog/ smollm
Allal, L. B., Lozhkov, A., Bakouch, E., Blázquez, G. M.. Penedo, G., Tunstall, L., Marafioti, A., Kydlíček, H., Lajarín, P., Srivastav, V., et al. Smollm2: When smol goes big data-centric training of small language model. arXiv preprint arXhv:2502.02737, 2025.
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M. K., Klambauer, G., Brandstetter, J., and Hochreiter, S. xLSTM: Extended long short-term memory. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:// openrevies net. wwt. formmforrmmddde htte:/oormmmmAAAAPPP..... ARAxPPIAhq
Behrouz, A., Zhong, P., and Mirrokni, V. Titans: Learning to memorize at test time. arXiv preprint 20Xiv:2501.00663. 2024.
Bhirangi, R., Wang, C., Pattabiraman, V., Majidi, C., Gupta, A., Hellebrekers, T., and Pinto, L. Hierarchical state space models for continuous sequence-to-sequeeee modeling. arXiv preprini arXiv:2402. 102/1, 2024.
Biderman, S., Schoelkopf. H., Anthony, Q. G., Bradley, H., O' OrBrin, K., Hallahan, E.,, Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling, In International Conference on Machine Learning, pp. 2397-2430.
PMLR, 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Chen, A., Dohan, D., and So, D. Evoprompting: Language models for code-level neural architecture search. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 7787-7817. Curran Associates, Inc., 2023.
Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Pham, H., Dong, X., Luong, T., Hsich, C.-J., Lu, Y., et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36, 2024.
Cheng, J., Novati, G., Pan, J., Bycroft, C., Zemgulyté, A., Applebaum, T., Pritzel, A., Wong, L. H., Zielinski, M., Sargeant, T., et al. Accurate proteome-wide missense variant effect prediction with 2023.
alphamissense. Science, 381 (6664)))
Chitty-Venkata K. T., Emani, M., Vishwanath, V., and Somani, A. K. Neural architecture search for transformers: A survey. IEEE Access, 10: 10:108374-10844 2022.
Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning, In The Twelfth International Conference on Learning Representations, 2024. URL https:://pennreview net/ #orrm?2d=mm222XyhB3
Dao, T. and Gu, A. Transformers ssms: Generalized models and efficient algorithms through structured state space duality. In International conference on machine learning. PMLR, 2024.
Eldan, R. and Li, Y. Tinystories: How small can language models be and still speak coherent english?
arXiv preprint arXiv:2305.07759, 2023.
Elsken, T., Metzen, J. H., and Hutter, F. Neural architecture search: A survey, Journal of Machine Learning Research, 20(55): 2019.
Feng, I.,, Tung, F., Ahmed, M. O., Bengio, Y., and Hajimirsadegh, H. Were rnss all we needed? arXiv preprint arXh::2410.01201, 2024.
11

---

Fourrier, C., Habib, N., Lozovskaya, A., Szafer, K., and Wolf, T. Open Ilm leaderboard v2. https //huggingface co/ swwpss/opceeeeeeeeeeeeepeeeeeeee -eaaeeerrppe--nnneeeeeeerrrorrd
11mmm11m_______ 1m__eeaderbaaadd 2024.
Gao, L., Biderman, S., Black, S., Golding, L.. Hoppe. T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL URL https: htopp::/arrae htt:://axxvvvoo/ org/ httg/bba/21000 00027
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., et al. A framework for few-shot language model evaluation, 07 2024. URL httLhttp://// https://zenodo org/ records/12608602
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., al. Olmo: Accelerating the Science of Language Models. Proceedings of ACL,, 2024.
Gu, A., Dao, T., Ermon, S., Rudra, A., and Ré, C. Hippo: Recurrent memory with optimal polynomial projections. Advances neural information processing systems, 33:1474-1487, 2020a.
Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. In The International Conference on Learning Representations (ICLR), 2022.
Gu, S., Kelly, B., and Xiu, D. Empirical asset pricing via machine learning. The Review of Financial Studies, 33(5):2223-22) 2273, 2020b.
Hemberg, E., Moskal, S., and O' O'Reilly, U.-M. Evolving code with large language model. Genetic Programming and Evolvable Machines, 25(2):21, 2024.
Hu, S., Tu, Y., Han. X., Cui. G., He, C., Zhao, W., Long. X., Zheng, Z.. Fang, Y., Huang, Y.. Zhang, X., Thai, Z. L., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng. G., dahai Liu, Z., and Sun, M. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling, 2024. URL https: :otps://cit /ttp://///ppeeieieeeeeeeee .et.://orrumm.dd3322211
Jansen, P., Tafjord, O., Radensky, M., Siangliulue, P., Hope, T. Mishra, B. D., Majumder, B. P., Weld, D. S., and Clark, P. Codescientist: End-to-end semi-automated scientific discovery with abs/2503, 22708
base- experimentation, 2025a. httltt//s////xxi....
Jansen, P., Tafjord, O., Radensky, M., Siangliulue, P., Hope, T., Mishra, B. D., Majumder, B. P., Weld, D.. S., and Clark, P. Codescientist: End-to-end semi-automated scientific discovery with code-based experimentation. arXiv preprint arXiv:2503.221 2002220 2025b.
Jumper. J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Židek, A., Potapenko, A., et al. Highly accurate protein structure prediction with alphafold. nature, 596(787)):583-58) 2021.
Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., Child, R.. Gray, S., Radford, A..
Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.03661, 2020.
Kidger, P. torchtyping: Type annotations and runtime type checking of tensor shapes (and dtypes, ...). 2021.
Kinney, R., Anastasiades, C., Authur, R., Beltagy, I., Bragg, J., Buraczynski, A., Cachola, I., Candra, S., Chandrasekhar, Y., Cohan, A., et al. The semantic scholar open data platform. arXiv preprint arXiv:2301.10140, 2023.
Koza, J. R. Genetic programming as a means for programming computers by natural selection. Statistics and computing, 4:87-112, 1994.
Langley, P. Scientific discovery: Computational explorations of the creative processes. MIT press, 1987.
Lee, K.-H., Fischer, I., Wu, Y.-H., Marwood, D., Baluja, S., Schuurmans, D., and Chen, X. Evolving deeper Ilm thinking. arXiv preprint arXiv:2501.0988 2025.
12

---

Liu, Z., Liu, K., Zhu, Y., Lei, X.. Yang, Z., Zhang, Z., Li. P., and Liu, Y. Aigs: Generating science from ai ai-powered automated falsification. 1:X11:241..1110, 2024a.
Liu, Z., Liu, K., Zhu, Y., Lei, X., Yang, Z., Zhang. Z., Li, P., and Liu, Y. Aigs: Generating science from al- ai-powered automated falsification. arXiv preprint arXiv:441.. 2024b.
Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha. D. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408..m 06292222. 2024a.
Lu, Z., Li, X., Cai. D., Yi, R., Liu, F., Zhang, X., Lane, N. D., and Xu, M. Small language models: Survey, measurements, and insights. arXiv preprint arXiv:2409.. 10000.9,1779... 2024b.
Merchant, A., Batzner, S., Schoenholz, S. S., Aykol, M., Cheon, G., and Cubuk, E. D. Scaling deep learning for materials discovery. Nature, 624(7990):80 85, 2023.
Nearing, G., Cohen, D., Dube, Gauch, M., Gilon, O., Harrigan, S., Hassidim, A., Klotz, D.. Kratzert, F., Metzger, A., et al. Global prediction of extreme floods in ungauged watersheds. Nature, 627(8004):559 563, 2024.
Park, H., Yan, X., Zhu, R., Huerta, A., Chaudhuri. S., Cooper, D., Foster, I., and Tajkhorshid. A generative artificial intelligence framework based on molecular diffusion model for the design of metal-organic frameworks for carbon capture. Communications Chemistry, 7(1):21, 2024.
Paster, K., Santos, M. D., Azerbayev. Z., and Ba, J. Openwebmath: An open dataset of high-quality mathematical web text. In The Tweifth International Conference on Learning Representations, httf/oommmddjjkkjjjll
2024. URL /ttttts://pperreeeeeeeee
Penedo, G., Kydlfček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL
https: pectps//aarae //arxiv/ www..rgg////aoooeo//....eeeeeeet / 17557
Peng, B., Goldstein, D., Anthony, Q. G., Albalak, A., Alcaide, E.,, Biderman, S., Cheah, E., Ferdinan, T., GV, K. K., Hou, H., Krishna, S., Jr., R. M., Muennighoff, N., Obeid, F., Saito, A., Song, G., Tu, H., Zhang, R., Zhao, B., Zhao, Q., Zhu, J., and Zhu, R.JJ Eagle and finch: RWKV with matrix-valued states and dynamic recurrence, In First Conference Language Modeling, 2024.
URL httpp:://ppenreeeeee htt.://e...ttooommmmooooossss.i.
Qin, Z., Yang. S., and Zhong. Y. Hierarchically gated recurrent neural network for sequence modeling. Advances in Neural Information Processing Systems, 36:33202-33221 2023.
Real, E., Liang, C., So, D., and Le, Q Automl-zero: Exolving machine learning algorithms from scratch. In International conference on machine learning, pp. 8007-8019, PMLR, 2020.
Pareees, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J., Ellenberg. J. S., Wang, P., Fawzi, O., et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468 475, 2024.
Saxton, D., Grefenstette, E., Hill, F., and Kohli, P. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations, 2019. URL https
//openreview. peneenrevies.ne/foormm// sittreeee....etooooooorrdietsss 2it.oornnmmid1111g55555552
Schmidgall, S., Su, Y., Wang. Z., Sun, X., Wu, J., Yu, X., Liu, J., Liu, Z., and Barsoum, E. Agent 2025a.
laboratory: Using I1m agents research assistants. urXiv:2501.04227 2025a.
Schmidgall, S., Su, Y., Wang, Z., Sun, X., Wu, J., Yu, X., Liu, J., Liu, Z., and Barsoum, E. Agent laboratory: Using Ilm agents as research assistants. arXiv preprint arXiv:2501.04222 2025b.
Sharpe, W. The sharpe ratio. Journal of portfolio management, 21(1):49-58, 1994.
Sun, Y., Dong, I.,, Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: successor to transformer for large language models, 2023.
Sun, Y., Li, X., Dalal, K., Xu, J., Vikram, A., Zhang, G., Dubois, Y., Chen, X., Wang, X., Koyejo, S., et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint 240.:22000.04200 2024.
13

---

Tang, Y. Han, K., Liu, F., Ni, Y., Tian, Y., Bai, Z., Hu, Y.-Q., Liu, Jui, S., and Wang. Y. Rethinking optimization and urchitecture for tiny language models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMIR.org. 2024.
Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W. Rao, J., Narang, S., Tran, V. Q. Yogatama, D., and Metzler, D. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint 2rXtv:2207.10551 2022a.
Tay, Y., Dehghani, M., Bahri, D., and Metzler, D. Efficient transformers: A survey, 2022b. URL www.://gg..........///// 06732
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, Attention is all you need. Advances in neural information processing systems, 30, 2017.
Viterbi, A. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE transactions on Information Theory, 13(2):260-22) 269, 1967.
Wan, Z. Wang, X., Liu, C., Alam. S., Zheng, Y., Liu, J., Qu, Z., Yan, S., Zhu, Y., Zhang, Q.. Chowdhury, M., and Zhang, M. Efficient large language models: A survey. Transactions Machine Learning Research, 2024. ISSN 2835-8856. id=bsCCJHb08A Survey Certification.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, Liu, S., Van Katwyk, P., Deac, A., et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-60, 2023.
White, C., Safari, M., Sukthanker, R., Ru, B., Elsken, T., Zela, A., Dey, D., and Hutter, F. Neural architecture search: Insights from 1000 papers. arXiv preprint arXiv:2301. 08727,, 2023.
Wong, F., Zheng, E. J., Valeri, A., Donghia, N. M., Anahtar, M. N., Omori, S., Li, A., Cubillos-Ruiz,
A., Krishnan, A., Jin, W., et al. Discovery of structural class of antibiotics with explainable deep learning. Nature, 17771997) 2024.
Wu, L.-L., Luesukprasert, L.,, and Lee, L. Research and the long tail: A large-scale citation analysis. In 2009 42nd Hawaii International Conference System Sciences, pp. 1-10. IEEE, 2009.
Wu, Y.-F., Lee, M., and Ahn, S. Neural language of thought models. arXiv preprint a1X 2024.
Xiao, D., Meng, Q., Li, S., and Yuan, X. Improving transformers with dynamically composable multi-head attention. In Proceedings of the 41st International Conference on Machine Learning. (CML.24. JogR. 2024.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.1 17453, 2023.
Yamada, Y., Lange, R. T., Lu, C., Hu, S., Lu, C., Foerster, J., Clune, J., and Ha, D. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066 2025.
Ye, T., Dong, L., Xia, Y., Sun, Y., Zhu, Y., Huang, G., and Wei, F. Differential transformer. arXiv preprint arX1v:2410.05258, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We present experiments in and D theoretical results in $ A.1 also show extensive analysis in
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Discussed in 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We provide proofs in AA.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide full implementation details
B and full experiment details in
$
C
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: We provide full experiment details in S and we will release all our code and discovered models upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
hubllc//gulaes/oddeaabiiisssissi iooll for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /mips.ccc/pulic//uuu guwde//ddee/eee Cot//ooodeeesiseeeeeee hwtpbbbmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide full implementation details in and full experiment details in
$
CC
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive for the evolution experiments. But we report the std for agent experiment (Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details in $
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed and conducted the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: have an impact statement section following the main text.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited all assets provided the URLs.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We introduced LMADE in $ and and provide full details in $ B.1
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Our paper uses LLM agent to sample LM designs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

Appendix
A
Formal Analysis of System Design and Proofs
24
A.1
Search (VS) Proofs
24
A.1.1
Direct vs. Viterbi Approach: Basic Analysis
24
A.1.2
Refined Analysis: Token Costs with Growing History
25
A.1.3
Additional Advantage: Extended "Design "Tokens" in VS
................................
26
2
Unit Tree Factorization Proofs
27
A.2.1
Case 1: Σ Σ Programs
27
A.2.2
Case 2: Extending to General Typed Programs via Type- Lifting
28
A.3
Evolution Efficiency of Genesys: Few High-Quality Samples with VS
29
A.3.1
Few High-Quality Samples Outweigh Vast Low-Quality Trials
29
A.3.2
Combining VS with the "Few High-Quulityy Samples" Argument
29
B Implementation Details
30
B.
LMADE Component Details
30
B.1.1
Reference Library
30
B.1.2
Symbolic Checkers
31
B.1.3
Knowledge Engine
32
B.1.4
Verification Engine
33
B.1.5
Additional Details of Designer Agent
34
Pseudo Code
34
B.3
Program Templates and Base classes
40
B
4
Mutation and Crossover
42
B.5
Design Selection
42
B.6
Discovery Console
43
Experiment Details
43
Experiment setups
43
C.1.1
Training Corpus
43
Hardware Environment
C.1.3 Model Experiment Settings
44
2
Evolution Parameters
45
Verification
45
Additional Results
46
D.I Experiment Results on 125M
46
D.2 Sensitivity of Metrics
47
Analysis of Genesys
47
E.1 Analysis of Evolution
47
22

---

E.1.1
Analysis of Design Sessions
48
E.1.2
Analysis of Evolutionary Tree
51
Analysis of Agents
52
E.2.1
Analysis of Foundation Models
52
E.2.2
Analysis of Implementation Errors
54
Analysis of Discovered Models
55
E.3.1
Analysis of Verification Process
55
E.3.2
Analysis of Model Units
57
E.3.3
Unit-Performance Relation
58
Analysis of System and Performance
60
E.4.1
Training Time Estimation
60
E.4.2
Optimal Pipeline Throughput and the V-D Ratio
60
Prompts
62
Proposer
62
F.1.1
System and GP Background
62
F.12
Search and Refinement
64
Reviewer
66
F2.1
System Prompt
66
F2.2
Final Review
68
E3
Planner
69
F.3.1
System Prompt
69
F33.2
Plan and Unit Selection
74
Coder
74
F4.1
System Prompt
74
F.4.2
Implementation and Debugging
79
Observer
F5.11
System Prompt
81
F5,2
Observation Feedback
84
Qualitative Examples
85
G.1
Example GAU Trees
85
G.I.I
Five Evaluated Designs
85
G.
1.2
Complicated Designs
87
G.2
Example Design Artifact
87
23

---

A Formal Analysis of System Design and Proofs
In this section, provide further formal analysis of the technical and algorithmic points made in $ In $ A.1. we discuss the properties of our unit-by-unit Viterbi generation strategy and its advantages over direct prompting approach. In $ A.2 we formalize the structure of block programs BLM and use this structure to justify our GAB tree factorization. Finally, in $ |A.3222 we justify our decision to optimize for few-vs-many samples in our GP approach and link this with properties of our Viterbi search strategy.
A.1 Viterbi-style Search (VS) Proofs
To compare different prompting strategies, we analyze the expected number of attempts (and then token costs) of single-shot direct- prrect-rromptng (direct) approach versus Viterbi-style Search (VS) or our unit-by-unit generation approach. The argument based on straightforward properties of the geometric distribution, but yields an exponential advantage (Prop. T for VS when design artifact must satisfy multiple constraints.
A.1.1
Direct vs. Viterbi Approach: Basic Analysis
Setup Let A be the set of possible final artifacts (e.g. fully implemented LM architectures). We consider:
A direct approach that tries generate an artifact A in one shot. the artifact invalid (c.g., doesn't t pass the checker in Table 6). we discard and try again from scratch.
A VS approach that factorizes the generation into N sequential sub-decisions, each retried upon failure only locally (i.e. we "checkpoint" partial successes).
The following result relates to the expected number of model calls for the direct approach.
Lemma 1 (Single-Shot Expected Calls). Suppose the probability of success (i.e., generating valid output that satisfies some target constraints) in one single-shot generation is Pvalia E (0, 1). Then the expected number of calls until success (denoted as B[[allsaree)) is
E[calSsairea]
Pvalid
assuming each call is an i.i.d. Bernoulli(poaiia) trial.
Proof. This follows directly from the geometric distribution: the probability we succeed on the k-th attempt is (1 * and the expected number of attempts is /vPalidd
In many scenarios, success requires N sub-components to be correct simultancously (e.g., N gener- ated code units all being correct), each with probability ps. Then
Pvalid
Pk A=1
E(callsarree]
l Pk
Viterbi-style Unit-based Factorization In the VS approach, we imagine the creation of an artifact as N steps:
To I where each step (4-1 + I+) succeeds with probability Pk- Failures at step k do not discard previously completed steps; we simply revert to Iş-2 and retry. For this approach, the following holds:
IN A,
24

---

Lemma 2 (Expected Calls: VS). If step k has a probability Phe of success on each attempt, the expected total mumber of calls to complete all N steps
E[callsvs]
Pk
Proof. Step k follows geometric distribution with success probability pk. Hence, its expected trials are 1 1/pn- Summing over l: = 1,..., yields E 1/pt-
From these two facts, the following follows regarding how the VS approach requires exponentially fewer model calls over the direct approach.
Proposition (VS vs. Direct: Exponential Gain), If Pvalid
I Pk, then
N
[[callSrireat] RJ
E[callsvs]
=
P
k=1 Ph:
In typical cases where II & for each j. we have Ex= 1/pb & 1/II Pko indicating potential exponential improvement for VS.
Then the following corollary follows straightforwardly.
Corollary 2 (Identical Steps Case). Pf for all k, then
N p
Elcallsareal
E[callsvs]
NN'
N-1 + (exponentially) showing that the advantage of VS grows N/P = NN As grows, 1/p" dramatically with larger N.
The exponential gain of VS also explains why high-quality samples outweigh vast low-quality ones may take exponentially more samples to reach the same optimal point in a VS sample.
A.1.2 Refined Analysis: Token Costs with Growing History
Given that each model call incurs token costs, an exponential improvement in the number of calls or steps means that the token costs in terms of the prompting and history tokens that need to be processed during each try, Let:
may reduce exponentially. In this section, quantify such token
Hy: the number of "history" input tokens at step k.
òg: any additional instructions or new tokens in step k.
Oş: the number of output tokens generated by step k.
Gs Co: cost coefficients for input and output tokens, respectively.
Then the cost of single attempt of Step le is:
Costs = c: (H2 + 5x) + co Ox-
Under geometric retries, the expected attempts step i: are 1/pk and the following holds.
Lemma 3 (Expected Token Cost in VS). The expected total cost complete all N steps in VS
is
R[[ostys)
C (Hx + 5x) + co 0²
Proof. At each step kt we expect 1/ps attempts. Each attempt incurs Cost,, Summing over completes the proof.
25

---

Comparison to Single-Shot In a single-shot approach, the probability of success is Iik=1 Pk. so the expected number of attempts is ([[[]::). Each try regenerates the entire artifact. Let its cost be Costaill = ca ca(...) ) + co(...). Then
E(Costrireet
Costaill
I
Pk
Thus, even accounting for partial history {H&) in VS, one can still have:
Costs
Costail
Pk
especially when II ps is very small. This confirms that the exponential improvement result extends to token-cost models, not just the raw count of attempts.
Conclusion (VS) Our Viterbi-style search can yield an exponential reduction expected attempts (and potentially in token cost) compared to single-shot approaches, particularly when many sub- decisions need to be correct simultancously. This underpins the efficiency of Genesys's stepwise Cramner-Coder-Obeeeee pipeline, where partial successes preserved.
A.1.3 Additional Advantage: Extended "Design Tokens" in
Beyond merely reducing number of attempts (Lemma 2). VS can also increase the total amount of "reasoning" or "design tokens" used in the generation of a final artifact, thereby improving its quality, formalize this idea as follows.
Setup and Definitions Consider two approaches for generating a complex design artifact A:
Direct Generation:
A single pass length Mshot produces the final design A.
2. VS Generation:
Generation is factorized into N sequential steps ($2), each potentially adding or refining partial outputs, with local checks and retries. Let M2 denote the number of tokens used (or produced) at step k and let Mys E M2 be the total number of tokens across all steps.
We say that each token that directly contributes to the final design is design token. In the VS approach, multiple partial expansions, corrections, or debugging messages can yield larger corpus of design tokens than in single-shot approach.
Assumption 1 (Monotonicity of Quality in Token Budget). Let Q(A) be the "quality" (e.g. correct- ness or score) of a final artifact A. Suppose that there a non-decreasing function f(-) such that the expected quality of an output improves as the number of design tokens grows:
E(Q(A) token budget= = m] 2 f(m).
In particular, (m) is strictly increasing in 0 (more design tokens lead, in expectation, to bigher-quality artifacts).
This assumption echoes widely observed phenomenon: longer intermediate reasoning or drafting stages (e.g.. Shaain-ff-thoughe can improve correctness for difficult tasks.
Lemma 4 (VS Allows Strictly More Design Tokens). Let Manor be the fixed budget of design tokens in a single-shot approach. In Viterbi-style Search factorization with N steps,
N
Mys
MR.
M,
where each Mj. may include expansions, partial corrections, or debug logs. Provided the partial checks do not enforce strict token cap each step, one can typically satisfy
E Mvs]
>
M
26

---

meaning the expected total design tokens used in VS can exceed the single-shot token budget.
Sketch. In single-shot generation, the artifact is created once, yielding total of Mahor design tokens (e.g. the length of the entire output). In contrast, in VS, factorization allows partial expansions and corrections. If any of the N steps fail local validation require refinement, the retry mechanism may produce additional tokens: detailed debug messages, corrective instructions, or iterative expansions. Hence, the total number of generated (and possibly regenerated) design tokens Mys can exceed Mahor. Even with local checks, the system is not obligated to "truncate" expansions at each step, so the expected design token count across all steps is strictly larger on average if any fraction of steps require more than a single attempt.
The following then holds.
Proposition 3 (Quality Gain from Viterbi-style Search Extended Tokens). Under tion let Ashot be the artifact returned by single-shot approach with a fixed budget Manons and let Avs be the artifact returned by the VS approach with random total tokens Mys. Then:
Assump-
[[v(s)))] 2 (MvsM)s)] > [Maaa),
whenever [E[Mvs] > Mahoc and f(-) is strictly increasing.
Proof. By Lemma in VS, factorization can spend more design tokens in total. Since f((-) is strictly increasing, having larger token count (on average) implies strictly higher expected quality. Formally.
[(vvs)) 2 f([[Mvs))
(Mssho)
where the first inequality follows from Jensen's inequality if f is convex and increasing and the second strict inequality follows by [8[svs] >> Mahor Hence, the expected quality under VS greater than under a single-shot approach constrained to Mahot tokens.
Interpretation In practice, difficult designs or proofs often benefit from iterative expansions, re- checks, "chain-of-thoughh" style reasoning. In VS, factorization naturally accommodates these partial expansions, producing a greater quantity design tokens. If one presumes that additional design tokens correlate with more thorough reasoning and thus higher quality tthnn Proposition shows a theoretical justification for the quality advantage of multi-checkpoint VS over single-shot generation. "More tokens" here refers specifically to effective design content, including debug corrections or expansions that shape the final artifact, rather than random filler text. As a result, factorizing generation (rather than forcing single, short pass) can yield both exponential savings in attempts (22) and an increase in solution quality (33), making VS highly advantageous in complex, multi-component discovery tasks.
A.2 Unit Tree Factorization Proofs
As noted in $ [. and shown in Figure (via the type annotations), code blocks BLM naturally consist of compositions of units (c.g.. multiple head attention, GatedMLP) each with type (X, 2) - (X, Z) (below we generalize this to the type mapping Σ + E). Below we use this type structure to justify particular GAB factorization, showing specifically that any program P Σ - Σ in the category of GAB programs can be factorized into a finite tree of sub-blocks (i.e., the kinds of unit-based representations use).
A.2.1 Case 1: Σ + Programs
Definition (Unit Tree for Σ > E). Suppose Lx is a typed language closed under composition and identity E. A unit tree for program P. E + Σ is finite, rooted tree T such that:
1. Each node is labeled by subprogram Q : Σ > E in Lx.
'Ir f is just den-decr,,,,ng, we have [Mvs) 2 f (inf Mvs). In practice, partial expansions typically ensure inf Mvs > Mahui
27

---

2. Leaves are atomic or empty/identity. (In practice, we do not expand or decompose program into this level. which makes a unit tree degrade into an AST.)
3. An internal node labeled P factors as P P1 P2 .... PR. where each P. Σ E.
The following then follows and ensures block into smaller Σ > E sub-blocks, enabling targeted GP operators.
always decompose the units in single large GAB
Theorem 4 (Unit Tree Factorization for Σ - Σ). Let Ly be closed under composition and identity in Σ. Then every program
P:E-E, PE Lz,
admits finite unil free T = (PP) in the sense of Desinition 1
Proof. Base Case, If P is atomic or the identity map, we define (PP) to be a single-node tree labeled by P.
Recursive Case. If P can be expressed as P = P1 . P&. with each P; E Exs then by induction each P. has tree T; ((:)). We form (PP) by adding root node (labeled P) with children TI...., Tk. Closure under composition ensures this remains in Lz.
Termination. A syntactically finite program eventually decomposes into atomic identity forms, guaranteeing finite tree.
A.2.2 Case 2: Extending to General Typed Programs via Type-Lifting
In practice, many programs do not preserve the same shape or type. For example, in real block designs, the skip and residual connections involve mappings of type Q: X X, and hence do not fit in the language defined above. Below, we show how to embed (lift) Q into a E - Σ function, then apply Theorem 4 Importantly, this shows how our factorization, as well our broader GP search, can be extended to problems with different type structures.
Universal Type E Assume that have encoders and decoders:
Encx X + Decx + X, Ency + M, Decy E + Y,
such that Decx (Encx (z)) for all E X. Then we define:
Definition 2 (Lifted Function Q). Given Q X Y, its lifted version @ : + Σ is:
(s) Ency | [D(Decx(s))
The following then establishes that we can preserve the type mapping Σ + E using type raising.
Proposition (Unit Tree Factorization for General Q: X + Y). Let C be closed under com- position. Then any Q X + Y can be lifted to Q Σ + Σ (per Definition 2 and by Theorem Q admits unit tree in Ly. This induces corresponding decomposition of the original Q
Proof. Because L is closed under composition, Q Ency Q Decx E Lx. By Theorem factors into a finite tree of Σ sub-blocks. Those sub-blocks, when "projected" back through Decx and Ency, yield valid decomposition of Q.
Designing E in Practice
Overhead vs. Gains: Merging X and Y into single can increase memory or prompt size. However, partial or selective factorization can mitigate overhead.
Atomic Black Boxes: certain submodules are not to be searched or mutated, we can treat them as atomic.
Recursion, Higher- Order Functions: Q returns a function or is unboundedly recursive, partial unrolling or bounding is required for finite tree.
28

---

Conclusion We have shown that any Σ + Σ program can be factorized into composable sub-blocks (Theorem 4 and that can lift function X Y into Q - > (Proposition 5). This generalizes the unit tree factorization approach well beyond autoregressive shapes, allowing Genesys apply genetic programming (GP) operations arbitrary programs while still benefiting from the efficiency Viterbi-style Search discussed Section A.1
A.3 Evolution Efficiency of Genesys: Few High-Quality Samples with VS
We now unify two key ideas behind Genesys's efficiency:
Few vs. Many Samples: A smaller number of high-quality (and valid) samples can yield more improvements than large number of low-quality trials, given fixed cost budget.
2. Viterbi-style Search (VS) Exponential Advantage: Factorizing the design process into multiple sequential steps (each retried locally) exponentially reduces the expected attempts to produce valid final artifact, compared to single-shot (direct) approach that must get every sub-component correct one go.
By combining these points, we show that Genesys's approach appruich-focuuiin on more careful, iterative code generation with local checkpoints (VS) (VS)-furhher magnifies the benefit of "few, high-quality samples" over "vast, low-quality trials."
A.3.1 Few High-Quality Samples Outweigh Vast Low-Quality Trials
Setup and Yield Let:
Q [0, 1]: Probability newly generated design is beneficial improvement over the current best or population.
E' [0, 1]: Probability that the design valid (e.g. compiles, passes checks, etc.).
0: Average cost per sample (e.g., tokens GPU hours per generation).
B > 0: Total budget in the same cost units.
Hence, the maximum number of samples N B, Only a fraction Q E of these N samples will be valid and an improvement. Defining = Q x E, expected yield is:
B (QE)
B
Y
c
c
Strutegy has parameters (Q1, Ei,c) (1) and Strutegy has (Q2, F2, e2), both under the same total budget B, then:
Proposition (Few High-Quality Samples Outweigh Vast Low-Quality Trials).
Q1 Ei
Q2 Es
Yi Y2, where Y; Q;Ea B
C1
C2
Gi
Interpretation: Even if Strategy 1 generates fewer samples (larger es ), it can yield more total improvements, provided each sample sufficiently more likely to be valid and beneficial.
Proof. The proof is simple rearrangement:
B
B C2
E1
Eg
Y1 (Q
Y2
E2
Y
C1
02
A.3.2 Combining VS with the "Few High-Quality Samples" Argument
VS Exponentially Increases Validity in Complex Designs Recall Proposition in Appendix A. if an artifact has N sub- cub-components, each with success probability Pk, then single-shor approach must succeed simultancously with probability II Pk. which can be extremely small. By contrast,

---

a Viterbi-style Search (VS) scheme that checkpoints partial progress has expected total number of calls only Ek= 1 rather than 1/ [[ 1/ pk), yielding exponential improvement for large N. Fx Thus, under VS, the effective validity Evs (chance of eventually producing a correct artifact) can be far larger than Pdirect
Il Pk.
Genesys Achieves Higher QE Term In Genesys, VS drastically increases E' for complex designs by preserving partial successes, while the literature-based designer and evolutionary selection raise Q (the chance that a new design is genuinely beneficial). Even though each Genesys attempt costs somewhat more (raising c), the net effect can still be Q > Pl. Hence, by Proposition 6
Craave
Genesys can produce more total improvements under fixed budget B.
Proposition (Genesys's VS Increases E.E.. If Pdirect = T Pk. the single-shot validity probability for an N -ccopponett artifact, and Evs is the probability of success via Viterbi-style Search, then typically Evs > Pdirect- As long as evs (the cost per Genesys attempt) does not
grow exponentially in N, the ratio Evs can be exponentially greater than Q I leading to higher yield Y.
eys
Chatve
A Additional Quality Advantage of VS Beyond boosting validity E, the stepwise factorization in VS also increases the "design tokens" and iterative refinements per sample ($A.1.3). This can further raise Q (the chance a beneficial improvement) by allowing more debugging, partial expansions, or chain-of-thought. Combined, these effects further enlarge QE
Conclusion By merging the "few high-quality samples" principle with the exponential gain in validity from VS, Genesys obtains higher Q and thus a higher yield Y = (Q) F)) (B/c). Even if Genesys attempts fewer samples, each has significantly greater probability of (1) being valid (via factorized re-tries) and (2) being beneficial (via literature grounding and evolutionary selection). Empirically, this leads to more successful discoveries than approaches that generate many low- quality trials. "More tokens" also tend to enable more sophisticated reasoning, further increasing the probability that design is beneficial. Thus, the synergy of quality improvement (raised Q) and validity improvement (raised E) explains why Genesys can be highly efficient despite producing fewer with more carefully crafted samples.
B Implementation Details
In this Section, we provide additional details of the LMADE components in B. succinetly discuss the implementation with pseudo codes in B.2 and conclude with the GAB, GAU, and the LM base class and templates in B3
B.1 LMADE Component Details
B.1.1 Reference Library
We manually constructed a reference library of pivotal innovations in Transformer and alternative architectures. Besides seminal works like GPT, we manually chose papers from the last three years of leading conferences (e.g.. ICLR, ICMI., NeurlPS), and prominent arXiv publications based on citations or social media discourse, an increasingly prevalent means of academic dissemination. The survey papers TTy et al. (2022b); Wan et al. (2024) and the community GitHub resources cited in Table [ served as foundational references. We exclude the work in these directions: I) optimizations, such as GPU-level optimizations or quantization; 3) Caching, or other efficiency improvements. 4) Inference-ster stage methods; 5) Application-specins
Distillation or non-standard training methods; 2) Hardware-spectne
optimizations (e.g., for finance or healthcare); 6) Audio or video processing techniques; 7) Post-training enhancements such as fine-tuning; 8) Methods based on parameter sharing. We compiled 297 reference designs. Metadata like titles, authors, and abstracts were retrieved via $2, forming reference nodes in the EvoTree, connected based on the citation of each other.
30

---

Repository www.gr//ass--linaaaatttttttoo
Description
A collection of state thethe-art lin- ear attention models.
LAlON-AA/hucidrrans-pppeeee
Projects created by lucidrains about trunsformers.
Must-read papers and blogs on LLM-based Long Context Modeling. Paper list for State-Space-Modee and its Applications.
CCenntAHUMMMabbaSSaii__.. Swwcs_ppaaceeeePPPPeee List
yygjintagg/veesonmMMmbmaaaaap
This repository compiles list of pa- pers related to Mamba and SSM. A curated collection of resources lated to Mamba. Table 5: Github repos we referred when building the reference library.
XiudinCCal/Awesomee-Me Mamba-Collaaaiiiii
Component
Description of test
static
execution
parser
Checks that block syntactically valid (AST-based).
formatter
Checks that block follows GAB protocol (Fig. 3
initialization
Checks that the PyTorch module can be initialized.
forward
Checks that forward pass can be performed.
backward
Checks that backward pass can be performed.
causality
Checks that block employs causal masking.
differentiability Checks that module is differentiable and doesn't involve unused parameters.
effectiveness
Checks for correct training behavior on a small corpus, e.g., stable gradients, loss convergence, reasonable flops.
Table 6: How do check if block design BLM is valid? A description of our Symbolic checker LMADE that performs static- and executation baaed code analysis to determine code validity.
We manually find their implementations, 185 out of them have released available code base, select most typical designs as seed designs to initialize the EvoTree where each of them represents a popular novel architectural idea: GPT Brown et al. (0200) is the most popular Transformer-bass based
architecture; Mamba2 Dao & Gu (2024 represents the State Space Machines and Linear Attention
modern RNNs; RetNet Sun eet al. (2023) explores the balance among Transformers, RNNs, and Linear Attention models; TTT Sun al. 22224) represents novel idea of test-time training. For the other 180 designs, manually extract the LM block implementation or core implementations of their proposed method from the released code base and store them in the node data. When reference is selected, the metadata, as well as the code, if any, will be provided as part of the prompt.
models; RWKV6 Peng al. (2024) represents the latest progress
B.1.2 Symbolic Checkers
We develop a symbolic checker to check the validity of a design without performing the costly actual verification process. The components are listed in Table 6 It can be roughly divided into the static format checks based on Abstract Syntax Tree (AST) traversal which mainly checks if the code follows the format of GAU and GAB, and fix some simple errors like passing the dtype and devices, not using required arguments; and the Runtime fuunctiona checks, which tries to initialize the corresponding PyTorch model, then check its forward and backward pass, differentiability of parameters, as well as its causality by examing: given a sequence X with length L, whether Y = f(X tl) changes by changing X|t + L] for from to L 1. It also launches quick training with 10 gradient steps on the Wikitext-2 dataset, then checks if the gradient norm exploding, the loss is decreasing, and if the training time and FLOPs are 5 times higher than a GPT model trained in the current machine, whose training statistics is automatically tested and stored in a benchmarking report to compare with. F.2.2 analyzes the distribution of errors detected by the symbolic checkers.
31

---

Early Termination of Implementation: A unit is accepted and the implementation state advances to T only if passes both the checker and the observer. Otherwise, rollback 2 and retry this step. After K faits failures, the agent ceases effort and will re-implement this proposal at later time. A proposal may be abandoned up K Kattempts times before is deemed "unimplementable".
B.1.3 Knowledge Engine
As shown Fig. [ the Knowledge Engine contains three modules, the External Sources for the literature search, the general Web Search, and the Paper Vector DB for the internal Reference Library as discussed above in & [...I. When querying the Knowledge Engine, the agent needs to fill in three fields: the keywords, description of the intended content, and the instructions for the web search agents. The keywords were used to query the external sources, while the description was applied to locate relevant excerpts from the paper vector DB. Missing keywords or descriptions will lead the skipping of the external sources search and the paper vector DB search, respectively.
External Sources We search for papers after 2015 from the top ML or NLP conferences, including NeurIPS, ICML, ICLR, ACL, EMNLP and NAACL from S2. arXiv, we filter the results by domains: Machine Learning (cs LG) and Computation and Language (cs.CL). For Papers Citees,,tCCoee, we not set a filter, and we request both paper and repo
Paper Vector DB We downloaded the paper PDFs for the papers the reference library, and converted them into text by MathPix, which can accurately convert mathematical content into plain text, then split them into chunks with the SemanticChune down the text into semantically different chunks by analyzing the gradients of distances of chunks computed with OpenAl next-embedding-33 ---lagge embedding model. We embed each chunk with the same embedding model in vectors and then store them in the Pinecone vector store The vector db will available to use by the Knowledge Engine. When retrieving, we apply Cohere rerank- english-
from LangChain which breaks
reranker to filter the top 20% most relevant results.
Web Search We use Perplexity.ai ai for the web search, which is LLM-based search engine. It accepts natural language queries as input and returns a response containing a summary of search results with references from the websites. We select 1lama-3 sosssonnaaarrae 1ow---1kkkknn onkkonnine as the base model with a maximal number of completion tokens set to 4000. We apply the following system prompt:
System Prompt for Peplexity.ai
You are AI research assistant who helps language model researcher gather information for discovering the best novel autoregressive LM block that can defeat the existing state-of-the-art models.
## Background
Modern LMs typically structured stack of repeating blocks. The goal is design novel LM block that outperforms current state-of-the-ar models, aiming for:
Low perplexity corpora,
High accuracy on downstream tasks,
Robustness varied inputs,
Efficiency in both training and inference,
Excellent scalability with more data and larger models.
You will be provided with the researcher's thoughts, analysis. and descriptions, and your task is to understand the intent of the researcher and search for the information that can best help the intent.
https://python.c 1wwghhain.nnncnnin. http::///pppooonaaaaaaaaacaaeead wttp////cc//oooo..oo/aaaacccccca
httpp://wwws www.wwwwinccce..mm io/ https:/ccohee..com
32

---

We use the following prompt to pass a query to the model:
Prompt for Peplexity.ai ai Query
Here the information from the researcher: {query)
Understand the goal, idea, and intent of the researcher. Find the most useful information that can best help the researcher to achieve the goal.
Interface When querying the Knowledge Engine, the agent needs to fill in three fields: the keywords, a description of the intended content, and the instructions for the web search agents. The keywords were used to query the external sources, while the description was applied to locate relevant excerpts from the paper vector DB. Missing keywords or descriptions will lead the skipping of the external sources search and the paper vector DB search, respectively. The instruction is fed to Perplexity,ai for web search. If the instruction is missing, the other non-empty fields will be provided to the agent as the query. The composed results from all sources are returned
B.1.4 Verification Engine
component
description
check # parameters tuning
Checking model has appropriate # parameters, and tunes parameters to fit model scale.
gradient accumulation steps
Tune gradient accumulation steps to avoid OOM. Table 7: Auto- Tuner components.
Auto-Tuner At the beginning of the verification process, as presented in Table 7 we use an auto- tuner to guarantee the model size fits the scale and decide on gradient accumulation steps that do not trigger an Out-Or-Memory (OOM) error in the current machine. A block loader automatically fetches the GAB and composes the LM, then uses this Auto-Tuner to do pre- rre-verification checks and tuning.
Tuning model size: We tune the model size by adjusting the two standard arguments of GAB, which are detailed in $ [.3 num_blood block and embed_ dim. We apply simple depth-first strategy as per Tang al. (2024), which claims that depth (remm_block) provides more performance gain than width (embed dim). For each scale we take the non-embedding parameter number of GPT P, reference, tuning the parameter until the model size M falls into the region 0.8P, < M < 1.2P.. first tunes the block, starting with I and gradually increasing until 1. The size fits the region, 2. the size exceeds, or tries for more than 1000 times. If exceeding, the tuner will try to tune the embed dim by gradually reducing, every time reducing 16, the embed dim may cause error as some operation may depend on the embed dim (c.g.. attention heads), thus we will check the forward pass in every attempt, tune it until the size 1. fits the region, 2. the size smaller than the lower bound. smaller, the tuner gives up and reports an error.
Finding gradient accumulation steps: We tune the gradient accumulation steps as it theoretically does not influence the training process compared to batch size, which can also overcome the OOM issue. We tune it with a fast, test training of 10 gradient steps on wikitext-2, we start from 1 and iteratively double until no OOM error triggered.
Once the tuning is completed, the tuned model and parameters are passed to the trainer for the next steps.
Trainer and Evaluator checks and tunes from the Auto-Tuner, the trainer launches the training and reports progress to Weight & Biases | We use the Eval framework trained models. Once training is complete, the trainer passes the model to the LM-Eval, which then automatically runs the evaluations and returns the report.
We use Huggingface trainer to train the model. Once model passes
(3 to evaluate the downstream performance of
httpp::/uandb.ai
33

---

component
description
Grad norm monitor Loss monitor
If the grad norm is too high (> le4). The the loss exploding le4) vanishing I 0)
Step time monitor
If the step time is too high (around 10 times) com- pared to a reference GPT model with the same scale.
Exception handler
Monitoring if there is any error occur throughout the verification process.
Table 8: Auto-Tuner components.
Runtime Checker As presented in 8 we apply runtime checker monitor the entire verification process, specifically, we monitor the gradient norm, loss, and step time for every training step, besides, we catch any error that occurs during the whole process, if any of these problems are caught, the verification process will be terminated and the design will be recorded and marked as erroneous in this V-Node and will not be selected in this node verification. As some errors happen due to environmental settings unexpected situations in the node (c.g., being preempted, connection lost), only if a design marked by more than three V-Nodes as erroneous, will be recognized as an erroneous design. Table Table@reports reports the runtime error rate of different evolution setups.
B.1.5 Additional Details of Designer Agent
Self unit tests and debugging assistance We force the agent to generate at least one unit test for each unit implementation, the unit tests are decorated with @gau_test test for being able to be detected by the checker. The checker will run the unit tests and catch any output and results, then bring them back part of the check report. We also encourage the agent to write assertions and assistive prints to help it debug the code; all outputs will be caught and returned to the agent.
Hybrid foundational models Instead of choosing fixed foundation model for each agent (i.e., Proposer, Reviewer, Planner, Coder, and Observer), decide on distributions of models for agents (e.g., 0.7 for GPT-4o and 0.3 for Claude-3.5 Sonnet); a different agent may have different distributions. Before design task, the models for the agents are randomly sampled based on these distributions.
Internal Unit and Proposal Search As discussed in $4.2 we allow the reviewer and observer to search from the previous proposals and units check for self-replication. We store all the proposals and unit codes along with the documentation in a library that can be queried by comparing the cosine distance between the embedding of the query proposal/unit code and the items in the library; the ones with the shortest distances would be returned. In addition, we also return the sibling proposals that based on the exact same parents, with the query, if any, randomly select most two siblings.
B.2 Pseudo Code
In this section, we provide the extended algorithmic details of our designers and different components of Genesys.
34

---

Algorithm The Design Process (DESIGNMODEL)
Input: Evo Tree (the current evolutionary tree of designs), Librury (reference library)
Output: proposal (high-level design proposal), implementation (the final LM code)
Function Propose Svoofreee, Library):
0: for
to Kattempts do
0: π + None // no proposal yet
0: for
MAX ROUNDS do
0:
<- PROPOSER. PROPOSHR.SEARCHEEE SEARCHADDRRRNNRA EvooT ree, Library, π)
0:
end for
0: P None
// no review yet 0: for 4- MAX RRONDS do
0:
ree, Library, p)
0: end for
0: if p.rating 2 THRESHOLD then
0: return (T. P) accept proposal
0: end if
0: end for
0: raise "Failed to propose valid design"
Function DesignModel( EuoTree ree, Library):
0: proposal Propose( EvoT ree, Library)
0: implemenlation + IMPLEMENT( veepoeeppooo
0: return (proposal, implementation)
Algorithm 2 Compose GAU Tree To GAB Code
Input: GAUTree (the hierarchical GAU structure)
Output: gabCode (a string or code object representing the final composed GAB)
Function CCtpg//eeTTooodeCCAATTeTeeeee
0: let gabCode + AATTATTTORRROTTAAAACCCTTTTTTT
//Start with a minimal GAB that calls
the root unit
0: let units t TopolegicaOrier((AAUTTe)) // or any valid traversal for sub-units
0: for each unit in wnits do
0: if' wnit. immpleeenee = True then
0: gabCode codeOf(unit)
0: else
0: gadCode += placeholderCode(unnt))
0: end if
0: end for
0: return Code
35

---

Algorithm 3 Viterbi-style Search Implementation with GP
Input:
Parents (one or more parent designs, or empty), Proposal (the high-level design), Evo Tree (for references / re-use)
Output: A valid GAB implementation (GAU tree + code)
Function Implement( Evo['ree, Proposo,))
0: if Parentss = then
0:
GAUTree + CopyOf( Parents (0])
0:
uninopleeented <- Propossa..SeectedSSuuii // for MUTATION
0: else if Parents)] then
0: GAUTTre + NEW GAUTREEEeMPpPt root)
0:
unimoplemented < CAUTTee.root) for CROSSOVER
0: else
0: GAUTree + NEWGAUTREE((mmpl root) // from DESIGN FROM SCRATCH
0:
wnimplemented - [GAuUThee.root]
0: end if
0: while unimplemented # do
0: (plan, unit) t- PLANNER GAUThee', unit, Proposal)
0: for tries to Kfutts do
0: GAUTrec' t CopyOf(
0: if Parentss > 1 then
0: reusePool <- ALLCGAUUNITS(PPreenl
forceReuse True else 0: reusePool RECOMMEENDREUSEEUUTTSZZZOOOO wnntes unit.description)
0:
0:
0: forceReuse + False
0: end if
0: (code, children) < CODER GAUTree' plat,,lan, Proposal, reuscPool, reuss
0: if MF-FORATTHECCKER (code) then continue // retry if formatting failed end if
0:
0:
0:
GA CAUTree' Cuurree'.apddaeeeee code, childven)
0:
gabCode +- COMPOSETOCCDEE GA CAUTree")
0:
if FUNCTIONALITYCHEEKK (gabCode))
OBSERVER( (oode,, GAUTree', Proposal) A
then
0:
GAUTTre GAUTTee' II accept changes unimplemented. extend children)
0:
0: break // proceed to the next unit
0: end if
0: end for
0:
end while
0: return GAUTree
36

---

Algorithm 4 Quadrant-Basee Selection with Scheduler
Input:
Evo Tree (set of designs, each with fitness & confidence), Mode (e.g., Design or Verify).
pRondom)() (t) (scheduler probability of picking from initial seeds), paaplore (probability of choosing exploration quadrant), K (size for top-K selection), (current iteration or time step) Output: A selected design from Evoeree
Function QuadrantSllect Evol're reee, Mode, pExendom pEzplore, K):
0: comment // Possibly pick from initial seeds (random)
0: sample r + Uniform(0.1)
0: if pRandom(()) then
0: return RandomlyPickSeed((en
0: end if
0: comment 2. Compute medians for fitness & confidence
0: Food f Median d.fitness
d E Evo Tree
0: med f Median ddconfidence d E Evo Tree)
0: comment 3. Partition into quadrants
0: Q1, Q2, Q3. Q+ 0
0: for each design d in EvoTree do
0: if (d.fitness 2 (med) and (d.confidence 2 Cmed):
0:
(d)add(d) // good confident else if (d. (d.fitness 2 Fmed) and (d.confidence < (med):
0:
0: Q2-add(d) // good & confident else if (d.fitness < Fmed) and (d.confidence 2 Ciedd) )
0:
0:
(d)) // poor & confident
0: else
0:
addddd(d) 0: end for
// poor & not confident
0: comment 4. Choose quadrant(s) based on the Mode
0: sample r Uniform(0,1)
0: r S pExplore:
0:
Q Q3 if Mode "Design" else Q4
0: else
0:
++ 1--15 if Mode "Design" else Q2
0: end if
0: comment 5/ Top-K style selection with small prob picking from outside top-K // or by other priority
0: BoorDDesceedinggyFiinses
0: chosen 0 we may pick multiple or just
0: for + 1 to K do Uniform(0..
0: sample
0: if' < a // small e.g. 0.1
0:
// pick from outside top-K in Q
0:
d + RandomFrom [K:))
0:
else
0:
// pick next from top- K portion
0:
d < Q 0: end if
0:
chosen.add(c)
0: end for
0: return chosen
37

---

Algorithm 5 Ladder of Scales Budget Control and Scale Selection
Input:
usedBudgets (dict: scale - used verifications), verifyBudgets (dict: scale + total budget), Evee (for usage info), design (the current design verify)
Output: A chosen scale, or None if none is available
Function AssignLooSBudgetsi usedBudgels, verifyBudgets ): comment // 1. Identify the lowest scale, e.g., "14M"
0:
0: lowestScale --
w... lowestScale Used + susedBudggts(lowesSSeee
0:
0:
comment Initialize local data structures
0:
loSBudgets - { // Desired usage at each scale
0:
Budget < (
0:
loSBudgets LuSBuudgesslowwesssaeell <- lowestScallesst Used [lowestScale] if toSBudgets lowestScalel] > BurreyBBuuddssslwesSSeee
0:
BuuulableBudgellloweeeee 0: else
0:
+- 0
0: availableeBudgeelllwwwssse +-1
0: end if
0: comment 3. Sort scales ascending (e.g.. 14M < 31M 70M)
0: scales t SORTEDSCALESS (vrrEe(vvrfyBuugger))
0: comment // 4. Build usage for higher scales with selection ratio
0: for from to (len( scales) 2) do
0: currScale scales i]
0:
nextScale + scales + 1]
selectRatio verifyBudgets /ne.ea/dddeeeeeessleeleee (exx/calee] verifyBuugeers(uurSSatee
0:
0: loSBudgets nextScalee + int (osssaeesssccurSsaaeee selectRatio)
0:
if LoSBudgets nnextccale] PerifyBudgets/iee (nxttSclee
0:
pvailableBudgeetnexxSSS + 0: else
0:
availableBudgge//eeeteedd +- IoSBudgers/nextSeate BeedBBudgets/nexxSeull
0: end if
0: end for
0: return availableBudget
Function SelectScale( design, EvoTree, verifyBudgets );
0:
comment Gather usage info usedBudgets + Evo Budggeegg
O:
0:
availableBudget <- Assignt.oSBudger (us.BuggsssseddBugger
verifyBudgets)
0:
comment // 2. Find which scales the design has not verified
0:
verifiedScales t- Evo Tree-get Sese wew.geeeeriiedScaleeeeeeeee (desig)))
0:
unverifiedScales ALLSCALES( (venulf((vrryybbee (verifyBudgess) \ verifiedScales comment Among the available, pick the lowest unverified
0:
0: candidateScales { E availableBudget, availableBudge((s] 0} n unverifiedScales
0:
if len (andiiatteScules)
0:
return None
0: else
0:
return FINDLOWWSSTSSAALlCCciitdrr&o.
0:
end if
38

---

Algorithm 6 Genesys Evolutionary Loop
Input:
BvoTree Tree (population of designs), Library (reference library), Budgels (per-scale verification budgets)
Output: Evo Tree updated with new designs and verification results
Function Design( EvoITree, Library):
0: (seed, refs) - SELECTSSEDPAREENTS (EEESSSEEDDPPEESSSEEEEEEEEEE
(EvoTrre,, Library) Librrr))
0: (proposal, impl) +- DESIGNMODDEL(sedd, refs)
0: EvoTree. www.rraaddeeeegipppppooocccc impl)
Function Verify(Evoo) ree, Budgets):
0: (design, +- EXPERIMENTER AMPRRMANTBRSSLCCCTOREEEE.EE.EESE Budgets)
0: if scale None then
0: return // no budget left
0: end if
0: report (UNTRANEEVAL scale)
0: Eeeorre..pdatcFiine sereeuupdttciinsssessseepe report)
O: Pudgets[scale, < Budgets(scale]
Function Evolve(Eroo) EvoTree, Library, Budgets):
0: while s with Budgets(s]
do
0: async Design( EvoT ree, Library)
0: async Verify( EvoT ree, Budgels)
0: end while
0: return EvoTree
import
import torch. as nn from model, discoveery model. utils modules inport GAUBase, gau_test, UnitDecl
YOU CAN IMPORT KORE NODULES HERE
# YOU CAN DEFINE MORE CLASSES OR FURCTIONS HERE #
class UnitHane (GAUBaSe):
10
FILL THE DOCSTRING HERE
11
def .__ini__ (self, enbed_dim: int, block_loc: tuple, kvarg_all: dict, davice-Rone, dtype-None :ekaagg=): YOU CAN ADD MORE ARGS#
12
13
wactaryykvarg. super () ...0....im init.. (enbed_din, block_loc, kwarg_all) DD NOT CHANGE COMPLETING raise XotInplleeetedError
("davice": device, "dtypa"; dtyps]
14
HERE
16
17
18
YOU CAN ADD NORE FUNCTIONS HERE
19
20
_forvard (self, *+2):
21
THIS CODE MUST BE COMPLETED YotImpleneatedB:ror
22
23
24
WRITE YOUR UNIT TEST FUNCTIONS HERE
25
0gau_test DO NOT CHARGE THIS DECORATOR #
26
def unit_test (aaat_nnaeddavieellle ASSERTIONS PERFORM TEST, PRINT raise MosIaplenentedError YOU MUST IMPLEMENT THIS FURCTION
dtype-None) KEEP THE ARGS
27
28
30
# DECLARE ALL CHILDREN GAUS HERE (EITHER EXISTING OR NEV)
31
CHILDREN OIITDRRE_.ECLAARTTOON
DO HOT RENOVE THIS LINE
32
UnitDecl
33
unitn unitname=*", Hane of the child GAU requirenents-"" Requirenente # inpute-(), List of argunent names outpats-[] # >,
34
35
argunent
37
38
... ADD NORE CHILDREN GAU DECLARATIONS HERE ...
39
]
Figure 11: Code template for creating new block units.
39

---

B.3 Program Templates and Base classes
In Figure 3] we simplify the presentation to involve just a single Pytorch module GABBase. In our system implementation, we have separate classes for units (Fig. 12). called GAUBase, in addition to GABBase, which is specific to full blocks. We use two classes since units and full blocks have slightly different initialization and constraints. Importantly, however, both modules share the same type structure. Below we show the code template for each, as well as the code template for full autoregressive models called GAMBase (standing for Generalized Autoregressive Model Base, Fig. 13 In Fig. II we show the template that our system fills in when designing new units.
class GABBase (un Konuled
*an Base class for Generalized Autoregressive Blocks a. def __init__ (self, enbad_dim: int, block_loc: tuple): super in__init__.0
self, enbed_din eabed_din self blf.bbooc__oo block_loc (layer_idx, a_block)
def _sor, 1, .+2):
raise YotIuplenentedd:ror
10
11
(self, **2):
12
ForFord pass of the model"*n
assert le seapee
3, "Input shape must (B,L,D)"
14
assert d. - self. embed_dim. "Input shape must be (B,L,D)" self, if isinstance(Y, tuplo):
15
.+Z)
16
17
Y, z_ ===
18
19
2_ -) assert Y.shape shape, "Dutput shape must (B,L,D)* assert isinstance (2, dict), "Z must be dict* upudate return Y,
21
22
23
24
25
GAUBase (nn. Module): (on.uul));
26
Base class for Generalized Autoregreseive Units
..init_.. embed_dim: int, block_loc: tuple, kearg_all: dict): super () init........
28
29
self embed_din enbed_din self.block.p kelr.aaar.all kearg_all kearga of all units in GAB
30
block_loc (layer_ _ix n_block)
31
32
33
def _forvard(eelf, x, .+2): XotImplamenteddrrro
35
36
def (erlf x, ++Z):
37
(t.(t.shapa)
'Imput shape ())
38
aasert x.ahape[-i] - self embed,din, "Input shape must be (B,L,D)" -parane signactee (f
forvarrd) parenetere
40
fk: for k, in 2.itans
_parana)
42
Y self forvardo **_2)
43
ieinetance tuple):
44
45
else:
2_
47
assert yhapa =- X. shape, "Output shape must be (B) aasert isinstance (Z., dict), " must diet z. update (2_) return Y,
48
49
50
Figure 12: Base classes and Pytorch modules for model units (GAUBase) and block designs (GABBase). While both are functionally similar, the GABBase.
40

---

(nn Module):
211 Generalized Autoregreseive Modelg
def -_init_. (self, d_modal: int, n_block: int, sacab_size: int 60277,
norm.epsilon: float
device dtype None):
self, self.factory_
("device"; device, "dtype": dtype)
super _...t))tt___.
self.d_noded d_model sebs.enbedding an. (vicaab.ai d_nodel, f=aely.fatott_.keeee
10
block_config geb_conlig()
12
self. blocks - on KoduleList
debddedddddndddodde block_ in_,,dd,,,nll block), devicesdavice, dtypazdtype, *sblock_cont'g
14
15
for layer_idx in range (n.block)
16
})
17
self norm out no. LayerNor (d_model, ops-norn.opailon, factory.
18
19
(self, input_ids):
21
staten_atates self self. enlf..
22
intermediate_ vare ( for block self.blocks
23
24
hidden_states, intermediate_ vara - block(
bidden_states, **iaternediato_vvr
26
hidden_stand staee_stttes_t meli.norn_ent (tiddenn__nddd bidden_states
27
28
29
30
class CAB (AB
31
def -.init... (self, anbed_dim: int, black_loc: tuple, device-Rone dtype- **vargg))
33
factory kuargs {("derice': device, "dtype": dtype)}
super (). . init.. (enbed_din, block_loc)
35
relf.root ROOT_ kwarg.all-kwarge wwfactory.kverge, **kwarge)
block_,oe-blokk,to,,
36
37
38
daf _foorarr x, *+2):
I, self (X, +*2) return x, Z
40
Figure 13: The code template for full autoregressive models.
41

---

B.4 Mutation and Crossover
(A)
RAWSS
(B)
Marmaa
Serch Wufioitii
Lirg
DOLeyUirer
Leptieen
Examples of the mutation (A) and cross-over (B) operations used in Genesys over GAU trees to create new designs. In A, a variant of the RWKV6 block created by replacing the KeRV6Feedforward unit (red) with new unit Spectral wecectrrlldappiii SelaptiveRiVVeAteetiiio (blue). In B, a new block created via novel combination of units the Mamba2 and GPT blocks.
Figure 14:
B.5 Design Selection
Confidence vs Design Rating Quadrant
Good Design Low Confidence
Good Design High Confidence
1000
800
200
Design Rating Rank
Design Rating
600
400
200
Media
Poor Design
Poor Design
-00
Low Confidence High Confidence
Confidence
Figure 15: How do we select imput designs for evolution? visualization our quadrant system used for design selection where designs in the evolution tree (points) are scored and ranked according their fitness or aggregate empirical performance (Design Rating) and confidence (i.e., number of model scales at which design verification has been performed).
42

---

B.6 Discovery Console
Design Artifact Viewer
cacopploge Naccy sttero daggisss
Hemezero
Proposal for streamretnet
streamretnet
Nevdes lod
Selected Unit: MultiScaleRetention
Class Wicttcalleetteteeee GAuban):
Jairgg Coeccaat dispers
Figure 16: A sereenshot of our discovery console available at /ttttps://geeess allen. ai/ that can be used for running full system and viewing discovery artifacts. Here we show details of the StreamRetNNetMLP design (SRN in Table 4 using the Viewer tab (top), which shows the original proposal and review (drop-downs in middle) and the agent-authored authored GAU code design with concrete unit-by-unit implementation details (tree and code on bottom).
C Experiment Details
C.1 Experiment setups
C.1.1 Training Corpus
Studies in small-scale LMs Eldan & Li (2023); Abdin et al. (2024) : Allal et al. (2024) and high-quality datasets Penedo al. (2024) demonstrate that LMs converge faster more educational datasets with fewer training tokens compared to the lower-quality ones, and the smaller LMs can outperform larger-scale ones with high-quality data. This motivates us create small, high-quality educational corpus that allows us to train the models with fewer tokens required, thus improving the verification efficiency. We build our corpus upon SmolLM Allal al. (2024), a high-qualit, dataset for training nigh-performance small LMs. It is hybrid of subsets, and we filter samples from each of them.
Subset
Ratio
Tokens
Train
Test
Eval
Hud
70.0%
24.25B
23.67B
0.299
0.29B
Cosmopedia-v2
15.0%
5,20B
5.08B
60M
60M
Python-Edu
8.0%
2.83B
2.75B
40M
40M
Open OethWebaath
5.5%
1.90B
1.86B
20M
20M
StackOverflow
1.0%
0.4B
388M
6M
6M
DeepMindMath
0.5%
0.2B
194M
3M
3M
Total
100%
34.78B
33.94B
0.42B
0.42B
Table 9: The statistics of Smool.M-17/--orppn The ratio of each subset in the mixture and the
number tokens.
43

---

Firstly, we filtered samples with score >= 4 from Fine WeeeWebedu edu Penedo et al. (2024). which rates the educational level of each sample from to Then, we filter other subsets while keeping the same mixture as SmolL .M. Python-Edu also provides a rating of samples, and we set cutoff of 3.65 to keep the mixture. All other subsets (Cosmopedia-v2, Open OpenWehMath Paster et al. (2024), DeepMindMath Saxton et al. (2019). and StackOverflow ( do not provide the sample rating: thus, we randomly select from them. This results in an around 1/8 high-quality subset SmolLM Corpus, which we call SmolLM-1A 1/rrpus Statistics presented in Table ?
Following Gao et al. (2020), we randomly sampled IGB of data from the original SmoILM for each of the test and eval sets, respectively, then removed any verbatim from the training set.
Customed LM-Eval We customized the LM-Eval framework to allow it to accept the GAB models and add the hooks to the verification time checkers. Moreover, we optimized its evaluation speed overhead in LM-Eval during results processing to the stage where
for our tasks. We recognize some results are not cached and are recomputed. This may be due to some tasks having their internal processing, which makes such caching generalizable. We manually add this result caching to our benchmarks, which improves the speed of the verification process.
C.1.2 Hardware Environment
Our experiments are carried out mainly machines used as the V-Nodes, including three machines with 8 Nvidia A6000 48GB vRAM GPUS with 124 Cloud CPUs and 512G RAM, and five machines with 8 Nvidia L40S 48GB vRAM GPUS with 256 Cloud CPUs and ITB RAM. Two machines with 3 Nvidia A6000 48GB vRAM GPUS with Cloud CPUs and 254.3 GB RAM are configured as D-Nodes. All machines are running on Ubuntu 20.04. The D-Nodes may execute multiple design threads, while V-Nodes always occupy all resources for one verification thread. We dynamically maintain ratio between the Design thread and Verification thread to be around 2 to based on our analysis on $ E.4.2 In addition, implement our backend based on Firebase
of 10 machines from our internal cluster. There are
C.1.3 Model Experiment Settings
We train all models with single 8 Nvidia L40S machines introduced above for the discovered model evaluations in [5.3 While the discovered models are trained with the same settings from the evolution experiments, we train baseline models with their official implementation or implementation from architectures. Specifically, we use FLATT community framework for state- spa.tee---ppaee//mmaaaaa for training both Mamba2 and the Mamba2 repo (https: //github. com/ GPT models, the official RWKV repo model, and the official TTT implementation (https://g) thub. com/ ttt-lm-jax) for the TTT model; For RetNet, apply its FLAA implementation. We apply the hyperparameters from their model card in Huggingface with the same scale and directly find them from their official GitHub repositories papers.
com/
com B1 inniLi RWWV for the RWKV7
tramningaiinn/////
httper//iieeaae.gguglee.co
com/
44

---

C.2 Evolution Parameters
C.3 Verification
Knowledge Engine
Result limits
Embeding models
arXiv
PwC
$2
RefLib 5
Paper DB cohiere-embedd-engiiii--3330
openai-ext--mbcoddnn-----arg
5
Proposal
Perplexity settings
Unit code colere--mbbdd-english--33.3
Model size
Max tokens
Embeding distances
Large
4000
Proposal Cosine
Unit search
Unit code Cosine
Cut-off
Top-K
Paper DB Rerank ratio
0.5
0.2
Proposal search
Cul-off
Top-K
Siblings
0.5
Table 10: Detailed settings for knowledge engine.
Designer Agent
Verification Engine
Model Dist
G40
C35
01p
OIM
Context Len.
2045
Proposer
0.15
0.35
0.3
0.2
Optimizer
AdamW
Reviewer
0.2
0.3
0.25
0.25
Tokenizer
Llama-2-7b-hf
Planner
0.06
0.32
0.32
0.3
LR Sheduler
Cosine with min Ir
Coder
0.0
0.25
0.5
0.25
Min Ir rate
0.1
Observer
0.15
0.2
0.1
0.55
Warmup ratio
0.02
Rating Thresholds
Batch size
0.5M tokens
Reviewer
Observer 3.0
I Learning rate
4.0
14M
le-3
Max Retries
31M
1e-3
Max Search
Proposal
Debug
Retry
70M
1e-3
5
125M
6e-4
Table II: Detailed settings for Model Designer Agent.
Table 12: Detailed settings for verification engine.
Verification Engine and Knowledge Engine Table 10 and 12 showcase the VE and SE configu- rations, with specific training settings for varying scales detailed under VE. In SE, "Results limits" specify the count of items retrieved from external resources and the paper vector database ("RefLib")). For unit and proposal search, elaborated in & B.1.5| a "Cut-off" is set to exclude items beyond certain cosine distance, followed by selection of "Top-K" samples; in proposals, additional siblings with the same parent as the query design are included. We report the embedding models and distance measures used across various modules. the Paper Vector DB, Cohere reranker employed: initially retrieving K/r items (with 0 < r < as the rerank ratio), it reranks them and selects the top-K items.
Designer Agent and Selector & Experimenter The detailed settings for the quadrant selection in Selector and Experimenter (components for design selection designer and verifier, respectively) presented in Table 13 and [1 "Quardtile cutoffs" means the position to divide fitness and confidence ranks, 0.25 means the upper 25% designs the rank are regarded as good/confident. Crossover and design-from-se scratch operations can only be selected after their "Warmup rounds"". The number of parents decides which operation to perform, and this number sampled from the distribution in the table. We sample references from different types, the items in the reference library with and without reference code, and the previous designs. For different types, we randomly sample with predefined number.
45

---

Selector Experimenter
Quartile Cutofis
Seed Distribution
Fitness
Confudence
GPT2 0.3
0.25
0.25
RetNet
0.15
Warmup Rounds
TTT 0.15
Crossover
Design from Scratch
RWKV6
0.15
20
Mamba2
0.25
Num. Parents Distribution
Restart Scheduler
(Scratch)
(Mutation)
2 (Cross.
Restart Prob
Anneal.
0.05
0.75
0.2
0.05
10
Num. Reference by Type
Exploration
Reference
Ref. w/ Code
Prev. Design
Top-K K Noise
Quad.
2
2
2
0.05
0.15
Table 13: Detailed settings for Selector Experimenter.
Besides the Quadrant selection introduced in random start mechanism with a probability Pre gradually annealing by linear schedule that decreases from to Prs after Kys rounds, randomly revisits the initial five designs to foster diverse directions and prevent convergence on narrow paths. The restart is scheduled by a scheduler from 1 to the final restart probability. The "Seed Distribution" determines the probability of the seed designs to be selected when sampling from the seed tree in the restart. "Top-K Noise" decides the change to sample from non-top choices in the Quadrant Selection. "Quad." stands for the probability of sampling from the exploration pool other than the exploitation pool. The foundation model of each designer agent component is randomly selected by distribution "Model Dist." of different kinds of models. In "Max Rounds", the maximum times to query the knowledge engine ("Max Search"), fail by Reviewer ("Proposal") or observer and symbolic checkers ("Debug"), " and retry to implement proposal ("Retry") are provided.
D Additional Results
D.1
Experiment Results on 125M
Blimp
Wnli
RTE
WG
CoLA
SST2
WSC
IS
Mrpe
avg.
Random
69.75
43.66
52.71
48.78
50.00
49.08
49.82
50.03
31.62
49.49
GPT
83.22
57.75
56.32
51.93
50.05
50.44
52.75
51.79
59.07
57.04
Mamba2
86.13
59.15
58.11
50.28
50.69
$1.83
55.68
50.43
62.99
58.37
RWKV7
79.16
56.34
52.17
49.09
50.06
51.26
52.01
52.47
68.38
56.77
RetNet
79.16
56.34
54.15
49.64
52.78
50.69
$2.38
$1.92
56.37
55.94
TTT
81.76
54.93
48.01
50.04
51.14
51.61
50.92
52.24
60.29
55.66
VQH
90.27
57.75
$3.79
49.96
50.85
49.08
50.92
48.57
$4.66
56.21
HMamba
77.05
60.56
59.74
52.09
50.50
53.33
$2.75
51.12
68.38
58.39
Geogate 85.96
54.93
54.15
50.36
50.21
50.92
52.01
51.79
68.63
57.66
Hippovq 92.09
46.89
53.43
50.12
50.07
50.34
49.08
53.17
68.38
57.06
SRN
79.05
61.97
53.43
49.88
50.21
49.89
52.01
50.63
54.41
55.72
Table 14: Performance of human designs and Discovered Models on Various Benchmarks (125M
Parameters, 25B Tokens). Metries indicate accuracy percentages, with bold highlighting the best performance and underlined indicating the second-best best. Italics denote outlier performances.
Table presents the experiment results for 125M models discovered designs and human designs, all models are trained with 25B tokens with the same setting of 350M experiments in $5.3 The discovered model group achieved the best results out the benchmarks while also obtaining the highest average among all designs.
46

---

D.2 Sensitivity of Metrics
0.35
0.3
Neg. MDD
Improvament
Sharpe Ra
0.25
Volada
Fall wo
0.16
Full
0.0
0.08
-0.2
-0.3
-0.05
0
10
20
-00
40
30
70
50
100
Population Size (Step Size = 25)
Stop Size (Populattion Size 50)
Figure 17: Left: How the selection of population size affects metrics, fix step size of 25. Right: The impact of step size metries, fixing population size be 50. We present the difference of metrics between "Full" and "w/o Exp." for the first 500 designs.
We study how the selection of population size Sp and step size k; impact the metrics of the evolution experiments presented in {5.1 We fix the Sp 50 and k, = 25, the settings in our experiments, respectively. Then change the other one from I to 100 and compute how the metrics are affected. The present negative MDD and volatility, respectively, them the higher the better, like other metrics, and scaled for better readability. For the population size, besides the initial unstable region when about Sp 20, which is smaller than the step size, the "Full" system consistently presents an overall advantage compared to the "w/o Exp.". For the step size, despite high variances, the advantages stably persist as well.
results are presented in Fig. 17 where
make
E Analysis of Genesys
To obtain better insights into how to perform optimal evolution with Genesys and how to build more efficient evolutionary systems and discovery agents in general, provide detailed analysis of our full evolution experiment here.
E.1 Analysis of Evolution
We analyze the model design sessions that occurred during the evolution process. more insights by analyzing the entire EvoTree.
Then we obtain
47

---

E.1.1 Analysis of Design Sessions
1,000
Category
Failed (Implementation) (3.6%)
800
Failed (Proposal) (2.7%)
Failed (Verification) (9.7%)
Implemented & Verified (84.0%
Frequency
600
Figure 18: Distribution of the end states of design sessions. "Implemented & Verified" marks suc- cessfully valid design, while brackets for "Failed" mark the stages when design fails.
Implementation attempts
Figure 19: Distribution of number of attempts for implementing design proposals. Mean: 2.65, Median: 2.00, Std: 1.36.
Statistics of End States Fig. 18 shows the distribution of the end reasons of design sessions, which shows a full-prre full-process valid rate of 84.0%. Despite the high-quality model design agent reducing the design-stage error rate as low as 6.3%, there are still around 9.7% designs that end up not being able to be successfully verified. Resulting in 16.0% invalid designs, which is an unignorable waste, while verify-stage errors can be varied and unpredictable (e.g., some errors like divergence happen at a late stage of training verification).
1,500
1200
1.800
7.00
1,400
6,00
1,30
Irequatcy
Frequrnc
5,000
Freqancy
4.00
300
200
Num Impl Rounds
Num-CodingPP Round
Distribution of the Figure 21: Distribution of the number of rounds that imple- number of coding steps dur- ment units during each imple- ing each implementation round. mentation attempt. Mean: 3.17, Mean: 3.68, Std: 1.86, Median: Std: 2.32, Median: 3.00
Figure 20:
Figure 22: Distribution of lines of generated code in each coding step. Mean: 92.74, Std: 33.99, Median: 88.00.
5.00
Statistics for Implementation Stage Fig. 19 shows the distribution of attempts made to implement a design proposal. Most proposals can be implemented with 2 attempts. Fig. 20 presents the distribution of the number of rounds in each implementation attempt. Most attempt needs only one round. This is because we allow the agent to implement multiple units under a subtree at a time, while in mutation mode, which is the most frequent operation in our experiment, only one subtree needs to be implemented. Fig. [1] shows the coding steps in each implementation round, including the initial code generation and the later debugging steps with symbolic checker and observer feedback. takes a majority of 5 steps to implement a unit, showing the difficulty of generating single valid unit. can be exponentially more difficult to generate the whole block with the same complexity, and extremely hard to generate in single shot without VS. Fig. 22 visualizes the number of lines of code produced each coding step. With VS, the agent only needs to focus on the correctness of an average of around 93 lines of code every time, which largely reduces the complexity.
48

---

60
Frequency
50
18
8
Total cost
Figure 23: Total design cost distribution. Mean: 28.63, Median: 16.90, Std: 29.49.
80
Frequency
Frequency
50
22
20
Proposal
Implementation cost
Figure 24: Proposal stage cost distribution, Mean: Figure 25: Implementation stage cost distribution.
6.09, Std: 7.66, Median: 2.68.
Mean: 22.54, Std: 28.80, Median: 10.48.
Design Costs The statistics of the total cost are shown in Fig. 23 It takes 28.63 dollars on average
the designs in our experiment. The proposal stage takes a relatively low portion the cost as presented in Fig. 24 while the implementation stage incurs high cost as shown in Fig. 25
100
140
500
120
400
Frequency
Frequency
300
200
Proposal mating
Proposal rating
Figure 27: Proposal ratings from 01 preview reviewer agent (o1 preview 2024-09-12) Mean: 3.64, Median: 4.00, Std: 0.72.
Figure 26: Proposal ratings from all reviewer agents. Mean: 3.91, Median: 4.20, Std: 0.62.
49

---

Prequrtyy
200
Proposalrating
Propocd osing
Phoporal
Figure 28: Proposal ratings from Ol mini reviewer agent (o1
Figure 29: Proposal ratings from Claude 3.5 Sonnet re-
Figure 30: Proposal ratings from GPT-4o reviewer agent (gpt-4o 2024-08-061 Mean: 4.22, Me- dian: 4.20, Std: 0.13.
mini 2024-09-12). Mean:
viewer agent (claude-3-5 sonnet 20241022). Mcan:
3.78, Median: 4.10, Std: 0.72.
4.23, Median:
4.20, Std: 0.11.
Proposal Ratings Fig. 26 shows the distribution of all the ratings given by the reviewer agent. LLM agents have tendency produce encouraging outputs regardless of prompting that promotes strict and harsh outputs. a result, we take 4 as the passing border in our experiment. The rating also differs by model type. The OpenAl 01 produces more diverse ratings shown in Fig. 27 for 01-preview and Fig. 28 for 01-mini respectively, compared to GPT-4o in Fig. 30 and Claude 3.5 Sonnet in Fig. 29 which concentrates their rating in small band of few numbers. Especially,
all ratings given by Claude 3.5 Sonnet are above 3.0, while GPT-4o produces only different ratings.
0.25
(centeree)
Frequency
0.10
timestep
677
Riness, iinvss_strutiicattionn
Strutificatioon
Fitness
Figure 32: Proposal rating-Timestep Correlation. Correlation coefficient: 0.14
Figure 31: Design fitness Distribution. 0.61, Median: 0.60, Std: 0.04.
Mean:
0.15
Fitress (centere))
0.10
0.05
400
60
30
100
120
140
160
120
stratification
[centered]
066-0.55
0.65-0.7
H0.6
HQ.7
0.00
0.25
0.10
Figure 33: Caness-Cost Correlation, Correlation Figure 34: Proposal rating-Fitness Fitness Correlation. Correlation coefficient: 0.04
coefficient: 0.04
50

---

(A)
(B)
RwRve
Mambaz
RobNot
Mamba2
RatNat'
GPT
RWKVE
Figure 35: The evolutionary tree (first 300 designs) for full system (A) and system without experiment verification (B) (i.e., designs are selected randomly without fitness) shown with the five starting seed designs that drive discovery (GPT, TTT, Mamba2, RWKV, RetNet).
Fitness Fig. 31 shows the fitness of designs, which is nearly normally distributed. Fig. shows the fitness by time steps, where we sort the designs by their timestamp of sampling and take the ranking the step. The fitness is slowly improving over time, with positive correlation coefficient of 0. 14. We further analyze the correlation between fitness and design cost in Fig. 33 the correlation coefficient is near zero, though positive, showing that putting improve the design quality. Fig. 34 presents the correlation between proposal rating and fitness, similar design cost, has a low but negative correlation coefficient, which shows the agent rating not precise and that the rating can not effectively show the quality of a passing design. Despite low correlation, the design provides some guarantees on the quality of design, and the reviewer serves as a filter for the novelty and quality, which cannot be directly reflected simple serchmark-base fitness; moreover, the low-quality designs with low rating that are already excluded from this analysis may have lower fitness.
design cost does saliently
allows to apply our VS-based implementation process, which already
E.1.2 Analysis of Evolutionary Tree
Degre
Libayy
Figure 36: Evolutionary tree Figuue 37: Evolutionary tree of Figure 38: Distribution of node human designs from the reference"w/o Exp." evolution configura- degrees in the different configura- tion. Showing the first 300 nodes. tions.
library.
EvoTree Visualization We visualize the evolutionary tree human designs from the reference library. whose edges are citation relations in Fig.
36 Al-discovered designs from the "w/o Exp."
setup Fig. 37 and the "full" evolution in Fig. 10 Right. The connection patterns of the "full" evolution and "woo Exp." one that does not use fitness-based selection are largely different, where "w/o Exp." " is more randomly connected and "full" shows hubness pattern. can also shown from the distribution of node degrec in Fig. 38 that the reference library shows even distribution
51

---

of degrees, the degrees of "w/o Exp." are more concentrated in lower degrees, however still more even than the "full" evolution which has few ratios of degrees besides 0, and 2.
# node
# edge
Dens.
Deg.
Deg. std
# Com.
Max
Com. size
Library
297
582
6.62
3.92
3.5
5.04
29
55
10.24
Full
1454
1670
0.79
2.30
10.9
3.42
93
269
15.63
w/o Exp. 848
913
1.27
2.15
5.65
13.9
Table 15: Evolutionary tree statistics. "Deg." represents degree. "Density" measures the connectivity of the network, the more edges, the higher density; "Alpha" is the fitting parameter of power- distribution p(æ) x x- with degrees. "Com." means Louvain communities that detect node clusters with similar features.
Evo'Tree Statisties We analyzed the statistics of the EvoTrees of the three settings respectively in Table 15| The reference shows a higher density than the discovered EvoTrees; however, notice that the edges in the discovered tree are only the parent nodes, while the random references are not included. "Full" shows a high standard deviation of degrees, which is due to the hubness. A lower alpha in "Full" also shows that it is closer to long-tail distribution, which also results in its uneven community size distribution. The human EvoTree is closer to the random "w/o Exp." one, which leans more toward explorations, while the "Full" shows centralized pattern that focuses improving based on the known good designs.
E.2
Analysis of Agents
F.2.1 Analysis of Foundation Models
We analyze how the selection of foundation models may impact fitness and the overall cost, and explore the most cost-effective selections. We use abbreviations of models: C35 represents Claude 3.5 Sonnet 20241022, G40 represents GPT-4o 2024-08-06, 01P and D1M stand for 01 preview 2024-09-12 and 01 mini 2024-09-11 12 respectively.
Cost efstetictteeess
Cost effeecivvveess
0.52
0.02
Fitness
10
8
8
8
Proposal agont
Proposal
Proposal agent
Implagent
Impl agont
Impl
Figure 39: Cost-Effectiveness (fitness by cost) of proposal and implementation stage agents (Proposer and Coder). We plot the proposal or implementation stage costs, the fitness of the produced design, as well the cost-effectiveness of different proposer and coder models, respectively.
We analyze the cost-effectivenes of foundation models in different stages in Fig. 39 Proposals generated from GPT-4o and O1-mini show slightly higher fitness of the final design, while the low cost of GPT-4o makes it the highest cost-effective choice for the proposer agent. However, empirically, we observe that GPT-4o sometimes generates preservative designs that directly apply existing known structures, such as simple variants of Transformers, which may guarantee better fitness with a loss of novelty. However, for the course of scientific discovery, sometimes we prefer to perform more explorations on highly inspiring designs with a cost of lower fitness in the current setting. Thus, a mix of multiple agents for better exploration of design ideas is still necessary, and a manual check of proposal quality is important for practical deployment. For the implementation stage, we not include a GPT-4o agent as it sometimes uses a simplified implementation that deviates from the original proposal to pass the checkers. The O1 mini and Claude 3.5 Sonnet show great cost-effectiveness,, reflecting their promises on coding ability, especially, the Ol mini presents advantage in fitness for its implemented designs.
Cost- fost-Effectiveness of Proposer and Coder
52

---

0.04
Fitness (centered)
Cost effectivenes
0.03
Cost
0.02
0.01
0.00
C35
(35-00M
CB-OUP
640-33
GAD-01M
G40-0UP
03M
QUN-O3N
036-038
OIPC3S
01P-0IM
01P-0IP
C35
(3500MM
C3 0SP
640
G40-0IM
640 01P
01W-C33
OIM-DIM
031-03
OIPC3S
01P00M
01PO1P
CB5-35
(35-0IM
C35-0IP
640
640-03M
640-01P
01M-01P.
OIP
01P-0IM
01P OIP
Proposal Impll agent combination
Proposal- agent combination
Proposal Impl agent combination
Figure 40: End design fitness, design costs, and Cost-effectiveness analysis for the Proposal-
implementation stage agents (proposer and coder) combinations.
Proposer-Codes Combinations We then analyze the proposer and coder for integrity and explore the effectiveness of different combinations in Fig. 40 The combination of GPT-4o and O1-mini shows a fitness advantage despite our observation preservative designs as discussed above, while the O1-mini-O0 -nlii-Ol-a -imi combination also shows an above 0.04 centered fitness. The most cost-effective combinations are Claude 3.5 Sonnet with Claude 3.5 Sonnet or O1-mini, while the most expensive O1 -orevvveww does not show an advantage in our tasks.
0.0
Finren perreent
Cont eoteeteeiveeess
09
CD CO
CA G/
60000
GIM CS
OP Os
Oi Gi
EUPOMM
Figure 41: End design fitness, proposal cost, and cost effectiveness for proposal and reviewer agent combinations.
Proposal-stage Analysis analyze the combination of proposer and reviewer agents in Fig. 41 The combinations of the Claude-3.5 Sonnet and Claude-3.5 Sonnet O1-mini still show good cost- effectiveness. GPT-4o reviewer with an O1-mini performance also performs well. The O1-preview combinations show the lowest cost-effectivenees with middle fitness score.
53

---

Cost effectiveness
C35-C35-C35
C35-64
CB
C3-C
CB-0DM-640
CB-00M
C35 C3P-
CI50116-00
01P-00M 03M
010
C3901P-112
G40 600.C55.0.
GID C35
G4D 03M
610 01P-60
GIO D3M
610-01P-01P
OIM DM-CCSCC
01M GIM
OIN-C35-01P
CIMOIMC3
OIM OIM 640
03M-00040 03M-003--00
OIM 01M 01P
01M-0IPCCI5
01M-019-01M
CDM-1P-012
DJP-C3--355
01P C35-640
DIRCIS-0JM
DIP-CIS-01
DIPOIM-C35
OUP-OUN-GR0
01P (DPO D3M
O1P-01N-01P
01 01P-01--35
0IP-1I-6A0
01P-11P03M
G4D
G4D
V'anner-Coder-Obeee agent combination
Figure 42: Cost-effectiveness of Coaner-Coder-Obeeee
combinations.
Implementation-stage Analysis
We study the cost-effectiveness of combinations of planner, coder, and observer in Fig. 42 The GPT-4o planner with O1-mini coder and GPT-4o or Ol-mini observers shows the best cost-effectiveness that surpasses all other combinations by least 30%, showing great coding ability of O1-mini. Claude-3.5 Sonnet also shows a good coding power that all combinations with effeeecfffii above 20 have either an O1-mini or Claude-3.5 Sonnet as the coder.
Conclusion Our analysis suggests that the Claude-3.5 Sonnet suits the proposer. GPT-4o, O1-mini, and Claude-3.5 Sonnet are good choices for reviewers. The O1-mini or Claude-3.5 Sonnet are good coders that be combined with GPT-4o or Claude-3.5 Sonnet planner and observer.
F.2.2 Analysis Implementation Errors
1,500
1,400
1,200
Frequency
1,000
Frqueecy
800
600
400
FLOPs high
Forward enror
diverged
diveriigggggg
Loss not decressing
Not cousal
diffrrntaablee
Training too slow
Unmoecurable
Unparsable
Dipe
format
Shape
Found Module
Multipl GAU found
Unosecubbble
Whong GAU return
Jot. natfound
forward found
Model
Whong GAU
Unused
Functional type
Figure 43: The distribution of functional error types. The designs with fetal format errors may skip the functional checks.
Format error type
Figure 44: The distribution of format error causes.
We analyze the causes of the design's failure in symbolic checkers. Fig. 43 shows the distributions of functional error types while Fig. 444 presents the format error types. 91.0% erroneous designs have functional errors while 17.3% have format errors. Besides the fetal formats errors such as large portion of functional errors are incurred in the forward
not found GAU implementations. pass, model init, and incorrect tensor shapes in operations, causality and differentiability are also challenging while one common cause of differentiability from our observation is due to the redundant parameters that are not used in the backward passes. frequent format error is declaring children in the declarations while not instantiating them in the unit body. Unexecutable code is also common, while including multiple GAUs a single file, which we require make them separate, and passing wrong arguments to the GAU, such as missing templated args, are frequent.
54

---

E.3
Analysis of Discovered Models
E.3.1
Analysis of Verification Process
25
SCARO
Scal
330
133
3,0000000
20,0000000
380,000,000
Figure 46: The training loss by number of train- ing tokens. Each point marks trained model.
Figure 45: The training loss by the number of parameters models. Different colors mark different designs.
Training Loss We show how the training loss changes under different scales in Fig. 45 We can observe an overall trend the training loss decreasing by scales, which shows that most designs follow scaling laws Kaplan et al. (2020). Fig. 46 shows another view by comparing training losses and training tokens. The training losses present a decreasing trend as the training token increases.
std
mean
min
max
rand
#
max/rand>0.05
#b>200
mrpe
0.1465
0.4929
0.3113
0.6863
0.3162
408
Truc
True
0.1112
0.5011
0.3675
0.6329
0.6318
40430
False
True
wnli
0.0593
0.5017
0.3239
0.6479
0.4366
71
True
Falsc
blimp
0,0566
0.7044
0.5777
1.0
0.6975
65093
True
True
rte
0,0288
0.5064
0.4224
0.5884
0.5271
277
True
tiny TruthfuuAl
0.0185
0.5233
0.3967
0.5469
0.4866
100
True
False
wsc273
0.0179
0.5005
0.4322
0.6117
0.4982
273
True
True
inverse inverse_scaling
0.0141
0.4992
0.3868
0.5462
0.5003
29548
True
True
cola
0.0138
0.4979
0.4512
0.5452
1043
True
sst2
0.0138
0.4997
0.4495
0.5573
0.4908
872
True
True
qa4mre_2011 2011
0.012
0.1549
0.1167
0.2583
0.15
120
True
False
winogrande
0.0103
0.495
0.4538
0.5359
0.4878
1267
True
True
mnli
0.01
0.3432
0.3256
0.3564
0.3545
9815
False
True
sciq
0.0088
0.1998
0.0
0.219
1000
True
True
mnli mismatch
0.0084
0.3428
0.3255
0.3568
0.3522
9832
False
True
mathqa
0.0078
0.1984
0.1792
0.2352
0.202
2985
True
True
qnli
0.0073
0.4995
0.4746
0.534
0.5054
5463
True
True
qa4mre 2012
0.0073
0.2578
0.1938
0.2812
0.2562
160
True
False
qa4mre_200 2013
0.0069
0.1699
0.1479
0.1972
0.1655
284
True
True
openbookqa
0.0066
0.2664
0.244
0,286
0.258
500
True
True
are are_challe challenge
0.0054
0.2861
0.227
0.3012
0.2841
1172
True
True
tinyGSM8k
0.004
0.006
0.0055
0.078
0.0055
100
True
False
piqa are easy hellaswag
0.0036
0.5005
0.4908
0.5125
0.5022
1838
True
0.0033
0.2627
0,2508
0.2727
0.2618
2376
False
True
0.0011
0.2611
0.2504
0,2653
0.2603
10042
False
True
squad_completion
0.0011
0.0012
0.0
0.0047
0.0
2984
True
True
swag
0.0008
0.2539
0.2466
0.2566
0.254
20006
False
True
lambada_opendl
0.0004
0.0
0.0
0.0128
0.0
5153
True
True
triviaqa
0.0001
0.0
0.0
0.0002
0.0002
17944
False
True
tinyGSMSR
0.0
0.0055
0.0055
0.0055
0.0055
100
False
False
Table 16: Statisties of evaluation benchmarks results from all verifications performed in the full
evolution experiment, Sorted by the standard deviations.
55

---

Benchmarks analysis Table 16 presents statistics of evaluation results from the full evolution of each benchmark well the standard that select the benchmarks in the discovered model evaluation experiments in [5.3 We first focus on the standard deviations, which show "learnability" of task in our setting. If models perform similarly, then means
cannot be effectively learned. We consider only the ones with std >> 0.01. We then analyze the ratio between the best result and the random result. the best improvement less than 5%, we skip the task as is too challenging. Finally, more sufficient to provide a comprehensive evaluation. The only exception WNLI; we preserve firstly it has high standard deviation, and secondly, it is widely used benchmark that shows good evaluation quality.
consider the number of examples with 200 cut-off, which we regard
Benchmark Correlations Fig. 47 further analyzes the correlation of the selected tasks. Most tasks show low correlation with each other, with few pairs, inverse-scaling and blimp, rte, and wnli showing relatively higher negative correlation with coefficients -0.5. For the positively correlated tasks, WSC and cola show correlation coefficient of 0.35 while all other tasks are S 0.25. shows low overall correlation between tasks, which implies our task selections have a low overlap in the tasking topics and can evaluate different parts of the underlying abilities and potential of the design.
Task Correlation Matrix
1.00
mmpc
..2
.0.12
0.08
0.03
0.25
0.26
0.06
0.02
0.75
wili
0.2
0.11
00.
0.24
-0.15
0.25
-0.06
0.03
blimp-
-0.12
0.11
0.1
0.09
-0.52
0.11
-0.18
0.07
0.00
snerrsesccaiig...
0..52
-0.25
cola
-0.26
0.25
0.11
-0.16
0.35
-0.12
-0.2
0.06
-0.30
522
0.06
0.06
00.18
00.02
0.32
0.11
-022
0.05
-0.75
winogrande
-0.02
-0.03
0.07
0.03
0.09
-0.03
-0.06
-0.05
-1.00
mppc
bimp
me
09223
meeese scaling
cola
222
Figure 47: The correlations between the nine selected tasks based on the evaluation results.
56

---

E.3.2 Analysis of Model Units
830
430
700
Frequency
500
Froquencyy
400
230
194
123
346
Number Units
Usage
Figure 48: Number of units per design. Mean: 4.59, Std: 1.61, Median: 5.00.
Figure 49: Number of usages for the units. Mean: 5.47, Std: 34.48, Median: 1.00.
800
80
700
70
600
Frequency
Frequency
500
400
200
100
624
017
025
020
677
675
0.
626 0.88
Reuses
Units
Figure 50: Number of times being adapted to Figure 51: Ratio of reused units in designs. Mean: reuse units for the units that have been reused. 0.21, Std: 0.30, Median: 0.00. Mean: 3.10, Std: 5.06, Median: 1.00.
Unit Usage and Reuses Fig. 48 presents the number of units contained in designs, and Fig. 49 shows the statisties for the number of times a unit gets used. Some common units are frequently used, such as the RMSNorm and RotaryPositionn PosttionaallEmmmeddng With those existing building blocks, makes it easier build relatively complex solutions with around median number of units in design. Fig. [5]ssieer the case of unit reuse, which specifically refers to the case of adapting an existing unit new scenario with code modifications as compared to the direct usage above, which does not need to change the code. Fig, further presents the ratio of unit reuses. Model reuse provides a flexible way to reduce the complexity implementation and the accumulation of experience. The dictionary of units from previous designs is provided to the agent for access with embedding-bar based search analogous to deep learning frameworks that simplify novel model constructions by improving code reusability.
more blocks, which the
57

---

2
Figure 52: Unit network. The edges represent reuse relations between units.
E.3.3 Unit-Performance Relation
We explore the predicativity of model or task performance by unit. By using a Bag-of-Wordd (BoW) as simple unit-based descriptor of model architecture, which shows underlying con- stitutions of design (for example, GPT can represented as GPT MHA GatedMLP RMSNorm RocaryPositiionalEmeeddaaa we study whether it can be used to predict the model performance.
58

---

F-Fuld
LOOCV
Support
Training
Training
blimp
0.817 (# 0,004)
0.720 (# 0.032)
0.734 (t: 0.002)
0.633
684
291
34
inverse unveree_scaling
0.768
0.671 0.034)
0.743 (# 0.001)
0.655
310
openbookqu
0.621 (4 0.007)
0.481 (= 0.019)
0.618 (a 0.001)
0.413
157
466
338
99
challenge
0.805 0.010)
0.787 0.020)
(+ 0.001)
lambada openai
0.998 (2 0,001)
0.989 (z 0.002)
0.997 (2 0.000)
0.999
1059
mathya
0.684 (# 0.017)
0,524 (= 0.025)
0.675 (4: 0.001)
0.52
79
551
389
41
0.704 (+ 0.004)
0.548 (= 0.036)
0.648 0.001)
0.535
192
253
571
44
mali
0.672 (# 0.003)
0,532 (= 0.030)
0.572 0.001)
0.351
172
154
333
401
winegrando
0.753
0.605
0.709
piga
0.661 (4 0.011)
0,524 (= 0.050)
0.677 (+ 0.001)
0.542
87
593
365
15
qa4mre
0.826 0.004)
0.763 0.016)
(+ 0.000)
0.746
mali mismathh
0.655 (a 0,032)
0.475 (= 0.054)
0.547 (# 0.001)
0.331
168
224
325
343
guli
0.777 (# 0.007)
0.658 (= 0.042)
0.759 (+: 0.000)
0.656
714
312
hellaswag
0.837 (+ 0.002)
0.780 0.023)
0.832 (+ 0.001)
0.775
229
831
soig
1.000 (# 0.000)
1,000 (= 0.000)
1,000 (+ 0.000)
1.0
1060
44P
0.629
0,443
easy
0.749 (# 0.010)
0.614 (= 0.031)
0.704 (+ 0.001)
0.573
25
341
641
53
ga4mre 2013
0.654 0.006)
0,510 (= 0.030)
(+ 0.001)
0.436
ss2
(2
0.543 (= 0.033)
0.699 (+ 0.001)
0.485
538
469
mipe
0.631 (+ 0.01))
0.431 (+ 0.028)
0.561 (+ (+000)
0.288
278
190
245
347
swag
0.775 (+ 0.008)
0.700 0.048)
0.781 0.000)
0.68
730
wali
0.654 (2 0.004)
0.497 (= 0.049)
(2
0.405
297
311
425
27
gadmre 2011
0.005)
0.762 0.016)
(2
wsc273
0.679 (+ 0.008)
0.557 (= 0.036)
0.691 (.00))
0.538
82
592
354
32
tinGGSMKk
0.986 (+: 0.002)
0,990 0.004)
0.983 (+ 0.000)
0.99
1049
squad_ completion
0.801 (+ 0.009)
0.695 0.018)
0.707 (+ 0.000)
0.621
679
188
25
cola
0.733 (t 0.01
0.658 (= 0.028)
0.710
0.643
183
691
168
Table 18: Naive Bayes on tasks, Fl scores are presented. Support is the total number of each
segment; the values are divided by the 4 evenly spaced segments between min and max. Using even segments instead of quartiles to avoid repeated values. Excluded the triviaqa and tinyGSM8K, which have min and max too close.
Fitness
Precision
Recall
F1
Support
Precision
Recall
F1
Support
Training set (80%)
Test set (20%)
0.6-0.65
0.817
0.597
0.690
298
0.528
0.264
0.352
72
0.65-0.7
0.647
0.733
0.688
45
0.706
0.750
0.727
16
<0.6
0.778
0.930
0,847
471
0.635
0.871
0.735
116
>0.7
0,938
0,441
0,600
0.000
0,000
0.000
Accuracy
0.783
848
0.623
212
Macro avg.
0.795
0.675
0.706
848
0.467
0.471
0.453
212
Weighted avg.
0.791
0.783
0.774
848
0.580
0.623
0.576
212
5-fold CV F1
0.785 (+: 0.004)
0.658 (+ 0.024)
LOOCV F1
0.786 (+ 0.002)
0.658
Table 17: Use BoW of units to predict the fitness with Naive Bayes.
BoW to Fitness We divide fitness into four levels, then train Naive Bayes (NB) class over the BoW representations predict the fitness level; the results are presented in 17 In the test set, it achieves an Fl in Leave-One-Our Cross Validation (LOOCV) of 0.658, which shows some predictivity. While a BoW model and an NB classifier are simple and naive setups, we show that it is possible to predict design performance with its architectural components, which can be used to guide the model design process.
BoW to Task Performance We then use the same manner to predict each downstream task performance Table [8 tasks show a test LOOCV higher than 0.5, which indicates the predictability from unit to certain downstream abilities. This shows the possibility of providing fine-grain unit-level guidance for designing models with certain targeted downstream applications (i.c., what kind of unit combination may help to improve the performance in certain tasks).
59

---

E.4
Analysis of System and Performance
E.4.1
Training Time Estimation
14M
31M
70M
125M
14M
31M
70M
125M
14M
31M
70M
125M
Optimistic
Median
Single run (6)
46
123
498
3258
174
437
1338
8778
302
751
2178
14298
Token mult.
50
40
30
20
50
40
30
20
50
40
30
20
Num. runs
400
150
1000
150
40
1000
400
Total GPU has
255.6
218.7
249
289.6
966.7
776.9
669
780.3
1677.8
1335.1
1089
1270.9
Est. hours
31.9
27.3
31.1
36.2
120.8
97.1
83.6
97.5
209.7
166.9
136.1
158.9
Est. total
5.3 days
16.6 days
28.0 days
Table 19: Estimation of total Training time in a single 8x A6000 machine. The single run is the training time under training tokens be 20 times of parameters, while the estimated times are based on the training tokens in our setting.
We present the estimated training time in single 8x A6000 machine in Table We train fully optimized GPT model equipped with hardware optimizations like FlashAttentonn in different scales with training tokens to be 20 times the parameters within our setting and record the running time as the Optimistic case to, then a Median case tM with an unoptimized, vanilla implementation of GPT, the Pessimistic case is simple linear extrapolation tp = 2 x tm to We then scale it to training tokens and the number runs by simply scaling up linearly. It provides us with a simple way to estimate of rumning time and decide the budget.
E.4.2 Optimal Pipeline Throughput and the V-D Ratio
Consider distributed pipeline with two types of nodes:
D-nodes:
Responsible for design tasks. There are ND such nodes. In practice, each D-Node can launch multiple design threads in parallel. We assume single threads here for simplicity. Each design task takes an average time of TD to complete.
V-nodes: Responsible for verification tasks. There are Ny such nodes. Each verification task takes an average time of Tv to complete.
design cannot be considered "finished" until has been verified. Hence, the overall throughput of verified designs (i.e., finished products) depends on the capacity of both types of nodes in the pipeline.
Throughput formulation Assume each node works in parallel (each occupying a "lane"), and that work can be pipelined optimally so different nodes start at possibly shifted time points maximize steady-state throughput. The effective throughput θ (the number of verified designs produced per unit time) is determined by the slower stage of the pipeline. In steady state, each design task must be followed by verification task, so the throughput is
ND TO
Nv. Tv
min
This expression states that the system cannot output verified designs at rate faster than the rate of design (Np/Tp) the rate of verification (Nv /Tv) whichever smaller.
Optimal ratio of V-nodes to D-nodes Let r = Ny be the ratio of verification nodes to design nodes. Our goal is to choose to maximize the throughput Q Observe that if
No
ND Nv
TD
Tv
then verification is the bottleneck, and adding more D-nodes beyond certain point does not improve the final throughput of verified designs. Conversely, if
ND
Nv
<
TD
Ty
60

---

then design is the bottleneck, and extra V-nodes do not improve the throughput. Hence, the throughput is maximized when
ND
Nv
TD
Tv
Solving for Nv gives No
r² Tv
(3)
TD
That is, the optimal ratio of V-nodes to D-nodes matches the ratio of the average times required: Tv. In this balanced scenario, both design and verification stages have the same steady-state throughput, and no additional overprovisioning of either D-nodes or V-nodes will further increase θ
TD
Maximal throughput
When r", the stages are balanced, and the maximal throughput is
ND
Nv
+
max
where
Nv = ND-
TD
Tv
This completes the derivation of the optimal pipeline configuration for maximizing the rate of fully verified (and thus valid) designs.
Optimal V-D ratio for Genesys
The expected time of verification jobs for a design can be computed
as
E(Tv) = Σ to tePurrisu(s)
where la is the expected running time for verification scale S is the set of all scales, (u)rif((s) the probability of verify design on scale 8, which is computed as the accumulated selection ratio II 87; from the lowest scale 8.
Based on Table 2 the expected verification time Tv with a single 8 x A6000 node for the optimistic, median, and pessimistic cases is 189s. 566s, and 944s. Notice that it only considers the training time, which underestimates the actual time, we take an approximation of 30% of training time here. Tp The expected design time can be computed as i Err,y where TD is the average agent design time which we take 20 minutes here empirically, and Err is the verify time error rate, which is 8.61% 61% as presented in Table 2 It results in 21.9 minutes of TD. Thus, the optimal V-D ratio for system with a median expectation of verification time is about 0.56. Notice that each D-Node can execute multiple design threads at time, thus each V-Node can be saturated with around 2 design threads. It helps us to decide the total design threads in our evolution setting as discussed in S SC.I
In practice, we take a V-D ratio lower than this number, additional V-Nodes do not impact the evolutionary progress by exhausting the verification budget, which guarded by the LoS, while an inadequate V node may lead to the D-Node selecting parents without sufficient confidence. Due to the risk of connection instability, preemption, or any unexpected down of the machine, the availability of V-Node is not steady, lower V-D ratio increases the robustness of the system.
61

---

Prompts
E1 Proposer
F1111
System and GP Background
System Prompt for Proposer
You are language modeling researcher, your role is to propose a novel autoregressive language model (LM) block design.
## Background
Modern LMs are typically structured as stack of repeating blocks. Each block processes:
1. Input: A sequence of embeddings X of shape (B, L,, D), where:
BB is the batch size.
L. is the sequence length.
-D is the embedding dimension.
2. Intermediate Variables: (e.g., memory, states, caches) passed keyword arguments.
The block outputs new sequence of embeddings Y (same shape as X) and updated intermediate variables Z'. The overall architecture can be represented as follows:
"'yythn
tokens Tokenizer (sentence)
X Embeddings (tokens)
O # Initialized an empty dictionary, updated by each block. for block in Blocks:
X, bXock(X, **2)
output = LXgit=(X) ...
Your goal is to design a proposal for novel LM block that outperforms current state-of-the-
art models, aiming for:
Low perplexity on corpora,
High accuracy downstream tasks,
Robustness to varied inputs.
Efficiency in both training and inference,
Excellent scalability with more data and larger models.
#*#
Generalized Autoregressive Units (GAUs)
Each LM block is decomposed into smaller components known as Generalized Autoregressive Units (GAUs), which inherit from the following base class:
pyyhhn [GAU_BASS "'1
GAU has the following structure:
Input: A sequence of embeddings X and intermediate variables Z.
Output: A new sequence embeddings and updated intermediate variables Z', which can include newly computed values.
GAUs can be arranged hierarchically, with the output one GAU feeding into another. This structure allows block to be represented as tree of nested units, starting from root node.
62

---

Mutation Instruction for Proposer
## Instructions
Your task is improve seed design by modifying one GAU which may have multiple children GAUs, you will need to select one specific GAU in the seed to work on. You can add, remove, or replace existing child units or operations improve it. order make the improvements traceable and make an architecture factorizable which allows further analysis of the elements and factors that lead to better LM designs, we wish improvement you proposed should be "locality" that has controllable step size. Specifically, you are not encouraged to introduce a drastic change to the seed design. Your edit should influence as few existing or potential new units as possible. To itemize:
Top-down approach: Design the GAU from the top down, breaking complex blocks into smaller, manageable units that can be nested together.
Reuse existing units: You encouraged reuse the existing unit. Edit it only when it is necessary for you to perform your idea.
Creativity with constraint: Strive for a design that is innovative yet maintains the overall structure the existing model. Avoid drastic changes that would significantly alter the model's architecture.
Local modifications: Focus on making changes to a single GAU and its potential child GAUs. If your edits have involve multiple GAUs, select the shared root of these units. Ensure that your modifications do not interfere with the correctness of other parts of the model.
Simplicity and implementabilityy Prioritize designs that are relatively simple and feasible to implement. Avoid overly complicated structures that might be challenging to code or integrate.
Design your modifications in a way that allows for gradual tracking of differences across designs, facilitating an evolutionary path of improvement.
Evolutionary approach:
## Task
Here is the seed design for you to improve and some "ice-breaking" references for you to started:
(SEED)
Here the list of GAUs the seed design that you can select from:
(SELECTIONS)
Here are the sibling designs with the same seed, avoid proposing the same design as your siblings, and think of how to make your design unique and better.
(SIBLINGS)
You need to think about which GAU modify and how to improve it based on the instructions above.
Crossover Instructions for Proposer
## Instructions
Your task is to propose a new GAU design by combining multiple parent GAU designs, you will need to reuse the good GAUs from the parents produce better design than both. Your task is to best preserve the good elements of both and discard the potentially bad ones. You are encouraged introduce brand-new units but reuse them from the parents.
63

---

## Task
Here the parent designs includes the units that you reuse:
(PARENTS)
Here the sibling designs with the same seed, avoid proposing the same design as your siblings, and think of how to make your design unique and better.
[SIBLINGS)
You need to think about how to best recombine the parents based on the instructions above.
Design from Scratch Instructions for Proposer
Your task is to propose GAU design from scratch using the information provided.
(REFS)
F.1.2 Search and Refinement
Search Instructions for Proposer
You will start your research proposal process by investigation, ideation, and literature reviews. You have access to a powerful search engine that can query external academic sources (such as arXiv, Papers with Code, and Semantic Scholar), an internal library of research papers, and technical documents. And a web search assistant will collect information from the internet based on your instructions and ideas. You need to perform this process for multiple rounds until you think you have sufficient information and thoughts for you to provide the proposal.
Follow these guidelines in your response:
1. Search Keywords:
Provide up to 3 precise and simple keywords for external source searches. Each keyword
should be precise and specific term.
The keywords will be directly passed to the search frames arXiv, Papers with Code, and Semantic Scholar. The keywords formulation should be based the features of the search algorithms these websites.
Format: " "keywords DURR_KYYORDS'"
Internal Library Search:
Describe the content you want to find in the internal library. - The library uses vector search find relevant excerpts. So the description formulation should consider the features of the cosine similarity vector search algorithm.
Format: <*t description
D
Record Your Analysis:
Clearly articulate your motivation and thought process.
This helps the web search assistant understand and collect relevant information.
The search assistant is a LLM agent, so it will be able to understand your response.
You will need to record useful information and your analysis and thoughts detailed and comprehensive analysis note in your final response for future reference. As all the search results will be cleared after each round and you wont be able to access them again, you must record everything carefully. You need to include those parts in the analysis note:
64

---

1. Summary of your analysis.
All useful references with excerpts.
3. Key insights and detailed analysis that may help you. Future search plan needed or plan of next steps.
5. The list of references, use precise citation style.
Proposal Readiness:
-Include the exact phrase "I'm ready" only when you think you got sufficient information to formulate your proposal, otherwise, never include this phrase. And you will receive further instructions about the next step.
You are allowed propose without adaquate information, your first few readiness may not be accepted.
Do not give your proposal now, the proposal you give will not be considered, you will be able to give your proposal later with further instructions after you say "I' m ready".
Note: The search queries (if any) your responses will be still processed, and passed to you, but you will not be able to access the search engine afterward.
Refinement based on Review Instructions for Proposer
Your proposal has been reviewed by an expert. Please carefully consider the following feedback:
Review: [REVIEW]
Rating: [RATING) out of 5 ((PASS_OM OR NOT))
Suggestions: (SUGGESTIONS)
Based on this feedback, please refine your proposal. You will start your research proposal refinement process by investigation, ideation, and literature reviews. You have access to powerful search engine that can query external academic sources (such as arXiv, Papers with Code, and Semantic Scholar), an internal library of research papers, and technical documents. And web search assistant will collect information from the internet based on your instructions and ideas. You need to perform this process for multiple rounds until you think you have sufficient information and thoughts for you to provide the proposal.
Follow these guidelines in your response:
(... Omitted, identical to the search initial prompt above)
Proposal Output Prompt for Proposer
Here is more search results based on your last response, you will not be able to access the search assistant again after this, so do not include any more search queries in your response:
(SEARCH (ESULTSESSSS
Firtly, provide short model name for your design, like "Mamba", "Llama3", "GPT-4o" and so on. Wrap it in a quoted block like this: "" "mmodel_nen name MOOML_LDDENAE Then, give an abstract of your proposal that describes the core idea of your design in sentence. Wrap it in a quoted block like this: "*< abstract COUR_ABSTRACHS Next, give your proposal in the following structure:
65

---

## Proposal Structure
Maintain and update the following structure in your proposal throughout the process:
1. Title: A concise, descriptive model name for your proposed design. It should be single line level I heading. should also be the only level heading in your response.
2. Motivation: Explain the problem you aim to solve, incorporating insights from your research.
Related Work:
Summarize the current progress and related work based on your Investigation.
Explain how these findings have influenced or validated your design choices.
4. Problem Analysis:
Provide detailed analysis of the problem you're 're addressing. Describe the key concept or philosophy behind your proposed solution.
Provide mathematical or logical arguments for why your design is expected improve model performance.
Discuss potential trade-offs and how they are addressed.
5. Design Plan:
Outline your approach for the LM block design.
Specify the single GAU you't ve chosen to modify (excluding the root unit).
Provide detailed descriptions of modifications and new structures.
Include mathematical formulations and theoretical justifications for your design choices.
Implementation Guidelines:
Provide pseudo-code for the modified GAU and any child GAUs.
Include mathematical formulas necessary for implementation.
Offer step-by-step instructions for integrating the new design into the existing model. Conclusion: Summarize the expected outcomes and benefits of your proposal.
8. References:
List all sources used in the proposal, properly formatted.
## Key Points for Writing the Proposal
Detail is crucial: Your proposal must be clear, detailed, and precise. Do not worry about length; focus on the elarity of your ideas.
Mathematical rigor: Provide mathematical formulations, theoretical justifications, and logical arguments for your design choices. This adds credibility and helps in understanding the expected improvements.
Implementation clarity: Include clear guidelines for implementation, such as pseudo-code, mathematical formulas, and step-by-step instructions. This ensures that coders can implement your design without losing track of the overall structure.
Now please give your final proposal.
Please include the selection of the GAU you will modify. Be sure to wrap the selection in quoted block like this: seeeeection FOUR_SELECTION" And your selection must come from one of (SELECTIONS). Ensure there is one and only one <*< selection YOUR SELETSELTTT quoted block in your response. (for mutation only)
F.2 Reviewer
E.2.1
System Prompt
System Prompt for Reviewer
You are an expert in autoregressive language model research, and you have been asked to review proposal for improving the design of an autoregressive language model (LM) block.
66

---

In this system, the model is composed of smaller units called Generalized Autoregressive Units (GAUs). These GAUs form the building blocks of the LM. The proposal outlines changes specific GAU, and your role is assess the design strategy behind this modification.
## GAU Characteristics
Each GAU has the following characteristies:
sequence of embeddings and dictionary of intermediate variables such as memory, states, or caches.
Input:
A new sequence of embeddings Y and optional dictionary Z' of updated intermediate variables. The updated variables in Z' can be used to modify Z for subsequent units using 'z. update (
Output:
The system builds complex autoregressive model blocks by nesting multiple GAUs. The proposal you are reviewing will introduce modifications one GAU in this structure.
## Instructions for Reviewing the Proposal
1. Conduct Investigations before Reviewing:
Use the provided search functionality to gather information about existing research and implementations related the proposal.
You will be asked to conduct multiple rounds of search if necessary to gather comprehensive information.
In every round of search, you have to record all useful information and your analysis and thoughts in detailed and comprehensive analysis note for future reference. As all the search results will be cleared after each round and you wont be able to access them again, you must record everything carefully.
Assess Novelty and Meaningfulness:
Compare the proposal to the search results to determine its novelty. Evaluate whether the proposal introduces meaningful improvements or innovations compared to existing work.
3. Accuracy, Robustness, Efficiency, and Scalability:
Assess whether the proposed design can potentially improve performance key areas:
Low Perplexity: Can the design help reduce perplexity on language corpora? High Accuracy: Will it improve accuracy on downstream tasks such as text classification or generation?
Robustness: Does the design show potential for handling variant or noisy inputs effectively?
Efficiency: Evaluate whether the design improves efficiency in both training and inference (e.g., faster computation or lower memory usage).
Scalability: Consider whether the design scales effectively, providing better overall performance as the model size and data grow.
Strengths and Concerns:
Identify the key strengths of the proposed design and assess whether they contribute meaningfully to the model's success.
Highlight any concerns, including potential risks, limitations, or weaknesses in the design.
Clarity and Completeness:
Ensure that the proposal clearly explains the design and that all aspects are covered. Identify any missing, ambiguous, unjustified parts, and offer suggestions for improvement.
Theoretical Soundness:
Focus on the theoretical foundation of the proposal. Since empirical results are not expected this stage, evaluate whether the design theoretically sound and aligns with the stated
67

---

objectives.
No Expectation of Empirical Evaluation:
'The current review based on design and theory. You should not expect empirical results or fully implemented model at this stage.
The goal is to ensure that the GAU design is theoretically sound, innovative, and ready for further development and integration into the model.
## Proposal Information
Parent Design to be Modified: [PARENTS]
GAU Selected for Modification: (Mutation only) [SELECTION]
Proposal for Review: (PROPOSAL)
Here are the sibling designs with the same parents, check if the proposal proposed the same design as these siblings, if so, give low rating.
(SIBLINGS)
(TOP K. PROPOSALS)
F.2.2 Final Review
Review Output Instructions for Reviewer
(The search process prompts are skipped, similar proposer's)
Here is more search results based your last response, you will be able access the search assistant again after this, so do not include any more search queries in your response:
SEEARCH_RESULLE
## Review Process
Your review should include:
summary the search results and their implications for the proposal's novelty and
meaningfulness.
An assessment of the highlights and concerns regarding the design.
An evaluation of the design's accuracy, robustness, efficiency, and novelty. Suggestions for improvement, where necessary.
## Rating System
Assign a float value between and based on how well the design meets the criteria above:
1: Poor design with major issues.
2: Not good enough; significant improvement needed.
33 Good design but with room for refinement.
-4: Excellent design, well thought out and near approval.
5:: Outstanding design, highly innovative and strongly recommended.
68

---

You now have comprehensive information about the proposed GAU modification and relevant research in the field. Based on your analysis and the search results, provide final review of the proposal. Your review should address:
1. Clarity: Is the design clearly articulated, with well-defined objectives?
Innovation: Does the proposed modification introduce new and valuable improvements?
How does it compare to existing research?
3. Feasibility: Can the proposed design be implemented successfully within the given framework?
4. Scalability: Will the design scale efficiently with larger models or more data?
Accuracy and Robustness: How might the proposed changes impact model performance and ability to handle diverse inputs? Efficiency: Does the design offer potential improvements in computational efficiency or memory usage?
Provide:
comprehensive analysis the proposal's strengths and concerns.
2. Constructive suggestions for improvements or areas needing clarification.
A final rating (float number between and 5) based on the proposal's overall quality and po- tential impact. Wrap your rating in quoted block like this: "" "'rating YOUR_RAS "OUR_RATIGG" for example: <*< 'rating 2. 2.7 There must be one and only one <<< "'rating YOUR_RATINGE quoted block in your response.
Remember to be objective, strict, and fair. Approve the proposal only if it meets high standards of quality and offers clear value beyond existing approaches.
F.3 Planner
F.3.1
System Prompt
System Prompt for Planner
You are the Implementation Planner for autoregressive language model (LM) research team.
Team Goal:
The team's 's objective is to discover the best novel autoregressive LM block that can surpass existing state-of-the-art models. Success measured by:
Low perplexity corpora
High accuracy on downstream tasks
Robustness variant inputs
Efficiency in training and inference
Good scalability, providing better overall performance with more data and larger models
You are responsible for the implementation phase, collaborating with a coder and an observer to execute a given proposal.
## Background
Modern LMs are typically structured as a stack repeating blocks. Each block processes:
A sequence of embeddings of shape (B, L, D), where is batch size, is sequence length, and D is embedding dimension.
69

---

2. Intermediate variables Z (passed keyword arguments), such memory, states, caches, etc.
The block outputs a new sequence of embeddings Y (same shape as X) and updated intermediate variables Z'. Such block be written as:
" "python (GAB_BASE) ".
And LM can be written as:
"python
tokens Tokenizer (sentence)
X Embeddings (tokens)
0 initialized an empty dictionary which might be updated by
the blocks
for block in Blocks:
X, Z = block(X, *+Z)
output Lxgits(X)
## Generalized Autoregressive Units (GAUs)
GAUs are smaller components that compose LM blocks. They inherit from this base class:
" "'pythn (GAU_BASEL
Key points:
..M blocks can be decomposed into nested GAUs
GAAs share the same interface LM blocks
-GAUs can be arranged hierarchically and nested within each other
1. Proposal Reception:
The coder will receive a proposal to improve an existing LM block design.
2. GAU Selection:
You will select one GAU for the coder to implement or refine based on the proposal.
3. Template Adherence:
Thee coder will follow the GAU template:
<<< "'python
[GAU TEMPLATE)
4. Key Guidelines for the Coder:
Decomposition of Complex GAUs:
If a GAU is complex, the coder can decompose it into smaller child GAUs simplify implementation and testing.
b. Reuse of Existing GAUs:
If an existing GAU meets the requirements, the coder should reuse it instead of re- implementing. The coder encouraged to reuse existing GAUs and declare new ones only
70

---

when necessary.
Implementing Multiple GAUs:
If the proposal involves multiple GAUs, the coder should implement them separately in different code blocks.
Each code block should contain complete GAU implementation following the GAU template.
One code block should implement only GAU.
d. Limited Access to Other GAUS:
When working on GAU, the coder will have access only the current GAU's implementation and its children's implementations, not be able to edit other GAUs.
Code Testing:
'Tee code will be tested by the format checker and functionality checker well the unit tests provided by the coder.
An observer will be observing the implementation process to ensure that the coder is following the guidelines and the design proposal.
## Your Role as Planner
Progress Monitoring:
Review the current implementation status, including which GAUs have been implemented.
Implementation Sequencing:
Decide the optimal order for implementing the remaining GAUs, considering dependencies and priorities.
Detect if there is chance to reuse existing GAUs and point out to the coder.
Task Assignment:
Determine which GAU should be implemented next.
Guidance:
Provide clear instructions to the coder for the next implementation task.
## Instructions for the Planning Process
Review Current Status:
Overview Provided: You will receive updated overview of the implementation progress, including:
A list of units (GAUs) that have been implemented.
Any relevant notes on completed units.
Dependencies between units.
Analysis:
Identify which units are pending.
Understand dependencies and how they affect the implementation sequence.
71

---

2. Decide the Next Unit to Implement:
Consider Dependencies:
Prioritize units that unblock other units.
Ensure that the next unit can be implemented without waiting for other units to be completed.
Assess Priorities:
Focus on units critical the core functionality of the LM.
Consider units that may pose challenges and allocate time accordingly.
Enable Parallel Development:
Where possible, identify units that can be developed concurrentyy by different coders.
3. Provide Instructions to the Coder:
Specify the Next Unit:
Clearly state which unit the coder should implement next.
Include Implementation Key Points:
Provide any specific instructions or considerations for the unit.
Highlight important aspects such input/output specifications, handling of intermediate variables, or any deviations from standard templates.
NOTICE: do never provide any exact implementation details in your response as it may mislead the coder, only the key points that may help the coder.
Mention Dependencies:
Inform the coder of any dependencies that affect the unit.
Specify if the unit relies outputs from other units or if it provides essential functionality for upcoming units.
Communicate Effectively:
Clarity:
Use clear and concise language. Avoid technical jargon unless necessary and ensure it's well-defined. Actionable Steps: Provide instructions that the coder can act upon immediately.
Include any deadlines or time considerations if relevant.
5. Update the Implementation Plan:
Documentation:
Record the decision and instructions for transparency.
Update any project management tools or documentation to reflect the new assignment.
Monitor Progress:
Plan review the coder's 's progress and be ready to adjust the plan as needed.
## Key Guidelines
Alignment with Project Goals:
Ensure that the chosen unit aligns with the overall objectives of improving the LM as per the proposal.
Dependency Management:
Be mindful of the dependencies to prevent blockers in the implementation process.
Efficiency:
Optimize the order of implementation make the best use of the coder's time and skills.
Responsiveness:
Be prepared adjust plans based on new developments or changes in the project status.
72

---

## Additional Considerations
Implementation Guidelines Reminder:
Remind the coder to adhere to the implementation guidelines, including:
Use of the GAU template.
Proper handling of inputs and outputs.
Maintaining documentation standards.
Encourage Reuse:
Urge the coder to reuse existing GAUs when appropriate.
EEror Handling:
Instruct the coder handle missing arguments edge cases.
Future Dependencies:
Mention upcoming GAUs that depend the current task.
Final Notes:
Your careful planning ensures that the implementation proceeds smoothly and efficiently. By strategically assigning tasks and providing clear instructions, you help the coder focus on developing high-qualig units that contribute the overall success of the project.
Remember:
Your decisions directly impact the team's productivity. Thoughtful planning and clear communication are key.
Stay adaptable. Be ready to adjust the plan based on the coder's progress and any new information.
Facilitate collaboration. Your guidance helps coordinate efforts and keeps the project on track.
The following is the proposal improve the seed design by improving selected GAU: (SELECTION). (Mutation only)
## Parent Design Overview
(PARENTS) (Mutation and Crossover only)
## Proposal to Implement
(PROPOSAL)
### Review of the Proposal
[REVIEW]
73

---

### Rating: (RATING) out of
F.3.2 Plan and Unit Selection
Output Instructions for Planner
is round (ROUND) for the design implementation. Please make your plan.
#### Current Design Overview
[VIEW)
#*## Log of Progress
[LOG]
#### GAUs Available for Selection
(SELECTIONS)
Implemented GAUs ([MPLEMENTED)) Can be refined. Unimplemented GAUs ((UNMMPLEMENTED)) Need be implemented.
Note:
(PROTECTED) are protected and cannot be modified. You can only work under the subtree rooted at the selected GAU from proposer's selection.
Reminder: All unimplemented GAUs must be implemented eventually.
|REUSE PROMPT)
Please wrap your selection of the next unit to implement in quoted block like this: " "eeeeccioon " for example: " "seeecttoon GAU GAU_NAME"'. You must include one and only one selection quoted block in your response.
F44
Coder
E.411
System Prompt
System Prompt for Coder
You are the Implementation Coder for a team designing new autoregressive language model (LM).
The goal of the team discover the best novel autoregressive LM block that can defeat the existing state-of state-of-the-an the-art art models, measured in low perplexity in corpora, high accuracy in downstream tasks, robustness to variant inputs, efficiency in training and inference, and most importantly, good scalability that providing better overall performance with more data and larger models. Your role is to write the code to implement the given proposal.
## Background
Modern LMs are typically structured as a stack repeating blocks. Each block processes:
A sequence of embeddings of shape (B, L, D), where is batch size, is sequence length, and D is embedding dimension.
74

---

2. Intermediate variables Z (passed keyword arguments), such memory, states, caches, etc.
The block outputs a new sequence of embeddings Y (same shape as X) and updated intermediate variables Z'. Such block be written as:
"'python
And LM can be written
" "python
tokens = Tokenizer (sentence)
Embeddings (tokens)
Z O # initialized as an empty dictionary which might be updated by
the blocks
for block in Blocks:
X, block(X, **Z)
output = LXgit=(X) "
## Generalized Autoregressive Units (GAUs)
GAUs are smaller components that compose LM blocks. They inherit from this base class:
"'python [EAU_BASE]**
Key points:
.LM blocks can be decomposed into nested GAUs
-GAUs share the same interface as LM blocks
-AAUs can be arranged hierarchically and nested within each other
### Note
GAU is specialized nn.Module:
'The main difference is that GAUBase provides a structured way to handle inputs and outputs, including intermediate variables (Z))
You can define layers and implement logic just like in regular nn.Module.
2. Input and Output structure:
Input: X (tensor shape (batch, seqlen, embed_dimng (ined)ddmm) and Z (dictionary of intermediate variables)
-Output: Y (tensor of same shape as X) and updated (dictionary)
The ffowwrrd method:
This is where you implement the core logic of your GAU. It should take and any needed intermediate variables from 7. as arguments.
It should return Y and dictionary of updated/new intermediate variables.
Nesting GAUs:
You can create more complex GAUs by nesting simpler ones.
In the forward method of complex GAU, you would call the simpler GAUs in sequence, passing the output of one to the input of the next.
5. Initialization:
Use the provided embed dim, block loc, and kwarg all to initialize your layers and set up any necessary parameters.
75

---

### Instructions for the Implementation Process
1. You'll Il receive proposal of novel block design. Implement the GAUs based on the proposal.
3. Follow the GAU template:
" "'pythn SRAU_TEMPLATE,
### Key Design Principles:
Decomposition of Complex GAUs:
If a GAU is complex, you can consider to decompose it into smaller child GAUs to make the implementation and testing process easier.
Placeholder Declaration and Child GAU Calls:
You can declare and instantiate child GAUs in the parent GAU's iinit_ method as placeholders to be implemented later, like:
ppython
self. friilddinstancer
[CChildNames (...)
Call the child GAU the forward pass using this pattern:
"" "python
z['arg1]]
2[" 2['arg2']
Y, 2_ self (child_ instinnnt (x, ++2)
outi Z_ ge. out1', None)
out2 = z_ ge. out2', o'ut2''',,, None)
You can replace Y, Z, and Z_ with other variable names, but ensure the sequences (X,Y) are always shaped (B, D). Ensure all inputs/outputs, other than sequences, are passed via Z and Z_
3. Prepare Inputs and Outputs:
All inputs needed by child GAUs should be prepared advance. After finalizing the parent GAU, you won't ton't be able to modify when implementing the child GAUs. Always retrieve values from Z using (..ett v'vr', Nonee) or other default values to avoid errors. Similarly, when implementing GAU, you should also handle the case if input argument is not in Z or is None.
The system will handle placeholders for declared child GAUs by generating empty classes that accept and Z as inputs and return the same and Z as outputs. Your job to correctly prepare the inputs and manage outputs for each child GAU.
4. Implementation format:
You must include full implementations of units in your final response. You can provide multiple implementations of *different units* including the selected unit and optionally its children. You must wrape each implementation a block quote as follows: " "'python
{full implementation of a unit, unittests decorated with @gau_test, and children declarations}
All implementations must follow the format of the GAU template, and remember to keep the first line as the marker '# GAU LMMPLEMENTATION FILL' to allow the parser detect GAU implementation file. Only the code block wrapped by '** "pythhnn <<t and kept first line ## GAU BAU__MPPLLO IMPLEMENTATION order allow the parser successfully detect the code blocks, DO NOT nest any <<< python
will be considered as GAU implementation. In
76

---

blocks within the code block of unit implementation, c.g., in examples of the doc string, don't wrap the examples with "python " The class name of the GAU will be detected as the unit name of an implementation. Do not define any other GAU classes in this block. And the name of the class should be the unit name you are implementing. you working the root unit, the class name should be the model block name based on the proposal. Notice that it is block as you are implementing a LM block not the whole model. Remember keep the unittests and children declarations of each unit in the same file of the implementation. Do not define any other GAU classes in this block. And the name of the class should be the unit three sections: 1) the unit implementation. 2) the unittests (all unittests must be decorated with @gau_test, otherwise it will be ignored), 3) the children declarations. And always remember to declare children GAUs if there is any in your unit, either new, placeholder or reuse existing ones. Otherwise the linker will not be able find them. You can modify based on the implementations from the provided seed, but you should never simply copy them as your response. you want reuse unit, you can simply declare it in the children list without providing the implementation.
you implementing. In another word, each file must contain
### Implementation Guidelines:
No Access to Other GAUs:
When working on GAU, you will only have access to the current GAU's implementation and its childrens' implementations and not the internal details of other GAUs. Ensure interactions between GAUs are handled through and __
Child GAUs:
When decomposing GAU into child GAUs, ensure that the placeholder instantiation and calls are correct. You can choose to not implement them immediately, and placeholders will be provided. Ensure all input/output interfaces for placeholders properly handled in the current GAU if you choose to implement them later.
Doestring:
Provide docstring for the GAU, explaining its inputs, outputs, and purpose. Follow Py Tyrch''s 's style guidelines, as the docstring will help others understand the GAU's role and how interacts with other units.
Unit Tests:
Write at least one unit test for each GAU. Tests should cover core functionality and edge cases to ensure correctness. After the GAU is integrated into the model, tests will run automatically to validate its performance.
Interaction Between GAUs:
Ensure that all interactions between GAUs follow the defined interface. You will not be able to modify other GAUs besides the current GAU and its children in your response, proper input/output management is essential.
Iterative Design:
You will receive feedhack and go through iterative rounds of design. If your implementation introduces emrors or fails tests, you will need to debug and refine your GAU. The system will guide you through this process with error traces and diagnostics.
Reuse Existing GAUS:
If there is an existing GAU in the provided seed that can meet your needs, you should directly reuse instead of implementing again. You are encouraged to reuse existing GAUs. Declaring new GAU only if it is necessary.
## Guidelines for Designing the GAU:
77

---

Class Naming & Structure: Ensure that your GAU class inherits from 'GAUBase" and is named as specified in the proposal. You should only define GAU class in each implementation. Do not define any other GAU classes this block. And the name of the class should be the unit name you are implementing. If you working the root unit, the class name should be the model block name based on the proposal. Notice that it is a block as you are implementing a LM block the whole model.
If you are modifying based on an existing GAU, DO NOT use the original name, give a new name the new GAU you are implementing.
Ensure all the arguments introduced in the init function of the GAU class have either default value a way handle missing values. If argument is optional, handle gracefully. Missing argument handling is necessary to prevent checker failures unless N None valid value.
Ensure you are referring to the right class names in unit tests.
2. GAU Call Behavior:
-Tee GAU should always be called in this format:
ppythn
Y, z' self. (luntt_inttaceee- (x, **2)
additional inputs are required, pass them through (e.g., Z['ar''] value).
The output Y is always the updated sequence, and Z' contains the updated intermediate variables.
If extra outputs besides Y are expected, retrieve them from Z', e.g.: <** pyython
var = Z' Z'eget Dget(vva ('var', None) "
3. GAU Initialization:
-Always initialize GAU instance as follows:
"python self linstance_namee (unitname) diieeedddim-emee__ddded block_lo==block_
kwarg_all aataggalllkwarre mall=wwargaal. **self paactory_waarge
* **warggalll
If you need pass extra arguments the unit, include them in kwarg ganggl
For example, suppose you introduced two additional arguments, argl and arg2, you can pass them as follows:
"python
kwarg_all, UnitName (..., kwarg.all=kwarg.en
('rggal""ar"g'===... arg2"]=...
**kwarg_all)
4. Embedding & Block Location:
embed dim specifies the input dimension. block Ioc is tuple (block idx, block) that locates the GAU within the network where block_ide starts from 0, allowing you to implement block-specific behaviors (e.g., varying architectures operations between blocks, initializing intermediate variables acrossing blocks in the first block).
5. Module Definition:
Avoid using GAU instances inside nm.Sequential. You can use nnn. or nn.M Dict.
Do not define any MndMe classes in your code. Declare child GAUs instead and do not implement them in your code.
6. Placeholder Management:
Placeholders for child GAUs will be automatically handled by the system. Avoid manually
78

---

implementing placeholders at this stage. You will be prompted to implement them later when necessary.
When declaring placeholders for child GAUs in your implementation, follow the proper syntax and ensure correct input-output handling.
Design Approach:
Name GAUS meaningfully, Each GAU should represent distinct unit with clear function in the architecture.
Follow top-down design approach: if the operation is complex, decompose it into child GAUs and define their placeholders. Ensure each placeholder aligns with the broader structure of the model, ready for future implementation. Or you can implement the children immediately insperate files wrapped by different python code blocks in your response.
9. Be Consistent:
Ensure your implementation(s) remains consistent and fits seamlessly into the overall system architecture. Avoid introducing errors, inconsistencies, or redundant code. Your GAU should operate smoothly alongside other GAUs and should not introduce any deviations from the overall design philosophy.
## Proposal
[PROPOSAL]
## Review
(REVIEW)
### Rating: (RATING) out of
## Implementation Plan
This is the current plan and instructions from the an Implementation Planner in your team:
[PLAN]
As background, the proposal is going to improve the following seed design by im- proving the unit: [SELECTION). And all your implemented unit will be put into this seed design. Please thinking how your code to be work with the existing code. (Mutation only)
(PARENTS) (Mutation and Crossover only)
F.4.2 Implementation and Debugging
Refining an Existing Unit
#### Refining an Existing Unit.:
Below a tree of the GAUs that compose the language model (LM) block you are implementing, and you will continue to implement, and the details of the GAUs:
[VIEW]
Below is the specification for the GAU selected by the planner to be implemented:
Specification: (SPECIFICATION;
79

---

Children list: (CHILDREN)
Current Implementation: [IMPLEMENTATION)
Observer Review: (REVIEW)
Observer Rating: (RATING) out of (Passing score 3)
Observer Suggestions: [SUGGESTIONS)
Please refine the unit based on the information provided.
Implement a New Unit
#### Implement a Newly Declared Unit.:
Below is tree of the GAUs that compose the language model (LM) block you imple- menting, and you will continue to implement, and the details of the GAUs: (VIEW)
#**Н GAU Declaration:
You will start your implementation by implenting the GAU decalred below. Please ensure that your design and implementation align with the details provided:
[DECLARATION]
CIRUSE_UNIT BIEEE-UNT_PIRRRREE PROMPT)
Please start your implementation.
Debugging Instructions for Coder
Your design has undergone checks by the format checker, functionality checker, and has been reviewed by the observer. Unfortunately, did pass. Below is the feedback:
Format Checker: This report assesses whether your code adheres to the required format guidelines.
Format Checker Report: [FORMAT CHECKER REPORT)
Functionality Checker: The functionality checker evaluates two critical aspects:
1. Unit Tests: It executes the unit tests you provided for the GAU to ensure your design works as expected within your own test cases.
2. Whole Model Integration: Beyond testing the GAU in isolation, the functionality checker integrates your GAU into the larger language model (LM). compose the tree of GAUs as the LM block. It generates any necessary placeholder classes for unimplemented units and verifies the functionality of the entire LM, including forward pass, backward pass, efficiency and causality.
Functionality Checker Report:
(FUNCTION SOEROKORRCEEERRRRCCC REPORT)
Observer Review:
Review: (REVIEW)
80

---

Rating: (RATING) out of ([PASS_OR MRASS__RR_AA (R_NOT)
Suggestions from the Observer:
(SUGGESTIONS)
[REUSE_UNIT_ PROMPTT
Please try to fix the code based on the information provided.
E,5
Observer
F,5.1
System Prompt
System Prompt for Observer
You are the Implementation Observer for team designing a new autoregressive language model (LM) based on Generalized Autoregressive Units (GAUs). Your role is to review and provide feedback on the code written by the Implementation Coder, ensuring aligns with the proposal and follows best practices.
The goal of the team is to discover the best novel autoregressive LM block that can defeat the existing tfate-of-the-ar models, measured low perplexity in corpora, high accuracy in downstream tasks, robustness to variant inputs, efficiency in training and inference, and most importantly, good scalability that providing better overall performance with more data and larger models. Your role is to write the code to implement the given proposal.
## Background
Modern LMs are typically structured as stack of repeating blocks. Each block processes:
1. A sequence of embeddings X of shape (B, L, D), where B is batch size, is sequence length, and D is embedding dimension.
Intermediate variables (passed keyword arguments), such memory, states, caches, etc.
The block outputs new sequence of embeddings Y (same shape as X) and updated intermediate variables Z'. Such block can be written as:
" "'pythnn {GAB_BR [GAB_BASE)
And an LM can be written as:
" "python
tokens = Tokenizer (sentence)
Embeddings (tokens)
0 # initialized
empty dictionary which might be updated by
the blocks
for block in Blocks:
x, = block(X, **2)
output Logits(X) "..
## Generalized Autoregressive Units (GAUs)
GAUs are smaller components that compose LM blocks. GAU implementations must inherit from this base class:
81

---

"'pyhhn (GAU_BASE]*'*
Key points:
-LM blocks can decomposed into nested GAUs
-GAUs share the same interface LM blocks
-GAUs can be arranged hierarchically and nested within each other
### Note
1. GAU is just a specialized nn.Module:
The main difference is that GAUBase provides a structured way to handle inputs and outputs, including intermediate variables (Z).
You can define layers and implement logie just like in regular nndMod.
Input and Output structure:
Input: X (tensor of shape (batch, seqlen, embed_dim) and Z (dictionary of intermediate variables)
-Output: Y (tensor of same shape as X) and updated Z (dictionary)
3. The foowwrdd method:
-This is where you implement the core logic of your GAU, It should take X and any needed intermediate variables from Z arguments.
It should return and a dictionary updated/new intermediate variables.
Nesting GAUs:
You can create more complex GAUs by nesting simpler In the forward method a complex GAU, you would call the simpler GAUs in sequence, passing the output of one to the input of the next.
Initialization:
Use the provided embed dim, boock_loc, and kwarg all to initialize your layers and set up any necessary parameters.
## Implementation Process
The coder needs to implement proposal that try to improve an existing LM block design by refining one GAU. Each GAU implementation must follow this GAU template:
""' "'pyton [GAU_TEMPLATE)
1. Decomposition of Complex GAUs:
a GAU is complex, the coder can consider decomposing into smaller child GAUs to make the implementation and testing process easier. The coder can declare and instantiate child GAUS in the parent GAU's init method as placeholders to be implemented later.
2. Reuse Existing GAUs:
If there is existing GAU in the provided seed that can meet the needs, the coder should directly reuse it instead of implementing it again. The coder is encouraged to reuse existing GAUs. Declaring new GAU only if it necessary.
Implementing multiple GAUs:
If the proposal is to implement multiple GAUs, the coder should implement them separately in different code blocks. Fach code block should be complete GAU implementation following the GAU template. One code block should only implement one GAU.
82

---

## The proposal and corresponding review for the design to implement
### Proposal to Implement
[PROPOSAL)
### Review the Proposal
(REVIEW)
#*## Rating
(RATING) of (Passing score:
## Your Responsibilities:
Code Review: Carefully examine the code produced by the Implementation Coder for each GAU. Look for:
Proper declaration and use of child GAUs
Efficiency and performance considerations
Potential bugs or edge cases
Proposal Alignment: Ensure the implementation aligns with the overall proposal.
3. Innovation Assessment:
IIdentffy any novel approaches optimizations introduced in the implementation.
Evallate the potential benefits and risks of these innovations.
Conniddrr how these innovations align with the overall goals of the language model design.
Check that docstrings are comprehensive and accurate, and that unit tests adequately cover the GAU's functionality.
Docstring and Test Review:
Prepare clear, constructive feedback for both the Implementation Planner and Coder. This should include: Identified issues or potential improvements
5. Feedback Compilation:
Suggestions for refinements or alternative approaches
Commendations for particularly effective or innovative solutions
6. Integration and Scalability:
Consider how well this new GAU integrates with existing GAUs in the model.
Evaluate the potential impact on the overall model's performance and scalability. Assess whether the implementation allows for future extensions or modifications.
Code Quality and Potential Issues Identification:
Ensure the code is well- strucurruc, readable, and maintainable.
Flag any potential issues or vulnerabilities in the implementation.
Consider edge cases scenarios that might not be adequately addressed.
-deetiff any parts of the code that might benefit from further optimization or refinement.
8. Provide Suggestions for Improvement: Provide specific suggestions for improving the
code and the design. And provide helps for the coder implement the design.
## Guidelines:
Approach each review with critical yet constructive mindset
Consider both the technical correctness and the strategic value of the implementation Look for opportunities to improve code quality, efficiency, or innovativeness
83

---

Be specific in your feedback, providing clear examples suggestions where possible Consider the balance between faithfulness to the proposal and potential improvements Flag any potential issues that might affect the integration of the GAU into the larger model
Remember, your role is crucial in maintaining the quality and coherence of the overall implementation. Your insights will guide both the Planner in making strategic decisions and the Coder in refining their work. Strive promote design that pushes the boundaries of current language models while ensuring robustness and scalability, as emphasized in the original system prompt.
F.5.2 Observation Feedback
Output Instructions for Observer
#### Current Design Overview:
Below is a tree of the GAUs that compose the language model (LM) block, along with details about each GAU:
(VIEW)
The coder is refining the GAU (UNIT NAME).
As background, the proposal is going to improve the following seed design by improving the unit: (SELECTION). (Mutation only)
(PARENTS) (Mutation and Crossover only)
### GAU Specification and Implementation:
GAU Specification: (SPECIFICATION)
Design Idea (Analysis): (ANALYSIS)
Full GAU Implementation: [IMPLEMENTATIO]]
(REUSE UNIT [UNTPPPOMPT
### Potential Similar Unit Codes from Previous Designs
Check the novelty of the implemented unit by comparing it to the following unit codes (whether is similar or copying) any:
(UNIT_ (UNIT_CODES]
## Format and Functionality Checks
The implementation has undergone checks by the format checker, and functionality checker.
Format Checker: This report assesses whether the code adheres to the required format guidelines.
84

---

Format Checker Report:
JFORMAT_CHECKER CHECKER |
Functionality Checker: The functionality checker evaluates two critical aspects:
1. Unit Tests: It executes the unit tests provided with the GAU implementation by the coder. Beyond testing the GAU in isolation, the functionality checker integrates the GAU implementation into the larger language model (LM). It composes the tree of GAUs as the LM block. It generates any necessary placeholder classes for unimplemented units and verifies the functionality of the entire LM, including forward pass, backward pass. efficiency and causality.
2. Whole Model Integration:
Functionality Checker Report: (FUNCTION SUINCTIO_CCECCERR CHEECER_REPORTT
## Response Requirements
Prepare a comprehensive feedback report including:
Overall assessment (1-5 rating, with being excellent). Wrap your rating in quoted block like this: " "'rating YOUR_ WUU__RATIG"" for example: "'rating 2. 7.7 There must be one and only one << rating TOUR_RATING* quoted block in your response.
2. Strengths of the implementation
3. Areas for improvement and specific suggestions for refinement or optimization
4. Comments
innovation and potential impact and any concerns about integration or
scalability
5. If any of the checks failed above, you need provide detailed analysis that helps the coder
to debug the code and pass the checkes, take this as your first priority if the checks failed.
6. Recommendations for the Coder
Remember, your insights crucial for guiding the Coder in refining their work. Strive to promote a design that pushes the boundaries of current language models while ensuring robustness and scalability. Be sure you include your rating quoted block like "* rating YOUR_RATING"* in your response.
G Qualitative Examples
G.1
Example GAU Trees
G.1.1
Five Evaluated Designs
Below is a brief description of these five new designs adapted from the design artifacts produced during discovery (see Figure 16
VQH:
This design takes inspiration from mamba2 and other SSM models and involves novel selective gating mechanism and vector quantization technique. This allows for efficient memory compression and dynamic information flow. also integrates hierarchical memory in the style Lee et al. (2025); Wu et al. (2024). which aims to enhance contextual understanding.
HMamba [HierarchicalMabbbll] This design variant Mamba2 that integrates hierarchical state space modeling (inspired by (Bhirangi et al.) [024] Qin et al.) 2023)) with double-layer Mamba architecture. This modification aims to enable long-range range dependencies to be handled more effectively. This structure supports efficient processing of extended sequences without significant computational overhead.
This transformer variant replaces the standard multi-head attention with a new attention mechanism called Geometric Gated Multi-head Attention that aims to address certain positional biases in standard attention. This architecture also supports robust feature extraction and nuanced contextual representation.
Geogate
HippoVQ. recurrent architecture based on Gu al. (020aa) that employs event-driven scale selection and hierarchical polynomial memory (extended with vector quantization),
85

---

optimizing memory usage based on event importance. The adaptive scale integration mechanism ensures that relevant information prioritized during processing.
SRN (StreamRet.Ne
This design expands RetNet and includes Multi-Scale Retention mechanism and StreamRetNerMLP mechanism for efficient streaming inference. Inspired by Xiao et al. (2023), such mechanisms aim to balance memory management and computational efficiency. This design aims to be particularly effective for real-time applications that require rapid processing of incoming data streams (see Fig. T6p.
wnd LingarAttent Attentrnttnn
AdaptiveRUSNore
ASIntegra
aator
dierarcch Pranni
GGraphConoolutio
FastiTIlines
anar
EventDetecto
Lis
RCCcchelliragger
Hierarchical RMSNorn
MetaLearner,
HiPPoProjector
GatedT TLinear Hierarchice wwd.aaallappaaoo Eivettentttoonnio 10E Eiceenntttteii
Figure 53: What kind of new units does our system produce? Word cloud of the core terms in the proposal documents indicating the kinds of mechanisms being developed.
Figure 53 shows word cloud with the names of the different units and their frequency during discovery. Further details of designs can be found at hictps://genesss allen. ai/ As shown in Figure [6] this link provides live access to our design console that can be used to view design artifacts and experiment details (e.g., links to the original training runs in Wandb).
Mamba2
RetNet
GrometricGatedMMHH
RMSNorm
StreamRetNattMLL
Hieracchiallaambalayerr
RMSNorm
MultdScaleRetentioon
GatedMLP
RettayycaatinnaEEmeediiggg
SSDMinimalDiscrete
TileManager
AclaptiveKVCache
Figure 54: Geogate-GPT
Figure 55: Hierarchical-Mamba
Figure 56: StreamRetNet
HipPOvQ
HPMemory
EDSSModule
ASIntegrator
SelectiveGatingunitt
HeerachicalMeenoyynntegraa
VectorQuantization
EventDetector
ScaleSelector
HiPPOProjector
Figure 58: VQH
Figure 57: HiPPOVQ
86

---

Beyond quantitative metrics, the architectures discovered by Genesys exhibit distinct structural features that contribute to their performance:
VQH (Fig. 58):
Utilizes Selective Gating and Vector Quantization, allowing for efficient memory compression and dynamic information flow. The integration of Hierarchical Memory Integrators facilitates enhanced contextual understanding.
Integrates hierarchical state space modeling with double- layer Mamba architecture, enabling the capture of long-range dependencies effectively. This structure supports efficient processing of extended sequences without significant computa- tional overhead.
Hierarchical-Mamlle (Fig. 55):
Geogate-GPT (Fig. 54): Combines Geometric Gated Multi-Head Attention with Gated MLPs and Rotary Positional Embeddings, enhancing attention dynamics and positional encoding. This architecture supports robust feature extraction and nuanced contextual representation.
Employs event-driven scale selection and hierarchical polynomial memory, optimizing memory usage based on event importance. The adaptive scale integra- tion mechanism ensures that relevant information is prioritized during processing.
HiPPOVQ (Fig. 157
Implements Multi-Scale Retention and StreamRetNenMLLP for efficient streaming inference, balancing memory management and computational efficiency. This design is particularly effective for real-time applications requiring rapid processing of incoming data streams.
StreamRetNer (Fig. 56p:
These architectural innovations not only align with current state-of-the-art designs but also introduce novel configurations that push the boundaries of language model architecture design. The diverse structural elements across the models demonstrate Genesys's capability to explore and optimize a wide range of architectural paradigms, tailoring each to excel in specific aspects of language understanding and generation.
G.1.2 Complicated Designs
AdapaietiicccchcaMMenoTTT
McmHierBlod
Hlerarchicallaye
Residua
ResourcaAllcaaoe
AcaptiveSteneAemmaer
Selectioellpdater
HaraachhialSubaayyrr
rirearbiclllappiiettteeiiin
Me MemorryManager
Dynomic.ayerMoom
SwIGluMPP
RotaryPPss dineen
2t
PageelkltentinnCehhe
BlookwtsePrccessrr
MemorySSate
Maribalayer
RMSNorm
Figure 60: MemHierGPT
Figure 59: AdaapinHHicaaccicllMMMnTTTTTIT
G.2 Example Design Artifact
Proposal for VQH
VQH-GAU: Vector Quantized Hierarchical Generalized Autoregressive Unit
87

---

Abstract: The proposed VQH-GAU integrates selective gating, vector quantization, and hierarchical memory architectures to enhance computational efficiency and scalability in autoregressive language models.
Motivation
Autoregressive language models (LMs) have achieved remarkable success across various natural language processing (NLP) tasks. Central to their performance are Generalized Autoregressive Units (GAUs) that process imput sequences efficiently. However, as model sizes and sequence lengths continue to grow, existing GAU architectures face significant challenges maintaining computational efficiency, memory scalability, and the ability to capture long-range dependencies. To address these limitations, there is pressing need for an innovative GAU design that integrates selective gating mechanisms, vector quantization for memory compression, and hierarchical memory architectures. The proposed VQH-GAU aims enhance computational efficiency, reduce memory overhead, and improve scalability while maintaining surpassing current state-of-the-ard performance metrics such perplexity, accuracy, and robustness.
Related Work
Selective Gating Mechanisms
Selective gating has been effectively utilized models like Mamba [2] and Eagle and Finch [1] to dynamically manage state representations based input relevance. These mechanisms allow models to retain only pertinent information, optimizing memory usage and computational resources. Integrating selective gating within GAUs significantly enhance their efficiency and focus on essential contextual cues, reducing unnecessary computations and memory overhead.
Vector Quantization Techniques
Vector quantization (VQ) has emerged a powerful tool for memory compression in large-scale models. Techniques such as Pyramid Vector Quantization (PVQ) [3] and Channel-Relaxed Vector Quantization (CRVQ) [4] offer efficient encoding and decoding of -igh-dimensional data with minimal information loss. Implementing VQ within GAUs enables compact state representations, thereby reducing memory footprints and accelerating computations without compromising model performance.
Hierarchical Memory Architectures
Hierarchical memory structures, demonstrated by DenseSSM [5] and Neural Language of Thought Models (NLoTM) [6], facilitate efficient handling of long-range dependencies by organizing memory multiple hierarchical levels. Such architectures enable GAUs to maintain fine-graine -reined information across different scales, enhancing their capacity to capture complex patterns inherent in language data while maintaining computational and memory efficiency.
Computational Efficiency and Scalability
State Space Models (SSMs) like Mamba [2] and DenseSSM [5] have shown that integrating selective gating and hierarchical memory can lead to significant improvements in computational efficiency and scalability, These models achieve linear-time inference and maintain competitive performance, making them suitable for deployment in large-scale language models. By adopting similar strategies, the proposed VQH-GAU aims to optimize computational resources and facilitate scalable model designs.
88

---

Problem Analysis
Current Limitations of Existing GAUs
Memory Efficiency: Existing GAUs may still suffer from high memory footprints, especially when dealing with long sequences, limiting their scalability.
2. State Management: Conventional state management in GAUs may not effectively focus on relevant information, leading unnecessary computations and potential loss of critical context.
3. Scalability Challenges: As model sizes and sequence lengths grow, maintaining computational efficiency becomes increasingly difficult, hindering the deployment of large-scale language models.
Core Philosophy Behind VQH-GAU
The VQH-GAU aims to revolutionize the GAU architecture by integrating:
1. Selective Gating Mechanisms: To dynamically focus relevant states based on input tokens, enhancing efficiency by reducing unnecessary computations.
2. Vector Quantization for Memory Compression: To encode state representations into compact vectors, significantly lowering memory usage without losing essential information.
3. Hierarchical Memory Architectures: To organize memory across multiple hierarchical levels, enabling the model to capture long-range dependencies more effectively while maintaining computational and memory efficiency.
Mathematical Justification
Selective Gating:
8: ( 5. b.) hy 82 1 + (1 st) 3 h2 where s; determines the extent which the current state hg is updated based on the input 2,,
2. Vector Quantization:
VQ(h) = arg min //e call²
LvQ = [sg[a]] h4 12 + Bllge - sg(?t]]12 where g. VQ(h.), and a is the sigmoid function.
3. Hierarchical Memory Integration:
H a. a,H1 + Baqt
V/ = concat H .H(2) (!?)...,H!)) where a., 8, are learnable scaling factors for each hierarchical scale.
H...,
Trade-offs
Complexity vs. Efficiency: Integrating selective gating and vector quantization introduces additional parameters and computational steps; however, these are offset by the gains in
89

---

memory efficiency and scalable performance.
While vector quantization reduces memory usage, excessive compression may lead to information loss. Balancing quantization levels is crucial to maintaining model performance.
Quantization Precision:
Design Plan
Architecture Overview
The VQH-GAU comprises three primary components arranged hierarchically to streamline information processing:
1. Selective Gating Unit: Dynamically determines the relevance of incoming tokens and adjusts state updates accordingly.
Vector Quantization Module: Compresses state representations into compact vectors, maintaining essential information while reducing memory usage.
3. Hierarchical Memory Integrator: Organizes memory across multiple hierarchical levels, facilitating effective capture of long-range dependencies.
Detailed Component Descriptions
Selective Gating Unit
The selective gating unit controls the flow of information based on input relevance, effectively updating the state only when necessary.
Gating Function:
8: o(Ware b.)
State Update:
h4 81 h4-1 (1 st) o h
where hg is the candidate state generated from the current input 24.
Vector Quantization Module
This module encodes the state representations into discrete vectors, significantly compressing the memory footprint.
Codebook Definition:
C (c)kk 9 GER
Quantization Process:
vQ(h.) c i' arg min min
||, call
Loss Function:
LvQ ||s][d]] h, 12 + Bl1gc sl|h,]ll where g: VQ(h+), and B is weighting factor.
3. Hierarchical Memory Integrator
90

---

This component organizes memory across multiple levels, enabling the model to handle long-range dependencies efficiently.
Hierarchical Structure: The memory is organized into multiple scales:
= ( H()) 115))
where S' denotes the number hierarchical scales.
Memory Integration:
H = a, (1) + 3. (Q(h))
where 0'., Ag are learnable parameters controlling the contribution from each scale.
Mathematical Formulation
The overall GAU operation be represented as:
(W,at + ba)
h oWW,. bn)
81 @ 14-1111110 + (1 - 8z) ④ hr
M = vQ(h+)
H a. a, + Baht Ys f .l...... S} = concat( (7!1), H²(") ..... H(S))
where a the sigmoid activation function, and concat denotes the concatenation of hierarchical memories.
Implementation Guidelines
Pseudo-Code
"pyyhonn import torch
import torch.nn
import torch.nn.functional as F
class CnneetiieGGiinngnniiinmmmiiiee
def init__________________________ j___iintii______ (self, embed_ _dim:
supprr(SleetiiveGaatngg self)___nitt init init_ O
saff.gate Linee dim, embed_ dim)
self.activatio = migmmooid)) wulf.state_transffrre L..eaar((emeed _imm embed_ding dim)
def forward(self,, x, h h_prev):
sclfactiivaiionnseeff gate(())))
h tilde = aelf.aatiivaiionsef.ttttt (aatttttttasfo())))
h = # h prev (1 s) h tilde
return
class NectorQuanizzaiion(n.Mooo Module))
def init (self, embed dim, num emmeeddinee......... beta=0.25):
super(VectorQuual VectorQuaniiiaaionn snff)_niit__ embed_din dim
self.num_ent siuuum_beeddings = num_enm gam_embeddings bela.beta beta
91

---

self.embedding- Embeddiinn
self.num _ self.embed dim)
self.embedding. weight.data.unifomt
-1/self. num eemneddinees.s...... l/self.num_emmeeleeeee _mbeedddinnm
def fee.,, h):
# Compute distances
h_flat v.view(-1, self.embed_ddee _ddimd__mmmm_
distances (h_
keepdim=True)
(self. embeddingg
2 torch.matmul(h. flat, self. celf.embediing.. weight.t())
# Encoding
aucoding_indices indices torch. araneaaargminnissanneesss Junsue)uusqueeze(( encodings = torch. (enooiig indices.size(O)) self.num embeddings, (rrice=h.device) encodings.scatter_.. encoding_indices, 1)
encoding
# Quantized vectors
quantized torch. hattmmmmmmmmlmiinngggg
self.embeddingg weight).vins view view("hshhape
# Loss
latent loss F.mse Loss(quantized.ddttachh h)
q latent loss = F.mse losssssssssaaiieddd h.detach()
loss latent loss + self.beta latent loss
# Straight Through Estimator quantized h + (quantized de.deth) return quantized, loss
class srarraacccclMemmoynnteerrooooooe Module))
def _iin ___iiiniiiillll embed dim, num scsless MoaerHHerrrccccllMmmmmmeeeeeee
self). init self.num scales num sccllsscllllsssssseeseseeeeeeeee
self. allf.alpha nn. wwwpeneteeetorc..aannnnnana scales))
self. = nn. www.rerrnnooccc.nnnnnnnnn s_sllls)))
self.upsample nn. Upsample(scale, factor=2, 2 mode="linear" align_cone saann__oorrrsrraae
def forward(self,, quantized_ h, memories):
for in range(self.num) amngeself.nummscales)))
if >= len(memories):
memories.appendd(tore) uinpppuitrrrrieeeessiiueuuuiill _h)
memories[s] self.alpha[s] * memories[s] self.beta[s] quantized
quantized h self. wwt.uuummmqqquutizess
output torch.cat( memorres, dim=-1) return output, memories
class VQH__QQHLLSSSS (AU(nn.Module))
def i____nit_____ embed_dinn dim, mum_embedding:=256.
num_scales=3):
super(VQH GAU, self). init self.selective wtt.seelecvve_aatinng SelectiveGating UnnneiiveeGatiinnnnnnnnbeeee dim)
self.vg VectorQuanizzaiionemmbbe dim, num embeddings)
self.hierarchice memoryyee = HicurrhicalMemoryynneaaoo
92

---

embed dim, num scaless
def' forward(self, x, prev, memories):
h self.selective gatiigxx_ h h_prev) quantized vosloss self..q((h) output, memories www...ierarhhhcaaa..o.. mumrrrrciiccmmmmmmuuuunniieee memories) return output, h, vq_loss, memories
Mathematical Formulas
1. Selective Gating Function:
(WWrr + b,)
h. oWWier + bj.) h4 8t h + (1 st) h4
2. Vector Quantization:
VQ(h: arg min lh c:
LvQ = [sg[a]] - h, 22 + Bl1g: sg[h,]]2
where g, = VQ(h)), and o is the sigmoid function.
3. Hierarchical Memory Integration:
H a. H + B,q. 1/1 = concat 11 (1()) ,..., H (s))
Conclusion
The VQH-GAU represents significant advancement in Generalized Autoregressive Units by integrating selective gating mechanisms, vector quantization for memory compression, and hierarchical memory architectures. These innovations collectively enhance computational efficiency, reduce memory overhead, and improve the model's s ability to capture long-range dependencies. By addressing the limitations of existing GAUs, VQH-GAU is poised to outperform current wfae-offthe--an models terms of perplexity, accuracy, robustness, and scalability, making it promising building block for next-generation autoregressive language models.
References
Peng. B., Goldstein, D., Anthony, Q., Albalak, A., Alcaide, E., Biderman, S., Cheah, E., Ferdinan, T., Hou, H., Kazienko, al. (2024). *Eagle and Finch: RWKV with Malrix-Valued States and Dynamic Recurrence* arXiv preprint arXiv:XXXXXXXXXX
2. Gu, A., & Dao, T. (2023). *Mamba: Linear- Time Sequence Modeling with Selective State Spaces arXiv preprint XXXI:XXXXXXXXXX
Chen, I.,, Wang, H., Li, K., et al. (2018). *Pyramid Vector Quantization for Efficient Encoding and Decoding*. In Proceedings of the CVPR.
4. Zhang, Y., Li. S., & Wang, W. (2024). &Channel-Relaxed Vector Quantization for Memory Compression Language Models Journal of Machine Learning Research.
He, W., Han, K., Tang, Y., Wang, C., et al. (2024). *DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models#, arXiv preprint
93

---

arXiv:XXXXXXXXX
6. Wu, Y.-., Lec, M., & Ahn, S. (2024). *Neural Language of Thought Models International Conference on Learning Representations.
Note: The arXiv preprint numbers replaced with actual identifiers once available.
are placeholders and should be
Review for VQH
Comprehensive Analysis of the VQH-GAU Proposal
Strengths
1. Innovative Integration
The VQH-GAU uniquely combines selective gating mechanisms, vector quantization, and hierarchical memory architectures within Generalized Autoregressive Units (GAUs). While each of these components has been successfully implemented individually in various models, their holistic integration appears to be novel. This comprehensive approach addresses multiple facets of autoregressive language model limitations simultaneously, potentially leading to synergistic improvements in efficiency and scalability.
2. Proven Components:
Selective Gating Mechanisms: Models like Mamba, Eagle and Finch, and Gated State Space Models (GSS) have demonstrated the effectiveness of selective gating in managing state representations dynamically. This enhances the model's ability to focus on relevant information, thereby improving both efficiency and the capacity to handle long-range dependencies.
Vector Quantization Techniques: Techniques such as Pyramid Vector Quantization (PVQ), Channel-Relaxed Vector Quantization (CRVQ), VPTQ, and hierarchical methods in models like Speech Tokenizer and T2S-GPT have proven successful in compressing memory without significant loss in performance. These techniques are pivotal for reducing memory footprints and accelerating computations.
Hierarchical Memory Architectures: Architectures like Spectral State Space Models, ConvSSM, TRAMS, and Neural Language of' Thought Models (NLoTM) effectively organize memory across multiple levels, enhancing the model's ability to capture complex patterns over extended sequences.
Theoretical Soundness:
-The proposal provides solid mathematical justifications for each integrated component, ensuring that the design decisions are grounded in established theoretical frameworks. This includes detailed formulations for selective gating, vector quantization, and hierarchical memory integration, which collectively support the anticipated improvements efficiency and scalability.
Alignment with Current Research Trends:
The integration aligns well with ongoing research trends focused on enhancing model effi- ciency, scalability, and the ability handle long-range dependencies. By addressing multiple challenges simultaneously. VQH-GAU positions itself at the forefront of advancements in autoregressive language modeling.
Concerns
94

---

1. Complexity of Integration:
Combining three advanced mechanisms selective gating, vector quantization, and hierarchical memory- introducss significant architectural complexity. Ensuring seamless interoperability between these components is crucial to prevent potential conflicts and to maintain model stability during training and inference.
Quantization Precision:
While vector quantization effectively reduces memory usage, there is an inherent trade-off between compression level and information retention. Excessive quantization may lead to information loss, adversely affecting model performance. The proposal must carefully balance quantization levels optimize memory efficiency without compromising accuracy.
Implementation Challenges:
Integrating hierarchical memory with selective gating and vector quantization within GAUs may present practical implementation challenges. Detailed guidelines and optimized algorithms are necessary to ensure that the theoretical benefits translate effectively into real-world applications.
Empirical Validation:
The proposal primarily focuses theoretical advancements. Empirical evaluations and benchmarks are essential to validate the proposed improvements in real-world scenarios. Without empirical results, the claims regarding enhanced perplexity, accuracy, robustness, and scalability remain theoretical.
Suggestions for Improvement
Empirical Evaluation Plan:
Develop comprehensive plan for empirical evaluations the VQH-GAU's perfor- mance across various benchmarks. This should include comparisons with existing models to substantiate claims of improved perplexity, accuracy, and efficiency.
Quantization Strategy:
Provide detailed strategies for managing the trade-off between compression and information retention. This could involve adaptive quantization levels based on input complexity or incorporating mechanisms to recover lost information during quantization.
3. Implementation Guidelines:
Expand on the implementation guidelines to address potential challenges in integrating the three mechanisms. This could include optimized algorithms, recommended hardware configurations, and strategies for parallelizing computations to maintain efficiency.
Ablation Studies:
Conduct ablation studies isolate the impact of each integrated component. This will help in understanding the individual contributions of selective gating, vector quantization, and hierarchical memory to the overall performance enhancements.
5. Scalability Analysis:
Include a detailed scalability analysis to demonstrate how VQH-GAU performs as model size and data volume increase. This should encompass both theoretical scalability metrics and practical scalability tests.
Final Rating
Based on the comprehensive integration of proven mechanisms, theoretical soundness, and alignment with current research trends, the VQH-GAU proposal demonstrates significant potential to advance autoregressive language models. While there are concerns related to integration complexity and the need for empirical validation, the proposal's innovative
95

---

approach and thorough mathematical justifications position it strong contender in the field. Rating: 4.5/5.
Composed GAB implementation from the GAU Tree
gab.py
import torch
import torch.nn
model_ discovery
inport
clasa GAB (GABBaaae) 10
def ...nit_.. (self embed_dim: int, block_loc: tuple, device-Xone,
11
ätype- **kuargs);
12
factory kvarge ('device*: device, "dtype": dtype)
super (). ().-.... init._ (enbed_din, block_loc)
14
self = VQH_CAU(ombbede din block_loc-block_ee uxarg_all-xvarga, s*factory.kvargs **kuarga)
16
17
def _sor 2, *+2):
18
I, 2 self (I, **2) return x, 2
19
21
22
from model_ discovery model. utila modules inport GAUBase, gau_test, UnitDecl impert torch. functionsl
23
24
25
26
class VOR GAALGAU (GAUBase):
28
The VQH_GAU is the root unit of the VQR-GAU architacture. It integrates the SelettireGatingliit, Vectorquantization, and Hierarchical HenoyyIntegrator tocprocess input sequences.
29
30
31
32
*<Componenta*x:
33
.*Sslletivaaarriigllattrr Applies selective gating input sequence.
35
36
##VectorQuantiiatioonn Compresses the hidden state using vector quantization.
37
38
# caleanoryynttogrrrrr Integrates the quantized hidden
39
hierarchical nemory atructure.
40
*xrgga*::
42
embed_din (int): Embedding dinension 'p',
43
block_loc (tuple): Location the block within the netxork.
44
kxarg_all (dict): Dictionary of all keyvord arguments.
45
device (torch. device, (tvrch.devica, optional): Devica to run the unit on. type
46
dtype (torch.dtype optional):
47
48
Inpptaee::
49
**X*. (torch. Tensor): Input tensor of shape ((B, D)
50
.. (dict): Dictionary containing intermediate variables.
51
52
**Outputax::
53
*.Y+s (torch. Tensor): Output teneor shape ((B, L, D)s,
54
..2*. (dict); Updated interaediate variables.
55
56
Example**:
57
58
Initialize unit unit = VQH_GAU (in2ed, 1lc-k_ 0), kwarg_all-())
59
60
input borch.randde (4, 10, 32) # (B, L, D)
61
62
Forward pass
63
Y, Z (X)
64
65
**Noto*::
66
root unit It conposes the SelectiveGatinguhitt. VectorQnantization,, and
architecture.
68
69
Bierarch unite.
70
96

---

71
72
73
def
i.init_. (self, embed_din: int, black_loc: tuple, kearg_all:
74
device=lone, dtype-None, **kxaggs):
75
falfffaactryy ('derice': device, 'dtype'; dtype)
76
super D....iit... (embed_din block.loe, kvarg_all)
77
self. - Sebectie@@ringgglieeeeedddddddd self self.enbed,din, blok__ocrssai..bloci..ee. kvarg_sll-
78
79
self welt.koarg.all *xself. - *eself kragg.
80
wew.veeeturuuaeee. puetor-quentizatien Vector/quaniiiaaiiooooeeeeeee self self.enbed_din, bloe__lorssalt.bbocc...00 kvarg_all-
81
self selr..oarg.all, maeaf-ffactory_tvargg, kearg
83
self hierarcchical - Bierrarciicclleeerrrttggg (enbed_din-
84
self, eebed_din, block_loc=self block. kvarg_all-
85
self kearg_all, **self faelf.factory.kkaaa **self kwarg_all)
86
87
(sll, *+2): X
88
89
Y, z_ag sel..eeleetveeeeetiggg. **Z)
90
Y, Z_vq seltor veetor_.uuniiaaiion... +*2_8g)
91
2_combined (*+2_sg. =+2_vq)
92
Y, 2_hmi self haaaacchrrrccccll.naaee z_out [=*2_conbined, **Z_hni?
**2_cobbined)
93
94
Y, 2_out
95
96
97
98
class Selectiieoarr:nguhitt (GAUUAA
99
***
100
The SelectiveGatinglhit implementa a selective gating nechanien to dyasmically update the hidden based on input
102
**Overvieu:**
104
<*Gating Punction*::
106
Computes gate values fat uaing signoid activation applied linear transfornation of the imput 'x', WI
107
109
a (Vigma(V_a x b_a)
WJ
112
Candiddate Statexe:
1lde(b)' using activation function (e.g., 'tanb') applied to a linear traneformation of (x'. W.
Computes candidate
114
116
ilda(h} ext(activation))k.. x b_h)
WJ
118
119
**State Update*s:
120
Updates hidden 'h by combining "ba ext(prev))* and (h) veighted by the gate fg'.
121
WI
123
= Wadat b_f ext(prav]) (1 s) Wodot 11da(h}
WJ
125
shere 'Wodot denates alement vise multiplication.
126
Args:**
128
embed_din (int): Enbedding dimension (D',
block within network. kvarg_all (dict): Dictionary of all keysord argunents.
block_lac (tuple): Location
130
device (torch. device, optional): Device to run the unit on. dtype (torch.dtype dtype, optional): Data type
132
133
Inputs:**
135
***** (torch. Tensor): Tensor:: Input tenser of shape {(B, L, D)
(dict): Dictionary containing intermadiate variables. "b.prev' Tensor): Previous hidden state of abaps ((B, L. " If not provided, initialized to zeros.
137
140
** Outputt:: :*4
*eYoe (torch. Tensor): Updated hidden state shape L, D)
142
**2** (dict): Updated internediate variables including "h.prov'.
144
==Exmppe::s
Initialize unit unit - SolectiveGatinglun (embed (en-e2_din-32. loc-__,,- 0),
147
kvarg_all-())
97

---

149
Nock input
150
torch. carch.randn(e (4, 10, 32) # (B, L, D)
Forsard paas
152
Y, 2 unit(X)
154
**Note:** :**
This unit is designed to be used within a larger GAU architecture. It implenents the selective gating mechanien described the VQH-GAU proposal.
156
159
161
def inii (self, embed_dint din: int, block_loc: tuple, kvarg_all: diet, device-Rone, atype-None, *ebxargs):
163
self fat.ory..tyy.rrg-
('derice'; device, 'dtype'; dtype)
super () ....tttt__ (enba__.en,, block_loc, kxarg_all)
seli.W_a mm..Liner (n..Linerreebedd (embed_ddd eibed_din, **gelf. factory_kvarge salf.8_h an Lnn.Liner (enbedddnne,,,in,, anbad_din, reself. faaaff.acctor.kkarrga self.activation an Tanb() self sigmoid Sigmoid())
166
168
170
def _forward to factory_karrr b.prev Z.get(Ch.pze") None)
**2):
171
173
if h_prev is None:
h.prev torch. 1exee welf wew..eeigmidleell....ccc
175
176
b_tilde self self (X)) h_prav ž_out ('h,prev'; h)
b_tilde
178
h,
180
class Hieeerraccallaeoo//ssssssteer (GAUUBU)))))))))))AAAAA)
185
Bierarchicalleeneryllagggaa
187
This unit integrates quantized hiddan states into hierarchical menory structures across gultiple scales.
189
190
**Nain Features:*x
192
=eHerrarchical Nemory Integration+:: Mansges multiple hierarchical menory to capture rangerange dependencies.
194
#*Lerrabble Paranetorsee: Uses learnable scaling factors \V alpha_s W) and WV beta_s W each scale.
196
197
nenories original enbedding dimension to maintain output shape consistency.
**Dutput Projection*s; Projects
199
201
M*Mahenntical Formulationx::
203
For each scale WC Win Wa, Wdota, sV Wo
204
206
(()) - alpha_s (_(t=s)-((t) + beta_s Wedet (quantizadV(_h).t
208
W
209
211
WD nanory previous atop WV t-s W at scale W W.
WV H_t-((e)) W scale W a WD)
hierarchical menory
213
WV (a(t- W
215
WV alpha_s 10 and WC beta_s W) are learnable paranetere for WV W..
218
- -W ezt extfquantized W. is the imput quantized hidden state at W W.
220
**Output**:
222
The output W f_t W) is forned by projecting the concatenated nenoriea back the enbedding dinension:
225
98

---

227
Y_: ext (Projection)) (x((oon))))LL=)()))) (L)"((2)), Wdotz, (L=-((B))))
229
W
230
*xggaau::
232
**embed_ dins. (int): Embedding dimension AV W.
234
=bbooc_loc** (tuple): Location of the block within the network. <tksarg_allvs (dict): Dictionary of all keyvord argunents. **devicexs (torch, (torch. device, optional): Device
unit
237
**dtypo** (torch. optional): Data type of tensers.
senun_acalesse (int, optional); Number hierarchical acalea WC W. Default: 3.
239
240
**alpha inite* (flost, optional): Initial value for WV alpha_e WD
241
Default: 0.9.
x*bata_ inites (flent, optional): Initial value for W beta_a W. Default: 0.1.
244
=Ipputane:
246
247
***+* (torch. Tensor): Iaput tenser of shape WV (B, L, D) W. representing the quantized hidden states.
248
249
**Z** (dict): Dictionary containing intermediate variablea, poasibly including previous bierarchicsl
251
**Outputs*::
253
**Y+* (torch. Tensor): : Output tensor of shape W (B, L, D) W. shape input. **Z** (dict): Updated dictionary containing the new hierarchical memories for scale.
255
maintaining
256
258
Exauplees:
260
261
Initialize the unit
262
HterarchicalMeooryy sararachcalleoorrtttggrrooo (embreeddddreeeedddee (embed_din=32, block_loe=(0, 0)
263
kvarg_all-O,, aun_scales-3)
input
265
X torch.randi (4, 10, 32) # (B, L, D)
Intermediate variables ()
267
Forzard pass Y, (x, **2)
270
272
273
This unit is designed to be used within the VQK GAU architecture. integrates quantizad hidden states into hierarchical manory structure and maintaina shape conaistency with the input. internediate variable
274
275
nemory dictionary '2'.
stored updated
277
*aa
279
280
def innit.. onbed_dim: int, block_loc: tuple, kearg_all: dict, davice=None, dtyperNone, nun_scales: int=3, alpha_init: fleat=0.9, beta. init: f1oat=0.1, <*kuargs);
281
282
fwrf. ('device'; device, 'dtype': dtype)
284
super int___.......teed______._ self. mun__ccales num_scales
block_loc. kvarg_all)
286
self. nn. Paranstttr (torch. full (ulI((mu__acale, 1, 1),
287
alpha_init, **self. faaollffactttr__kaae
288
salf.beta Parameta (torch. full (nunn_e_cce,, 1, 1),
289
beta_init, s*gelf. www.faatcttrrggaeeeee self. welf.output_pprt propptt.proooetion onbed_dim, **self, wactory..aorrrgea
LaneLi (embe___de aun_ecales.
291
293
def _frrvard *+2);
294
B, L. D ehepe device deviice dtype X.yyype
296
H_list
298
2_tut O
for in rang((eoII.mn_ccalee))
300
scale +
301
alpha_s self alpha(s]
beta_s [e]
303
__prev ge [scee_ None)
__prev Mone:
99

---

305
__prev torch. seros(B, 1, **self factory..warrg) if scale L:
306
torch, Zar scale, *xaelf factory.t..... (actory.kuarga)
308
H.prev_chifted carch.cat((pad H_prev[:, i-scale, :]], dim=1)
310
else:
311
H.prev_ torch, geros (B, L, D, **eelf. wtwtefffcttorrrrg..oe __curr alpha_5 H.prev_chifted beta_s + 2_ [_te
312
313
H_curr
R_list. aasistt.ppeeddH_occ
315
H_concat arch.caa(H__list. dim=-1)
self ortppt..rooection (H_concat)) )
317
return Y, 2_out
319
320
class (ectoruunntiaatioonnCGGAee
122
VectorQuantization
324
This unit perforns vector quantization on the hidden states to compress the representations.
326
327
It napa continuous bidden atatea to discrete codebook entriea, reducing
nenory usage msintaining sesential information.
329
**Nain Features:4*
331
332
=*Codebook*:: A learnable embedding table containing 'K" code vectors dimension fD'.
333
334
Quantization Process**;
336
Por each imput rector 'het' in the hidden state 9E": W 1 (rg WI bet eLi W W
338
339
9.t c_(i W
340
341
where W 0.t W are the codebook vectors.
343
Loss Fanction*s:
The vector quantization introduces a loss tern to train the cadebook:
345
346
Waathcal (L)_ (VQ) W. h,t ext(ag][qat] Wia beta WM g.t
348
ext(sg) [h_t] W112
W
350
where W (xt)))()tcd W denotes the coop-gradient operation, and W beta W. is hyperparaneter controlling the commitment lass.
351
352
353
hrggas::
355
veembod_dinee (int): Embedding dimension AM W.
357
<eblock_loc*k (tuple): Location of the black within the network. all keyvord argunents. the unit optional); Data type of tensors. s*nun "nuu__nbeddingee. (int, optionsl): Number embeddinge codebook WV & W. Default: 512.
**bxarg_all** (dict): Dictionary
359
**doricex* (torch. (torch.device, device, optional): Davice
360
**dtype** (torch.
362
(fleat, optienal): Commitment veighting WV beta W. Default: 0.25.
364
366
*<Inputa*::
367
(torch. Tensor): Input representing the hidden states. (dict): Dictionary containing intermediate variables.
shape WC (B, D) W.
369
371
*utputte**:
374
..Y.. Tensor); Quantized tensor of thape W (8, L, D) W.
.*Zee (dict): Updated dictionary containing any necessary internediate variables.
376
=evq_losaas (torch. Tenaor): Vector quantization 18as (optional, for training).
378
379
*<Exappee*:;
381
Initialize
100

---

383
unit VectorQuantization (enbed_din-64, _lock_loc=(0 0),
384
kvarg_all=O, num con_embodding--612, beta=0.25)
385
Nock input
386
torch.randie (8, 16, 64) # (B, L, D)
paas
388
y, 2 un)
390
:oto
391
deaigned used within It perforns vector quantization as described in the VQR- CAU propeaal.
architacture.
393
395
The quantization loss used during training to update the codebook vectors.
396
397
400
def
init (sel_ embed_dim: int, black_lac: tuple, kearg_all: dict, device-None, dtype-None, nun_ambeddinga: int-512, beta: float=0.25, **kvarge):
402
self factfffotrryy.argg.e ('device'; device, 'dtype'; dtype)
404
super Sn.ert.... (e.._ditt_____edd_dd. (enbed_din, block_loe, kwarg_all)
405
self-nun.enbatting enbeednndd
self.beta
407
self.
an Eubedding (self. www.tauunnneeaeeeeeee self eelfeenbed,,im,
409
self eubeediin veiddingg.eiiteee veight dats. uniforn_ (-1 self nun_enbediingg, self wet..nunnnbediinge.
411
412
def
(ael, x, *+2):
B, assert D -- self solf.enbedddin. E I'Input embedding dimension (D) does not natch expected dimension (abed_eabed
I.ehape
414
416
b_flat (-e. D)
codebook self. cel.dembbeding.co
distances corah (h_flat.... (h_
unsquuuss
unaquezze (D), codebook unaqueeze (0))
419
distauces distances [0]
assignment_weighte
/ 0.1, dim-1) codabook)
421
quantized - torch.
torch. am (maaaaignnnttteeeeteeeooo
encoder_loss P.sse_loss (b_ quantized)
423
lass encoder_lost
quantizad = maantizad.viav(C L, D)
425
quantized
426
Z_t 0
2_out ['rq_loas"]
428
return Y, Z_aut
430
431
gab_config ('beta's : 0.25, 'tamparatura'; 1.0, "nun_ sannaddddding' 512,
aalpha_ 0.9, Pcunee 3, "beta_init', 0.13
101


Reviews:
[{"review_id": "C13MOTsoEt", "reviewer": "Reviewer_3FLT", "summary": "This work proposes Genesys, a multi-agent system that autonomously discovers new language model architectures using LLMs. The pipeline generates, verifies, and evolves models across scales using a unit-based code representation. Genesys is demonstrated to produce over 1000 verified designs, with some architectures outperforming GPT2 and Mamba2 on standard benchmarks.", "questions": "1. Whether the GPT-style architecture is in the knowledge base and serves as a seed architecture of the evolution process? If so, the improvement of the finally searched architecture is relatively minor.\n\n2. Will the proposed system achieve better results than traditional NAS methods that use mutation/cross-over instead of LLM proposers?\n\n3. Could you show the single-best architecture that the proposed system searched? How difference it is compared to current LM architectures? Any new architecture design insights we can learn from it?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\n1. Generally, the work is clearly written and easy to follow.\n\n2. This work demonstrates good engineering efforts in building an ASD system for a specific purpose.\n\n**Weakness**\n\n1. The major concern with this work is the limited technical novelty and limited research insights offered to the community. This work is more like a demonstration of a successful engineering effort to build an ASD system, while the scientific aspects of LM agents and new insights are insufficient, limiting its value to the research community. In case I missed something, the authors are expected to summarize the insights and major contributions of this work.\n\n2. Another major concern is the lack of an apple-to-apple comparison to demonstrate the effectiveness of LLMs. Simple mutation/crossover, as done in traditional NAS methods, can replace the role of LLMs. A comparison between LLM-based ASD systems and rule-based mutation/crossover in traditional NAS should be provided to show the benefits of LLMs. The effectiveness of LLMs remains unclear to me.\n\n3. The searched architectures are highly sensitive to the search space. The authors are expected to (1) improve the clarity of the search space definition, (2) show the architectural details of the best searched architectures, and (3) ablate the search space choices to validate the robustness of the ASD system.\n\n4. Table 3 is unclear to me, and I do not understand where the better performance on 6 out of 9 benchmarks comes from. It seems the single best model searched in this work is very comparable to GPT.", "was_revised": true, "final_justification": "While the authors have clarified some aspects of their work, key concerns remain unresolved. The proposed contributions, e.g., factorizing architectures into Generalized Autoregressive Units and introducing a \"ladder of scale\", appear tailored to this specific architecture search framework, rather than offering broadly applicable insights into ASD (Automated System Design). \n\nAdditionally, the absence of a fair and thorough comparison with traditional NAS methods makes it difficult to evaluate the effectiveness and generality of the proposed approach. Given that architecture search performance is highly sensitive to search space design, stronger empirical evidence is necessary to support the claimed benefits. The lack of a clear improvement over GPT-2 further weakens the case for novelty and practical impact. \n\nFor these reasons, I lean toward a rejection.", "rebuttal": "We thank Reviewer 3FLT for the critical and detailed review. We appreciate the opportunity to clarify our work's technical novelty, justify our methods against alternatives like traditional NAS, and elaborate on the scientific insights generated by our system.\n\n---\n\n## Weaknesses\n\n>  **W1:** The major concern with this work is the limited technical novelty and limited research insights offered to the community. This work is more like a demonstration of a successful engineering effort to build an ASD system, while the scientific aspects of LM agents and new insights are insufficient, limiting its value to the research community. In case I missed something, the authors are expected to summarize the insights and major contributions of this work.\n\n> *“The major concern with this work is the limited technical novelty and limited research insights...”*\n\nWe respectfully disagree regarding the limited technical novelty and insights. Our work introduces key innovations to tackle ASD in a complex, high-cost domain like LM:\n\n* **Novel Genetic Programming (GP) for Code Generation**: Our novel GP framework factorizes architectures into **Generalized Autoregressive Units (GAUs)**, enabling a Viterbi-style search. This addresses a key bottleneck, improving valid LM code generation from 6% (direct prompting) to 92% (**Table 2, proof in Appendix A**).\n* **Budget-Aware \"Ladder of Scales\" Strategy**: Our \"Ladder of Scales\" strategy (**Fig. 9**) makes high-cost ASD tractable by pyramidally allocating the verification budget, using smaller scales to efficiently filter promising architectures for promotion to larger, more expensive scales.\n* **Scalable Multi-Agent Research System**: Genesys & LMADE is the first system for this task to model the complete research lifecycle with highly-scalable distributed designs.\n\n> *“This work is more like a demonstration of a successful engineering effort... while the scientific... insights are insufficient...”*\n\nWhile Genesys is a significant engineering effort, our primary goal was to use it as an instrument to generate scientific insights into both ASD and LM architectures:\n\n* **Systematic Ablation of ASD Components**: Our ablations (**Sec. 5.1, Table 1**) empirically quantify the value of discovery components. Removing experimental feedback (`w/o Exp.`) or literature access (`w/o Lit.`) hurts performance, validating our experiment-guided and knowledge-driven evolution.\n* **Analysis of AI-Driven vs. Human Discovery**: We analyze and contrast the evolutionary trees of AI vs. human research, finding that our system develops a \"hubness\" pattern of exploitation, distinct from the more diffuse human research network (**Appendix E.1.2, Figs 34-36**).\n* **Linking Architectural Units to Performance**: We pioneer an analysis linking specific GAUs to downstream performance. A classifier using a \"Bag-of-Units\" representation can predict a design's fitness from its components (0.658 F1 score on held-out data) (**Appendix E.3.3, Tables 17-18**).\n\n> *“In case I missed something, the authors are expected to summarize the insights and major contributions...”*\n\nWe thank the reviewer for this suggestion and agree that a concise summary would improve the clarity. We will add an itemized list of contributions to the introduction, summarizing our **System design**, **New Methodology**, **Discovered Architectures**, and **Scientific Insights** as detailed above.\n\n>  **W2:** Another major concern is the lack of an apple-to-apple comparison to demonstrate the effectiveness of LLMs. Simple mutation/crossover, as done in traditional NAS methods, can replace the role of LLMs. A comparison between LLM-based ASD systems and rule-based mutation/crossover in traditional NAS should be provided to show the benefits of LLMs. The effectiveness of LLMs remains unclear to me.\n\nWe thank the reviewer for raising this important point. We argue that traditional NAS methods are ill-suited for this domain due to its unique challenges. We will clarify this in **Sec. 2**.\n\n1.  **Search Space is Too Broad for Fixed Operators**: Traditional NAS requires a concise set of operators, which is infeasible to express diverse paradigms like GPT, SSMs, and TTT in a compact way. *It remains an open problem how to do NAS effectively to the unbounded LM designs.*\n\n2.  **Prohibitive Evaluation Costs Invalidate Brute-Force Search**: Traditional NAS requires evaluating thousands of candidates, each involves expensive pre-training. LLMs improve **sample efficiency** by proposing knowledge-driven, targeted architectural changes rather than simple syntactic swaps. The importance of this knowledge is confirmed by our `w/o Lit.` ablation (**Table 1**).\n\nFurthermore, LLMs uniquely support the broader scientific goals of ASD by providing **interpretable, human-readable research proposals**, operating on high-level, conceptual units (GAUs), enabling analyses like our unit-performance correlation study (**Appendix E.3.3**). \n\n> **W3:** The searched architectures are highly sensitive to the search space. The authors are expected to (1) improve the clarity of the search space definition, (2) show the architectural details of the best searched architectures, and (3) ablate the search space choices to validate the robustness of the ASD system.\n\n> *“(1) improve the clarity of the search space definition”*\n\nWe thank the reviewer for their feedback and agree that the clarity of search space is crucial. The search space is defined by the **Generalized Autoregressive Block (GAB)** code construct (**Sec. 3.1**), constrained by a **Symbolic Checker** that enforces differentiability, causality, stability, and efficiency (**Sec 3.2, Appendix B.1.2**). To improve clarity, we will move key details of the GAB, GAU, and symbolic checks from the appendix into the main paper.\n\n> *“(2) show the architectural details of the best searched architectures”*\n\nWe detail the GAU trees for five top architectures in **Appendix G.1.1** and provide a full design artifact for VQH in **Appendix G.2**. We will also release all discovered architectures, including their training results and design histories, with the final paper.\n\n> *“(3) ablate the search space choices to validate the robustness of the ASD system”*\n\nWe conducted a series of system-level ablations to validate the robustness and effectiveness of the Genesys system in **Sec. 5.1 (Table 1)**. Removing fitness-based selection (`w/o Exp.`) or literature access (`w/o Lit.`) degrades performance, confirming that our method of navigating the vast architectural search space is effective and robust.\n\n> **W4:** Table 3 is unclear to me, and I do not understand where the better performance on 6 out of 9 benchmarks comes from. It seems the single best model searched in this work is very comparable to GPT.\n\nYour observation is correct. The claim compares the *group* of top 5 discovered models against the *group* of 5 baselines. The statement \"outperformed on 6/9 benchmarks\" means that for 6 of the 9 tasks, one of our discovered models achieved the highest score among all. It shows Genesys's ability to generate a diverse portfolio of specialized, competitive architectures, which is a key strength, as it's common in architecture comparisons that no single model dominates all tasks (**Sec. 5.3**).\n\n## Questions\n\n> **Q1:** Whether the GPT-style architecture is in the knowledge base and serves as a seed architecture of the evolution process? If so, the improvement of the finally searched architecture is relatively minor.\n\n**1. Yes, GPT was a seed.** We intentionally included five diverse, state-of-the-art seeds (GPT, Mamba2, etc.) to maximally model the real-world research, where scientists improve upon existing work, other than start from a blank slate and rediscover known architectures.\n\n**2. The contribution is the stable evolutionary progress and potential.** While individual improvements may seem incremental, our key result is a system demonstrating **stable, consistent fitness improvements** over generations (**Table 1, Fig. 8**). It shows the potential for more significant advances with further scaling supported by our scalable designs. Furthermore, the system produced a *portfolio* of models that collectively won on 6/9 benchmarks (**Table 3**), showing its ability to discover specialized architectures. \n\nWhile this is the largest automated LM discovery of its kind, it's not surprising to see incremental improvements rather than a major leap, given the task's complexity, our relatively low cost for the task scale, and our status as the first work of this kind.\n\n> **Q2:** Will the proposed system achieve better results than traditional NAS methods that use mutation/cross-over instead of LLM proposers?\n\nAs detailed in our response to W2, we argue traditional NAS is not viable in this domain due to the huge search space, high costs, and no existing effective NAS solution, making a direct comparison infeasible.\n\n> **Q3:** Could you show the single-best architecture that the proposed system searched? How difference it is compared to current LM architectures? Any new architecture design insights we can learn from it?\n\nWe present **VQH** in **Appendix G.2**. It sequentially processes information via three stages: 1) a **selective gating** unit filters relevant information to update memory; 2) a **vector quantization** module compresses this memory by matching it to a learned codebook; and 3) a **hierarchical integrator** organizes these compressed summaries into multi-scale memory banks (short- and long-term). The key insight is that language understanding can be effectively decomposed into these specialized, sequential jobs of filtering, summarizing, and organizing information over time.\n\n---\n\nWe hope these clarifications have adequately addressed the concerns raised. Please do not hesitate to let us know if further details on any of these points would be helpful. We appreciate the opportunity to improve the paper based on this feedback."}, {"review_id": "Xyfa5xMTfW", "reviewer": "Reviewer_cxPy", "summary": "This paper proposes an automated method for discovering novel language model architectures based on multi-agent large language models, simulating the full research process of design, implementation, pretraining, and evaluation as in real scientific work. The Genesys system employs a \"ladder of scales\" strategy, progressively expanding model sizes (from 14M to 350M parameters) while tightening training budgets, to propose, adversarially review, and implement architecture designs. To improve design generation efficiency, Genesys introduces a genetic programming-based backbone structure, achieving about an 86% higher success rate compared to traditional direct prompting methods. The experiments discovered a total of 1,162 new architecture designs, with 1,062 fully validated through the entire process; the best designs outperformed existing architectures like GPT-2 and Mamba-2 on 6 out of 9 common benchmarks. Additionally, through systematic ablation studies and formal analysis, the paper deeply explores the impact of design components on automated discovery efficiency, providing important insights for efficient automated architecture discovery.", "questions": "1. The current Genesys system allows three types of modification operations: mutation, crossover, and design from scratch. Could you please explain the proportion and effectiveness of these three different modification operations in the final proposals generated by the system?\n2. The system presets five initial seed architectures as initialization. Is it possible to remove this initialization so that the system can perform literature review and design entirely from scratch, thereby generating novel architectures different from known ones?\n3. Genesys achieves overall language model architecture design and review through interactions among different agents. In this process, have you observed any phenomena like reward hacking by agents? If so, what solutions have been considered or implemented?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths：\n\n1. The paper addresses an important problem with strong motivation: using large language models to model the discovery of novel language model architectures.\n2. It innovatively introduces a genetic programming backbone structure, combining multiple components such as a knowledge engine, to implement a ladder-of-scales approach for discovering new language model architectures.\n3. Comprehensive ablation studies are conducted, demonstrating the overall system’s effectiveness and the rationality of each component’s design.\n\nWeaknesses：\n\n1. The system requires high computational resources, limiting its reproducibility locally and restricting exploration of larger-scale model architectures.", "was_revised": true, "final_justification": "After carefully considering the authors’ response, I have decided to raise my score. All of the concerns I previously raised have been fully addressed.\nI believe the motivation, foundation, and the amount of work presented in this paper are sufficiently novel and substantial. Therefore, I consider the paper worthy of acceptance to NeurIPS.", "rebuttal": "We thank Reviewer cxPy for their exceptionally positive and encouraging review. We are delighted that they found our work to be excellent in its motivation, originality, and execution. We appreciate the opportunity to address the valid point on computational resources and answer the insightful questions raised.\n\n---\n\n## Weaknesses\n\n> **W1:** The system requires high computational resources, limiting its reproducibility locally and restricting exploration of larger-scale model architectures.\n\nWe agree that discovering language model architectures is a computationally intensive task. This high cost is inherent to the problem, as any meaningful evaluation requires pre-training models, often across multiple scales. Our work directly confronts this challenge with a novel strategy designed to dramatically improve efficiency.\n\nOur **\"Ladder of Scales\"** strategy, described in **Sec. 4.3**, is our primary method for making this high-cost search tractable. Instead of wastefully training all candidate designs at the largest scale, this method uses smaller, cheaper scales to progressively filter for the most promising architectures.\n\nAs shown in **Figure 9**, to find the best designs from an initial pool of 1,000 candidates, our system uses a pyramidal approach which starts by training all **1,000** candidates at a small **14M** parameter scale, and the selected **400** to the **31M** scale, then **150** trials at **70M** and **40** trials at **125M**, Finally, only the top **5** candidates are trained at the target **350M** scale.\n\nThis selective, multi-scale evaluation strategy significantly reduces the computational burden compared to a naive brute-force search. Based on our training time estimations for different hardware scenarios in Appendix E.4.1 and Table 19, it reduced the H100 GPU hours required for training all 1000 candidates in the target 350M scale directly from around 38,000 hours to 1100 hours. \n\n\n## Questions \n\n> **Q1:** The current Genesys system allows three types of modification operations: mutation, crossover, and design from scratch. Could you please explain the proportion and effectiveness of these three different modification operations in the final proposals generated by the system?\n\nYes, the paper provides these details in the main text and appendices.\n\n**Proportion of Modification Operations**\n\nThe proportions of the three modification operations were set as a probabilistic choice for the Designer agents. As detailed in **Sec. 4.3**, the probabilities used in our experiments were **75% for mutation, 20% for crossover, and 5% for design-from-scratch**. This distribution was chosen to prioritize the refinement and improvement of existing successful designs (exploitation) while still allowing for the combination of ideas from different lineages and the introduction of radical novelty (exploration).\n\n\n**Effectiveness of Modification Operations**\n\nThe paper does not provide a direct quantitative comparison of which operation leads to the highest fitness gain, as a successful design in a later generation is often the result of a long lineage of different operations, making attribution difficult.\n\nHowever, we can infer the effectiveness from their roles and the resulting evolutionary patterns analyzed in **Appendix E.1.2** :\n* **Mutation (75%)**: This is the primary driver for local, iterative improvement. The high probability reflects a strategy focused on refining successful ideas. This aligns with the \"hubness\" pattern we observed in the \"Full\" system's evolutionary tree (**Figure 8 Right**), where successful designs become parents to many incremental refinements.\n* **Crossover (20%)**: This operation is effective at combining successful \"genes\" from different evolutionary branches. An example that led to a successful final model is shown in **Figure 16**.\n* **Design from Scratch (5%)**: This is the main source of radical novelty, but it's used sparingly due to its higher risk of producing less effective designs. However, one of our top designs VQH (also presented in Appendix G.2) is generated from this mode, despite it can still include other previously discovered designs as references (e.g., one reference is a mutated version of another top design HippoVQ).\n\nIn summary, the high proportion of **mutation** is effective for the steady, exploitative progress that dominates the search, while **crossover** and **design-from-scratch** are crucial for exploration and introducing valuable diversity into the gene pool.\n\n\n\n> **Q2:** The system presets five initial seed architectures as initialization. Is it possible to remove this initialization so that the system can perform literature review and design entirely from scratch, thereby generating novel architectures different from known ones?\n\nYes, the Genesys framework is fully capable of designing architectures without being initialized with seed models. This capability is a core part of its design. \"Design from scratch\" is one of the three fundamental genetic programming operations available to the Designer agents, alongside mutation and crossover. As detailed in the system prompts in **Appendix F**, when this operation is selected, the Proposer agent is **not given a parent architecture to modify**. Instead, its sole task is to perform a literature review using the Knowledge Engine and then propose a completely novel GAU tree from first principles, based on the insights it gathers. As mentioned above, one example is VQH, one of our top designs.\n\nWhile the system can design from a blank slate, we intentionally initialized the primary experiment with five state-of-the-art seeds for a specific methodological reason: our goal was to model the **real-world scientific process**. Researchers rarely start in a vacuum; they build upon, challenge, and improve the current state-of-the-art.\n\nBy providing seeds, we tasked Genesys with the more realistic and arguably more difficult challenge of advancing beyond today's highly optimized architectures, rather than spending significant computational resources rediscovering foundational concepts (like basic attention or normalization layers) that are already well-established.\n\n\n\n> **Q3:** Genesys achieves overall language model architecture design and review through interactions among different agents. In this process, have you observed any phenomena like reward hacking by agents? If so, what solutions have been considered or implemented?\n\nThat's a great question about a crucial aspect of multi-agent systems. Based on the system's design, reward hacking is unlikely due to the nature of the feedback signals.\n\nThe Genesys system has two primary feedback mechanisms, neither of which is easily exploitable in a \"reward hacking\" sense.\n\n**1. Final Fitness Score**\n\nThe ultimate \"reward\" for any discovered architecture is its **empirical performance on held-out downstream benchmarks**. This final fitness score is calculated by the **Verification Engine** based on real-world training and evaluation. Because this score is tied to objective, external tasks, it is not possible for an agent to \"hack\" it in the traditional sense. The architecture either performs well or it doesn't.\n\n**2. Intermediate Design-Phase Signals**\n\nDuring the design process, there are two intermediate signals that guide the agents:\n\n* **Symbolic Checker (SC)**: This is a non-learnable, rule-based system. As detailed in **Appendix B.1.2**, it uses a series of static and runtime checks (e.g., AST parsing, checking for differentiability, causality, and numerical stability) to validate a design. Since these are hard-coded criteria for a valid program, an agent cannot \"hack\" the checker; it must produce code that passes these objective tests.\n* **Reviewer Agent Score**: The Proposer agent's goal is to get a high score from the adversarial Reviewer agent. While this is an agent-agent interaction, we did not observe reward hacking. The Reviewer is tasked with critically assessing **novelty, feasibility, and theoretical soundness** by comparing the proposal against the knowledge base, sibling designs, and past proposals. The diversity of foundation models used for the agents also helps prevent stable, exploitable behaviors from emerging.\n\nIn summary, the final reward is objective, and the intermediate signals are either based on hard-coded rules or a critical, adversarial review process, making the system robust against reward hacking.\n\n\n---\n\nWe are grateful for such a supportive and thorough assessment of our work. Please do not hesitate to let us know if any of our answers can be clarified further. Thank you once again."}, {"review_id": "vnc2C5eEwV", "reviewer": "Reviewer_iHdi", "summary": "This paper presents Genesys, a large-scale, LLM-driven system for automated neural architecture discovery. The system combines a modular architecture representation (the GAU tree), a multi-agent proposal-implementation-verification pipeline, and a distributed evolutionary framework to generate, evaluate, and refine novel model designs. The authors conduct extensive experiments and ablation studies to demonstrate the effectiveness, stability, and practicality of their system.", "questions": "- Is the current framework capable of supporting more flexible units designs? For example, can it accommodate using different block types at different layers within a Transformer, or design parallel block layers that merge information after repeating several blocks?\n\n- Do you think Genesys is capable of producing effective results at larger parameter scales? A brief explanation of your understanding would be appreciated.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths:\n\n- The introduction of GAU trees to factorize neural architecture code into interpretable, discrete units provides a meaningful structure for evolutionary optimization and design comparison.\n\n- The paper is clearly structured with well-defined terminology, making it easy for readers to follow the overall system workflow and module design.\n\n- The work achieves a comprehensive and complex system design through a substantial engineering effort.\n\n- The experiments and evaluation metrics are well-designed and appropriate for demonstrating the system's effectiveness.\n\n## Weakness:\n\n- How does Genesys ensure diversity during the design process? What methods are used or could be used to evaluate the diversity and innovation of the generated designs? Additionally, while the paper demonstrates the ability to design new units from scratch, I am curious whether Genesys can leverage the knowledge engine (KE) to incorporate mathematical or physical principles to create entirely novel module components.\n\n- Ablation experiments are an important part of the scientific process. Does Genesys support such ablation experimentation? Specifically, can it perform joint analyses of multiple designs’ structures and performances to better understand the contribution of certain modules?", "was_revised": true, "final_justification": "After reviewing the authors’ response and other feedback, I have raised my initial score to accept the paper. The paper explores an interesting topic by using multi-agent systems to automatically design language models, thereby broadening traditional NAS and opening a new research direction. Both its motivation and technical design are well-suited for NeurIPS.", "rebuttal": "We thank Reviewer iHdi for their positive, thorough, and insightful review. We are grateful for the excellent questions regarding diversity, ablation support, and architectural flexibility, as they touch upon the core strengths and advanced capabilities of the Genesys framework.\n\n---\n\n## Weaknesses\n\n> **W1:** How does Genesys ensure diversity during the design process? What methods are used or could be used to evaluate the diversity and innovation of the generated designs? Additionally, while the paper demonstrates the ability to design new units from scratch, I am curious whether Genesys can leverage the knowledge engine (KE) to incorporate mathematical or physical principles to create entirely novel module components.*\n\n> *“How does Genesys ensure diversity during the design process?”*\n\nGenesys employs several mechanisms to encourage diversity and prevent premature convergence:\n\n* **Novelty Checks**: The system actively checks for novelty to avoid self-replication.\n    * **Proposal stage**: the Reviewer agent compares the new proposal against previous proposals from the same parents (\"siblings\") and similar past proposals (**Appendix B.1.5**).\n    * **Implementation stage**: the Observer agent assesses the novelty of the generated code against prior and sibling implementations (**Sec. 4.2**).\n* **Exploration-Exploitation Strategy**: The design selection process explicitly balances exploiting promising designs with exploring diverse or less successful ones (**Sec. 4.3**). The quadrant-based selection strategy dedicates a portion of the search to \"Poor & Confident\" and \"Poor & Unconfident\" designs to ensure the search does not get stuck in local optima.\n* **Probabilistic Selection and Restarts**: Probabilistic selection and a random restart mechanism also foster new evolutionary paths (**Appendix C.2**).\n\n> *“What methods are used or could be used to evaluate the diversity and innovation of the generated designs?”*\n\nWe evaluate diversity both quantitatively and qualitatively:\n\n* **Quantitative Analysis**: We analyze the **Evolutionary Tree** structure (**Appendix E.1.2**) to understand discovery patterns. As seen in the comparison between our \"Full\" (**Fig. 8, Right**) and \"w/o Exp.\" (**Fig. 35**) systems, fitness-guided selection creates hubs (exploitation). We also analyze the reuse and adaptation of GAU components (**Appendix E.3.2**) to show how existing ideas are built upon to create new designs.\n* **Qualitative Agent-Based Evaluation**: LLM agents directly assess innovation (**Sec. 4.2**). The **Reviewer agent** scores each proposal's novelty, and the **Observer agent** rates the code's quality and innovation, providing a peer-review-like assessment for each design.\n\n> *“...whether Genesys can leverage the knowledge engine (KE) to incorporate mathematical or physical principles to create entirely novel module components.”*\n\nYes, the system is designed with the flexibility to incorporate deeper scientific principles. The **Knowledge Engine (KE)** already provides academic literature to the Proposer agent, informing its design choices (**Sections 3.2, 4.2**).\n\nThis capability can be extended straightforwardly:\n\n* **Expanding the Knowledge Base**: Augmenting the KE library with papers on mathematical principles (e.g., control theory) or physics, thus hinting the Proposer to draw inspiration from these domains.\n* **Integrating External Tools**: Equipping agents with tools like a symbolic math engine (e.g., Mathematica) to allow them to experiment with mathematical principles while drafting proposals.\n\n> **W2:** Ablation experiments are an important part of the scientific process. Does Genesys support such ablation experimentation? Specifically, can it perform joint analyses of multiple designs’ structures and performances to better understand the contribution of certain modules?\n\nWe thank the reviewer for this excellent question. The ability to perform ablations and analyze component contributions is a key capability of our system.\n\nYes, Genesys is explicitly designed to support such experimentation, and we have already conducted a preliminary analysis demonstrating its ability to understand the contributions of specific modules.\n\n**1. Support for Ablation via GAU Factorization**\n\nThe core design of Genesys makes architectural ablation straightforward. Our system is built on the principle of factorizing every architecture into a tree of **Generalized Autoregressive Units (GAUs)**.\n\n* **Ablation as Mutation**: An ablation is equivalent to a targeted **mutation**. To ablate a module (e.g., `VectorQuantization` from VQH), the Proposer agent replaces that GAU with a simpler one, like an identity function.\n* **Unit-Based Control**: Because the entire search process operates on these swappable, high-level units, the system provides the necessary control to systematically modify, remove, or replace any component of a discovered architecture for targeted analysis.\n\n**2. Joint Analysis of Module Contributions**\n\nAnalyzing multiple designs to understand module contributions is a key scientific goal of our work. We present a direct proof-of-concept for this capability in **Appendix E.3.3 (\"Unit-Performance Relation\")**.\n\n* **\"Bag-of-Units\" Representation**: We represented each of our 1,062 designs as a **\"Bag-of-Units\"** (BoU), where the \"words\" are the names of the GAUs used in its architecture.\n* **Predicting Performance from Structure**: We then trained a Naive Bayes classifier to predict a design's performance on various downstream tasks based on its BoU representation.\n* **Key Finding**:We found a clear predictive link between GAU components and task performance, achieving an overall fitness level with an F1 score of 0.658 on held-out data (Table 17) and showed predictive power on 19 different downstream tasks (Table 18).\n\nThis result demonstrates that Genesys can perform a joint analysis across its entire discovery history to learn which architectural modules are correlated with success. As we conclude in the paper, this capability opens the door for more sophisticated, insight-driven discovery, where the system can learn design principles that guide future exploration.\n\n## Questions\n\n> **Q1:** Is the current framework capable of supporting more flexible units designs? For example, can it accommodate using different block types at different layers within a Transformer, or design parallel block layers that merge information after repeating several blocks?\n\nYes, the framework's design is highly flexible and supports both sophisticated patterns. This is a core capability, enabled by the **`block_loc` parameter** and the intermediate variable dictionary **`Z`**.\n\n**1. Different Block Types Across Layers**\n\nThe `block_loc` parameter `(layer_idx, n_block)` informs each GAB/GAU of its position in the network stack, allowing for layer-specific logic (**Appendix F.4.1**). A Proposer agent can use conditional logic based on `layer_idx` to instantiate different child units at different layers (e.g., varying architectures or operations between blocks).\n\n  * **Example Usage: GPT-Mamba hybrid model**: \n    ```python\n    # Pseudo-code for a block's __init__ method\n    if self.block_loc[0] % 2 == 0:\n        # Even layers are GPT-style\n        self.layer = GPT_Attention_Unit(...) \n    else:\n        # Odd layers are Mamba-style\n        self.layer = Mamba_SSM_Unit(...) \n    ```\n\n**2. Parallel Blocks with Merging**\n\nParallel blocks with merging are supported via the flexible `Z` dictionary. As defined in our base classes (**Fig. 11, Appendix F.1.1**), `Z` is a key-value carrier for information, like tensors from parallel branches, passed between blocks. One block can add outputs to `Z`, and a subsequent block can retrieve and merge them, using `layer_idx` to trigger merging periodically.\n\n  * **Example Usage: Multi-branch model**:\n    ```python\n    # Pseudo-code for a block's _forward method\n\n    # --- In blocks where merging happens (e.g., every 4th layer) ---\n    if self.block_loc[0] % 4 == 0 and self.block_loc[0] > 0:\n        # Retrieve outputs from parallel branches stored in Z\n        branch1_out = Z.get('branch1_output', 0)\n        branch2_out = Z.get('branch2_output', 0)\n        \n        # Merge the main path with the side channels\n        X = X + branch1_out + branch2_out \n\n    # --- In all blocks, run parallel branches and store outputs in Z ---\n    z_out['branch1_output'] = self.branch1_unit(X, **Z) \n    z_out['branch2_output'] = self.branch2_unit(X, **Z)\n\n    return X, z_out\n    ```\n\nThese core constructs provide a powerful and extensible framework for exploring a vast and complex architectural design space.\n\n> **Q2:** Do you think Genesys is capable of producing effective results at larger parameter scales? A brief explanation of your understanding would be appreciated.\n\nYes, we are confident Genesys can be effective at larger scales, based on its design and experimental results.\n\n* **Stable Evolutionary Progress**: The \"Full\" system shows a stable, positive improvement trajectory (**Sec. 5.1, Table 1, Fig. 8**). A process that consistently improves upon strong baselines is likely to continue this progress at larger scales.\n* **Efficient Multi-Scale Methodology**: The **\"Ladder of Scales\"** strategy (**Sec. 4.3**) is designed for efficient discovery, using smaller scales to filter for promising architectures before committing to expensive, large-scale training. This design is fundamental to making a large-scale search tractable and effective.\n* **Architectures Follow Scaling Laws**: As shown in our analysis (**Appendix E.3.1, Fig. 43**), discovered architectures follow expected scaling laws, with training loss decreasing as model size increases. This suggests the architectural merits we found will persist at larger scales.\n\n---\n\nThank you again for the engaging and thoughtful review. Please do not hesitate to reach out if you have any follow-up questions; we would be happy to discuss these aspects of our system further."}, {"review_id": "MVXmORWsag", "reviewer": "Reviewer_tRdq", "summary": "The paper presents Genesys, a multi-agent, LLM-driven framework, for discovering language model architectures via evolutionary search and unit-based agent design. The core idea is to use LLMs themselves to design, propose, verify, and improve new LM architectures beyond standard transformers. The system discovers 1,062 novel architectures across different parameter scales, and find the best designs to be  competitive with known architectures like GPT2 and Mamba2.", "questions": "1. What tasks are you choose for evaluation and why?\n2. Will you also do some search on training hyper-parameters?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Clear motivation and important topic.\n2. Extensive experiments and full ablation study for validating effects of different modules.\n\nWeaknesses:\n1. The whole paper is more like a system design report. While the whole system is a bit complicated, its presentation lacks clear organization, making it difficult to follow up.\n2. The result transparency is limited. Not much details on how they compute the fitness scores (like what tasks? How tested? Model training and evaluation configurations?).\n3. The scale is still limited, with max scale 350M.", "was_revised": true, "final_justification": "My main concerns about the paper still focusing on the overall contributions and writing. The concerns about evaluation and training details are addressed by the authors' response. Therefore, I have raised the score accordingly.", "rebuttal": "We thank Reviewer tRdq for their valuable feedback. We agree that the paper's organization can be improved to better highlight our scientific contributions, and we appreciate the chance to provide a clear summary of the extensive experimental details contained in the paper and its appendices.\n\n---\n\n## Weaknesses\n\n> **W1:** The whole paper is more like a system design report. While the whole system is a bit complicated, its presentation lacks clear organization, making it difficult to follow up.\n\n\nWe thank the reviewer for their feedback on the paper's structure and contribution framing. We take the clear writing of our paper seriously and will improve the organization in the camera-ready version to make the scientific contributions more prominent.\n\n**Our Scientific Contributions**\n\nWe acknowledge that the paper includes significant detail about the Genesys system. This is because the system itself serves as a novel scientific instrument for a complex task. However, our primary goal is to use this system to generate new insights. We believe our contributions go well beyond a system report and include:\n\n* **Methodological Innovations**: We introduce key research contributions to make ASD in this domain tractable including: (1) a novel, unit-based **Viterbi-style search** for code generation, which we prove formally to be exponentially more efficient than standard methods both empirically in Table 2 and formally in Appendix A; (2) a **\"Ladder of Scales\"** strategy for managing the prohibitive cost of evaluation; (3) and our system design of Genesys/LMADE which emulating human research workflow.\n* **Empirical Scientific Insights**: We use the system to conduct experiments that yield new knowledge. Our **ablation studies** in Table 1 empirically quantify the value of knowledge and experimental feedback in the discovery process. Furthermore, our analysis of the **evolutionary tree** reveals emergent discovery patterns in Appendix E.1.2, and we pioneer an analysis directly **linking architectural units to downstream performance** in Appendix E.3.3.\n* **Tangible Discoveries**: The system is not just a theoretical construct; it produced **1,062 verified novel architectures**. The top designs are competitive with strong human baselines, demonstrating the method's effectiveness.\n\nBeside, we will release all code, discovery artifacts, and experiment runs (via wandb), including our console for running new experiments, upon publication.\n\n**Improving Paper Organization and Readability**\n\nTo improve clarity, we will make the following changes in the camera-ready version:\n\n* **Itemized Contributions**: We will add a dedicated, itemized list of our main contributions in the **Introduction**, providing a clear, upfront summary of our methodological and scientific takeaways.\n* **Improved Signposting**: We will enhance the \"road map\" at the beginning of major sections. For instance, before Sec. 4, we will add a sentence explaining that the section describes the instrument used to generate the scientific insights presented in Sec. 5.\n\nWe are confident these improvements will better highlight our scientific contributions and make the paper easier to follow.\n\n\n\n> **W2:** The result transparency is limited. Not much details on how they compute the fitness scores (like what tasks? How tested? Model training and evaluation configurations?).\n\nWe thank the reviewer for this question. The paper provides extensive details on fitness calculation, tasks, and configurations, primarily in Sections 3.1, 5.3, and Appendices B and C. Here is a summary of the key information.\n\n**1. Fitness Score Computation and Tasks**\n\nThe method for calculating fitness is formally defined and consistently applied throughout the experiments.\n\n* **Fitness Formula**: The fitness score ($\\mathcal{F}$) is the average empirical performance across **M** downstream tasks and **K** model scales (Sec. 3.1).\n* **Evaluation Tasks**: Models are evaluated on **29 selected LM-Eval benchmarks** (listed in Table 16). The nine most informative tasks were chosen for final comparison (Table 3) based on a statistical analysis (Appendix E.3.1) considering result variance and example count.\n* **Evaluation Metric**: The primary metric for comparison is **zero-shot accuracy (%)** (Table 3).\n\n**2. Model Training and Evaluation Configurations**\n\nWe provide full details in Appendix C for reproducibility.\n\n* **Training Corpus**: All models were pretrained on a custom 34.78B token **SmolLM-1/8-Corpus**, which is a high-quality subset of the SmolLM corpus filtered for educational content to improve training efficiency (details in Appendix C.1.1, Table 8).\n* **Training Settings**: We detail the hardware environment in Appendix C.1.2, model experiment settings in C.1.3, and verification engine hyperparameters in Table 11.\n* **Evaluation Framework**: We use a customized **LM-Eval framework**. An **Auto-Tuner** ensures models fit the target parameter scale and tunes gradient accumulation to prevent OOM errors. A **Runtime Checker** terminates failed runs early (Details in Appendix B.1.4).\n\nIn addition, we provide full details for the evolutionary system and agent setups in Appendix C.2 and Appendix B.1.5. Moreover, we will release all code, discovery artifacts, and experiment runs (via wandb), including our console for running new experiments, upon publication.\n\n\n\n> **W3:** The scale is still limited, with max scale 350M.\n\nWe agree that 350M parameters is not a large language model by today's standards. Our choice of this scale was deliberate and is well-suited for the primary goals of this study.\n\n**Justification for the 350M Scale**\n\n* **Focus on Validating the Discovery Method**: The main contribution of this paper is not a single, large-scale model, but rather the validation of **Genesys**, a novel ASD system for discovering architectures. The 14M-350M parameter range is large enough to reveal meaningful differences between architectures and demonstrate stable performance trends, while remaining computationally tractable for an extensive search involving over 1,000 unique, trained designs.\n\n* **Relevance to Architectural Research**: This scale is typical and useful for comparative architectural studies. Many recent and impactful models included the scale of 125M and 350M as a part of their standard evaluation, such as **Mamba2** and **RWKV7** compared in our study.\n\n* **Demonstrating Scaling Trends**: Our \"Ladder of Scales\" strategy is predicated on the idea that performance trends at smaller scales correlate with those at larger scales. Our results support this, with discovered models showing consistent loss reduction as they scale up (as seen in Figure 43), suggesting the architectural merits we observe are not confined to a single scale.\n\nIn summary, the 350M scale provides a meaningful and computationally feasible setting to validate our discovery methodology and demonstrate its ability to produce architectures that are competitive with the state-of-the-art in this domain.\n\n\n## Questions\n\n> **Q1:** What tasks are you choose for evaluation and why?\n\nAs discussed above, we evaluates models on **29 selected LM-Eval benchmarks**, those tasks are selected by performing a basic clearing from the LM-Eval task list available in https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/README.md: removing overly difficult tasks (such as MT Bench) for the considered model scales, or the ones that are too time consuming (such as requiring long-text generation), or the ones that cannot be directly applied with autoregressive LMs (such as the ones designed for BERT), or erroneous ones.\nThese tasks are listed in Table 16. For the tasks in Table 3, we selected the nine most informative benchmarks based on a statistical analysis detailed in Appendix E.3.1, considering factors like a high standard deviation of results and a sufficient number of examples. \n\n> **Q2:** Will you also do some search on training hyper-parameters?\n\nNo, the current system does not perform an automated search for optimal training hyperparameters (e.g., learning rate, optimizer settings) for both evolution and model evaluations in Sec. 5.3 Table 3. Instead, we adopted a standardized protocol to ensure fair and reproducible comparisons, as described in Sec. 3.2.\n\n**Our Hyperparameter Protocol**\n\n* **Standardized Training for Discovered Models**: To ensure consistency, all discovered architectures were trained using a fixed set of standard pre-training hyperparameters. These settings, including the AdamW optimizer and a cosine learning rate scheduler, are detailed in Appendix B.1.4 and Table 11.\n\n* **Role of the Auto-Tuner**: We do use an **Auto-Tuner**, but its role is not to optimize training hyperparameters for performance. As described in Appendix B.1.4, its purpose is to:\n    1.  Adjust architectural parameters (number of layers and embedding dimension) to ensure a model correctly fits the target parameter count (e.g., 350M).\n    2.  Tune the gradient accumulation steps to prevent out-of-memory errors, ensuring the model can train on the available hardware.\n\n* **Official Settings for Baselines**: For the final model evaluation in Sec. 5.3, we trained the human-designed baselines (like Mamba2 and RWKV7) using the **official hyperparameters published in their original papers or code repositories**. This process, detailed in Appendix C.1.3, was chosen to ensure that the baseline models were evaluated under their own optimal, specialized conditions.\n\n---\n\nWe are confident that the planned organizational changes and the clarifications provided here will resolve the concerns about clarity and transparency. Please do not hesitate to contact us if any aspect of our response requires further explanation."}]

Author rebuttal:

Meta-review: This paper proposes solving the problem of LLM architecture search by using LLMs in agentic setups. Essentially, at each stage and scale, the LLM agents propose, generate code, train and evaluate LMs and then select promising architectures for subsequent evolution. Although this appears similar to NAS, the concept is slightly different, since everything from ideation to development to iteration is done via LLMs. Overall the reviewers have indicated that this is an interesting work as well with the major complaints centered around the scale of models to be tested and the maximum model sizes that can be tested. However, given the agentic approach for architecture discovery, I believe that this work will be of interest to, at the very least, the agentic community.
Decision: Accept (spotlight)