Title: Language Modeling by Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Autonomous Scientific Discovery

Large Language Model Agents

Genetic Programming
Abstract: *Can we leverage LLMs to model the process of discovering novel language model (LM) architectures?* Inspired by real research, we propose a multi-agent LLM approach that simulates the conventional stages of research, from ideation and literature search (proposal stage) to design implementation (code generation), generative pre-training, and downstream evaluation (verification). Using ideas from scaling laws, our system *Genesys* employs a *Ladder of Scales* approach; new designs are proposed, adversarially reviewed, implemented, and selectively verified at increasingly larger model scales (14M$\sim$350M parameters) with a narrowing budget (the number of models we can train at each scale). To help make discovery efficient and factorizable, Genesys uses a novel genetic programming backbone, which we show has empirical advantages over commonly used direct prompt generation workflows (e.g., $\sim$86\% percentage point improvement in successful design generation, a key bottleneck). We report experiments involving 1,162 newly discovered designs (1,062 fully verified) and find the best designs to be competitive with known architectures (e.g., outperform GPT2, Mamba2, etc., on 6/9 common benchmarks).  We couple these results with comprehensive system-level ablations and formal results, which give broader insights into the design of effective autonomous discovery systems.


Peer reviews:
comments: SUMMARY:
The paper presents Genesys, a multi-agent, LLM-driven framework, for discovering language model architectures via evolutionary search and unit-based agent design. The core idea is to use LLMs themselves to design, propose, verify, and improve new LM architectures beyond standard transformers. The system discovers 1,062 novel architectures across different parameter scales, and find the best designs to be  competitive with known architectures like GPT2 and Mamba2.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Clear motivation and important topic.
2. Extensive experiments and full ablation study for validating effects of different modules.

Weaknesses:
1. The whole paper is more like a system design report. While the whole system is a bit complicated, its presentation lacks clear organization, making it difficult to follow up.
2. The result transparency is limited. Not much details on how they compute the fitness scores (like what tasks? How tested? Model training and evaluation configurations?).
3. The scale is still limited, with max scale 350M.

QUESTIONS:
1. What tasks are you choose for evaluation and why?
2. Will you also do some search on training hyper-parameters?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Genesys, a large-scale, LLM-driven system for automated neural architecture discovery. The system combines a modular architecture representation (the GAU tree), a multi-agent proposal-implementation-verification pipeline, and a distributed evolutionary framework to generate, evaluate, and refine novel model designs. The authors conduct extensive experiments and ablation studies to demonstrate the effectiveness, stability, and practicality of their system.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n- The introduction of GAU trees to factorize neural architecture code into interpretable, discrete units provides a meaningful structure for evolutionary optimization and design comparison.\n\n- The paper is clearly structured with well-defined terminology, making it easy for readers to follow the overall system workflow and module design.\n\n- The work achieves a comprehensive and complex system design through a substantial engineering effort.\n\n- The experiments and evaluation metrics are well-designed and appropriate for demonstrating the system's effectiveness.\n\n## Weakness:\n\n- How does Genesys ensure diversity during the design process? What methods are used or could be used to evaluate the diversity and innovation of the generated designs? Additionally, while the paper demonstrates the ability to design new units from scratch, I am curious whether Genesys can leverage the knowledge engine (KE) to incorporate mathematical or physical principles to create entirely novel module components.\n\n- Ablation experiments are an important part of the scientific process. Does Genesys support such ablation experimentation? Specifically, can it perform joint analyses of multiple designs’ structures and performances to better understand the contribution of certain modules?\n\nQUESTIONS:\n- Is the current framework capable of supporting more flexible units designs? For example, can it accommodate using different block types at different layers within a Transformer, or design parallel block layers that merge information after repeating several blocks?\n\n- Do you think Genesys is capable of producing effective results at larger parameter scales? A brief explanation of your understanding would be appreciated.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an automated method for discovering novel language model architectures based on multi-agent large language models, simulating the full research process of design, implementation, pretraining, and evaluation as in real scientific work. The Genesys system employs a \"ladder of scales\" strategy, progressively expanding model sizes (from 14M to 350M parameters) while tightening training budgets, to propose, adversarially review, and implement architecture designs. To improve design generation efficiency, Genesys introduces a genetic programming-based backbone structure, achieving about an 86% higher success rate compared to traditional direct prompting methods. The experiments discovered a total of 1,162 new architecture designs, with 1,062 fully validated through the entire process; the best designs outperformed existing architectures like GPT-2 and Mamba-2 on 6 out of 9 common benchmarks. Additionally, through systematic ablation studies and formal analysis, the paper deeply explores the impact of design components on automated discovery efficiency, providing important insights for efficient automated architecture discovery.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n\n1. The paper addresses an important problem with strong motivation: using large language models to model the discovery of novel language model architectures.\n2. It innovatively introduces a genetic programming backbone structure, combining multiple components such as a knowledge engine, to implement a ladder-of-scales approach for discovering new language model architectures.\n3. Comprehensive ablation studies are conducted, demonstrating the overall system’s effectiveness and the rationality of each component’s design.\n\nWeaknesses：\n\n1. The system requires high computational resources, limiting its reproducibility locally and restricting exploration of larger-scale model architectures.\n\nQUESTIONS:\n1. The current Genesys system allows three types of modification operations: mutation, crossover, and design from scratch. Could you please explain the proportion and effectiveness of these three different modification operations in the final proposals generated by the system?\n2. The system presets five initial seed architectures as initialization. Is it possible to remove this initialization so that the system can perform literature review and design entirely from scratch, thereby generating novel architectures different from known ones?\n3. Genesys achieves overall language model architecture design and review through interactions among different agents. In this process, have you observed any phenomena like reward hacking by agents? If so, what solutions have been considered or implemented?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes Genesys, a multi-agent system that autonomously discovers new language model architectures using LLMs. The pipeline generates, verifies, and evolves models across scales using a unit-based code representation. Genesys is demonstrated to produce over 1000 verified designs, with some architectures outperforming GPT2 and Mamba2 on standard benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1. Generally, the work is clearly written and easy to follow.\n\n2. This work demonstrates good engineering efforts in building an ASD system for a specific purpose.\n\n**Weakness**\n\n1. The major concern with this work is the limited technical novelty and limited research insights offered to the community. This work is more like a demonstration of a successful engineering effort to build an ASD system, while the scientific aspects of LM agents and new insights are insufficient, limiting its value to the research community. In case I missed something, the authors are expected to summarize the insights and major contributions of this work.\n\n2. Another major concern is the lack of an apple-to-apple comparison to demonstrate the effectiveness of LLMs. Simple mutation/crossover, as done in traditional NAS methods, can replace the role of LLMs. A comparison between LLM-based ASD systems and rule-based mutation/crossover in traditional NAS should be provided to show the benefits of LLMs. The effectiveness of LLMs remains unclear to me.\n\n3. The searched architectures are highly sensitive to the search space. The authors are expected to (1) improve the clarity of the search space definition, (2) show the architectural details of the best searched architectures, and (3) ablate the search space choices to validate the robustness of the ASD system.\n\n4. Table 3 is unclear to me, and I do not understand where the better performance on 6 out of 9 benchmarks comes from. It seems the single best model searched in this work is very comparable to GPT.\n\nQUESTIONS:\n1. Whether the GPT-style architecture is in the knowledge base and serves as a seed architecture of the evolution process? If so, the improvement of the finally searched architecture is relatively minor.\n\n2. Will the proposed system achieve better results than traditional NAS methods that use mutation/cross-over instead of LLM proposers?\n\n3. Could you show the single-best architecture that the proposed system searched? How difference it is compared to current LM architectures? Any new architecture design insights we can learn from it?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}