A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence
Authors: Huan-ang Gao
Jiayi Geng
Wenyue Hua
Mengkang Hu
Xinzhe Juan
Hongzhang Liu
Shilong Liu
Jiahao Qiu
Xuan Qi
Qihan Ren
Yiran Wu
Hongru WANG
Han Xiao
Yuhang Zhou
Shaokun Zhang
Jiayi Zhang
Jinyu Xiang
Yixiong Fang
Qiwen Zhao
Dongrui Liu
Cheng Qian
Zhenhailong Wang
Minda Hu
Huazheng Wang
Qingyun Wu
Heng Ji
Mengdi Wang
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks but remain fundamentally static, unable to adapt their internal parameters to novel tasks, evolving knowledge domains, or dynamic interaction contexts. As LLMs are increasingly deployed in open-ended, interactive environments, this static nature has become a critical bottleneck, necessitating agents that can adaptively reason, act, and evolve in real time. This paradigm shift ---from scaling static models to developing self-evolving agents --- has sparked growing interest in architectures and methods enabling continual learning and adaptation from data, interactions, and experiences. This survey provides the first systematic and comprehensive review of self-evolving agents, organizing the field around three foundational dimensions --- what to evolve, when to evolve, and how to evolve. We examine evolutionary mechanisms across agent components (e.g., models, memory, tools, architecture), categorize adaptation methods by stages (e.g., intra-test-time, inter-test-time), and analyze the algorithmic and architectural designs that guide evolutionary adaptation (e.g., scalar rewards, textual feedback, single-agent and multi-agent systems). Additionally, we analyze evaluation metrics and benchmarks tailored for self-evolving agents, highlight applications in domains such as coding, education, and healthcare, and identify critical challenges and research directions in safety, scalability, and co-evolutionary dynamics. By providing a structured framework for understanding and designing self-evolving agents, this survey establishes a roadmap for advancing more adaptive, capable, robust, and versatile agentic systems in both research and real-world deployments, and ultimately sheds light on the realization of Artificial Super Intelligence (ASI) where agents evolve autonomously and perform beyond human-level intelligence across a wide array of tasks.

Published in Transactions on Machine Learning Research (01/2026)
A Survey of Self-Evolving Agents What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence
Huan-ang Ga Jiayi Geng"l, Wenyue Hua Mengkang Hu Xinzhe Juan Hongzhang Liust, Shilong Liu Jiahao Qiu Xuan Qit, Qihan Ren"t, Yiran Wutt Hongru Wang ftx Han Xiao"i, Yuhang Zhou"t, Shaokun Zhang Jiayi Zhang", Jinyu Xiang, Yixiong Fang", Qiwen Zhaos, Dongrui Liu", Cheng Qian?, Zhenhailong Wang', Minda Hu', Huazheng Wang", Qingyun Wu', Heng Ji", Mengdi Wang
Princeton University, Princeton AT Lab, " "Tsinghua University, Cornegie Mellon University, c Univeessity of Sydney, Shannhhi Jiao Tong University, "Pennsylvania. State University, University of Michigon, Oregon State Universily, The Chinese Universily of Hong Kong. Fuudan Universily, The Hong Kong Universily of Science and Technology (Guangzhou), "hhe University of Hong Kong, "University of California, Santa Berbera, University of California San Diego, UUniversity of Edinburgh, Uniivrssit of Illinois Urbana- Chammpaign
Github Repo: http:://ggthibb..mm//aaaaaaal//l httpChaaaeee///SLL--vvolriigneea
'Equal contribution and the order is determined alphabetically, "Corresponding Author
Reviewed on OpenReview:
Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks but remain fundamentally static, unable to adapt their internal parameters novel tasks, evolving knowledge domains, dynamic interaction contexts. As LLMs are increas- ingly deployed in open-ended, interactive environments, this static nature has become critical bottleneck, necessitating agents that can adaptively reason, act, and evolve in real lime. This paradigm shift from scaling static models to developing self-evolving agents has sparked growing interest in architectures and methods enabling continual learning and adaptation from data, interactions, and experiences. This survey provides the first sys- tematic and comprehensive review of self-evolving agents, organizing the field around three foundational dimensions - what to evolve, when to evolve, and how to evolve. We examine evolutionary mechanisms across agent components (e.g., models, memory, tools, architec- ture), categorize adaptation methods by stages (c.g., intra-test-time, inter-test-time), and analyze the algorithmic and architectural designs that guide evolutionary adaplation (e.g,, scalar rewards, textual feedback, single-agent and multi-agent systems). Additionally, we analyze evaluation metrics and benchmarks tailored for self-cvolving agents, highlight appli- cations in domains such coding, education, and healthcare, and identify critical challenges and research directions in safety, scalability, and co-evolutionary dynamics. By providing structured framework for understanding and designing self-evolving agents, this survey establishes a roadmap for advancing more adaptive, capable, robust, and versatile agentic systems in both research and real-world deployments, and ultimately sheds light on the real- ization of Artificial Super Intelligence (AST) where agents evolve autonomously and perform beyond human-level intelligence across wide array of tasks.

---

Published in Transactions on Machine Learning Research (01/2026)
Planning Tool Calling
Learning & Evalsson Mechaniere
Explored
Alla Godel agent
amollagents
Workslow Construction
Menus
Level Intellgence
CPT-4
ChaiGPT agent
Cluude44
Self-evolving Agents
Next-generation Agentic Intel vnlelieence
Foundation A Agents
Learning from Feedback Experiance
LLMs
Execution via Tools & Planning
Our Survey
Language Understan sladdiggn & Generation
A conceptual trajectory illustrating the progression from large language models (LLMs) to foundation agents, and then to self-evolving agents our focus, and ultimately toward the hypothetical Artificial Super Intelligence (ASI). Along this path, intelligence and adaptivity increase, marking a shift toward more autonomous and agentic AT systems. The future directions beyond self-evolving agents remain open and subject ongoing exploration.
Figure 1:
1 Introduction
'It is not the most intellectual of the species that survives; not the strongest that survives; but the species that survives is the one that is able best to adapt and adjust the changing Charles Darwin
environment in which it finds itself."
Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. Yet, they remain fundamentally static (Luo et al. 2025a). unable to adapt their internal parameters when encountering novel tasks, evolving knowledge domains, or dynamic interaction contexts. As LLMs are incrcasingly deployed in open-ended, interactive environments, this limitation becomes a critical bottleneck. In such settings, conventional knowledge retrieval mechanismus prove inadequate, giving rise to agents capable of dynamically adapting their perception, reasoning, and actions in real time. This emerging need for dynamic, continual adaptation signals a conceptual shift in artificial intelligence: from scaling up static models to developing self-evoluing agents. While establishod techniques like Supervised Fine- Fine-Tuning (SFT) and Reinforcement. Learning (RI.) provide the mechanisms for improvement, define self-evolution not merely by the algorithms used, but by the locus of autonomy. Unlike traditional pipelines where human engineers curate data and schedule updates, a self-evolving agent is capable of continuously lcarning from new data, interactions, and experiences in real-time, leading to systems that are more robust, versatile, and capable of tackling complex, dynamic real-world problems (Wang et al. 2024aa). This shift is currently driving us toward a promising and transformative path to Artificial Super Intelligence (ASI), where the agents not only can learn and evolve from experience with an unpredictable speed but also perform or above human-level intelligence across a wide array of tasks (Wang et al., 2025g).
Unlike static LLMs, which remain constrained by their inability to adapt to novel and evolving contexts, self- evolving agents are designed to overcome these limitations by continuously learning from real-world feedback. This progression reshapes our understanding of agents. Self-evolving agents, as a core concept, represent a significant step forward in the cvolution intelligent systems, acting as intermediaries that pave the way for more adaptive and autonomous AI, as shown Figure]] [ Recent research initiatives have increasingly focused on developing adaptive agent architectures capable of continually learning and adapting from experience, such recent advancements in agent frameworks (Yin et al., 2025), prompting strategics (Fernando et al. 2023 and different optimization ways to evolve. Notwithstanding these advances, existing surveys predominantly address agent evolution subsidiary component within comprehensive agent taxonomies. Previous surveys primarily provide systematic overviews of general agent development, while offering limited coverage of self-
evolving mechanisms across constrained scenarios in self-cvolving agents [Luo et al. 2025a Liu et al. 2025a
Thhi quote is widely attributed to Charles Darwin, but it does not appear verbatim in his writings. The phrasing is believed originate from Professor Leon C. Megginson, who paraphrased Darwin's idcas. Despite its frequent misattribution, the quote effectively captures the essence Darwinian evolution and has since been popularized in both scientific and managerial literature.
2

---

Published in Transactions on Machine Learning Research (01/2026)
For example, Luo et al. (2025a) discuss several ways to evolve, such as self-learning and multi-agent co- evolution, while Liu et al.
(2025a) explicitly introduce the evolution in terms of different components of agents, such as tools and prompts. Moreover, some studies focus specifically on the evolution of language models themselves (Tao et al. 2024 rather than on the broader concept of agents. However, these works address isolated componenis rather than the holistic agent system. Therefore, there no systematic survey devoted to a dedicated, comprehensive investigation self-evolving agents as a first-class research paradigm. This gap has left fundamental questions underexplored: What aspects of an agent should evolve? When should adaptation occur? And how should that evolution be implemented in practice?
To the best of our knowledge, this is the first systematic and comprehensive survey focusing on self-cvolving agents, offering & clear roadmap for both theoretical inquiry and practical deployment. However, given that this represents a rapidly forming research area where conceptual boundaries are still being actively negotiated within the community, we frame this survey as a guiding synthesis rather than a review of a fully estab- lished paradigm. Instead of enforcing rigid boundaries, we aim structure the heterogeneous mechanisms emerging in the community into coherent framework. We organize our analysis around three foundational questions what, when, and how to evolve and provide structured framework for understanding each. Specifically, we systematically examine individual agent components, including the model, memory, tools and corresponding workflow, investigating their distinct evolutionary mechanisms (what to evolve of agent in Section 3); then we divide existing evolving methods according to different temporal stages with different learning paradigns such as supervised fine-tuning, reinforcement learning and inference-time evolving (when to evolve in Section 4). We finally summarize different signals to guide the evolution of agents, such as textual feedback scalar rewards, and also different architectures of agents to evolve, such as single-agent and multi-agent evolution (how to evolve in Scction 5). Furthermore, and benchmarks to track existing advancements of self-evolving agents, emphasizing the importance of co- evolution between evaluation and agents (Section 6). We also examine emerging applications in domains such as coding, education, and healthcare, where contimual adaptation and evolution are essential (Section 7). Finally, we identify persistent challenges and outline promising research directions to guide the development of self-evolving agents (Section 8). Through this systematic decomposition of self-evolutionary processes across orthogonal dimensions, we provide a structured and practical framework enabling researchers to sys- tematically analyze, compare, and design more robust and adaptive agentic systems. To sum up, our key contributions are as follows:
review certain evaluation metries
We establish a unified theoretical framework for characterizing self-evolutionary processes in agent systems, anchored around three fundamental dimensions: what evolves, how evolves, and when it evolves, providing clear design guidance for future self-evolving agentic systems.
We further investigate the evaluation benchmark or environment tailored for self-evolving agents, highlighting emerging metrics and challenges related to adaptability, robustness, and real-world complexity.
We showease several key real-world applications across various domains, including autonomous soft- ware engineering, personalized education, healthcare, and intelligent virtual assistance, illustrating the practical potential of self-evolving agents.
We identify critical open challenges and promising future research directions, emphasizing aspects like safcty, personalization, multi-agent co-cvolution, and scalability.
In doing so, our survey provides researchers and practitioners with a more structured laxonomy for under- standing, comparing, and advancing research of self-evolving agents from different perspectives. As LLM- basod agents are increasingly integrated into mission-critical applications, understanding their evolutionary dynamics becomes essential, extending beyond academic research to encompass industrial applications, reg- ulatory considerations, and broader societal implications.
3

---

Published in Transactions on Machine Learning Research (01/2026)
SPA
Reseeding Sell-
2022 PAre ql
Policy
ToxtL wereeiiiinn rall
SEL
PaBa Bugae
Wrang At Xinng piraa
Siitee Wang
Blodel
Amantion Hi
Leooon
2023
Madaan pizan crpelin
Comeetmnn Warrl 2023 sten
R00 2022 TLGENN
Wang
Finng
Memuory
Mamarz
Phann packur aichelie
Eayur Ahso et
ICB
Context
Pro
2023
tran el
Himar
Presupt
Biattar
Wane
pirer -ere Fin Wong piray Erongem your di (02a0)
What to evolve
Yoraae co ol
J22in Alia Alio Quacin Poanl ATLASS Houne partta
Creation
Masterry
20254
Tooitoginn (Sttbokk Ll. 20255.
Tooll 2024
ToolCee Wine Pront Ageut Slunee
2095
Dacain Gode
Selection
Muti (nc Plason or 200 COLT (2
Zhang
Sootingtum
AcoutSonarel Shang al Foant Darnin Gofel Machine (ehang
Single-Agcai
HE Al 221233 Crofii Atts (Ti et al parnh Alnhhu
Evol low
MASS
Architecture
APlour Zhars al botte ADAS [ii
Antolloor Tall wi
202te
Maulii-Ageul
CFTS Suakem Peniee 22124) Jeaerles oru.
JCL
Ludduxin Sinnn 20r1) SELE Madaan
Avzila AdaPlamerr
Intraetrer-time eil-evelation
SPT
SIFT [Laboooor
Zoolggr
LADDER Himonds Yoallyaam
2023 Tul Zuo
2026
Selt cvolving Agent
SFT
SBLE
2024
Strme
(022brr Chir
2023b
allfevolation
AL
RACCEN
SRd. (gt sin purce Ergints 15x1 al. 2024
Learning I Line Human Zhang
2025a Wel
Relloxion Siinn pin paea)
Plannce Sun st 2020 Salishm planng
Acunt
52 Aguahe
Teututal Pendback
Phantry
0150
Rasimnfeld et of 2026 Rull EulieEaaan Easgmbble Xu di ol 2020l
SRS] Simonis
Juternal Rewards
Kang at 2022
Langu 2o
Brward-baodd Coff-Evolation
Zhong
al. 2025)
aepttaan foree 2012 SIGA Bolons
Haang parent Eoeti Amn porms
Vang
mn P0-T0r bADE ind Mommine &e Yorlagunn curss
2022n1 SPIRAL
Juplicit Rewards
Reasrd Bnough (Song at al.
3025 Rndogenaus reward
2025e
How to evolve
Sal-Generated
STAli Ka-lkaan on al 10/22
VolTall
Hossetss et
2022 Ta
Rob
Initation & Demon- Learning
thing
Others
Surte Zhan et
Pang al. 2025 RiS
Popul beulsstionbbre Mechods
Single Agent
202241 (Cimm
Zhancest Rostin
2012e STL Summer
GENOME
Mulli-Agent
PunMAn Hure al. 20243]
At. (oleery Wnar Apanesin
Puondeen Fueer
Almanendr ne
MDTeaCCPT Clscn
Menury
Stobsle-Agent-B E Wong et ol. 20260
MobileSStwaadd Lia al.
20255
General Domain
Curriculam Learzing
Webill. Co st.al. alr
2021) Voyager Wang at al. 2023a
Mydel-Ayent
Soubbrn Dang at
2025h
Ul-Gente Nixo
2025a
Abvalute-
Where to Rooive
GUI
Codet Dang
Finalloial...
2025g Qommetppan WWng
20201
Spectaliznd Domats
Olierrs
Arste-coptlor Lin
2021
Vayager Wang
Figure 2: Taxonomy of self-evolving agents, in which agents are analyzed along the what, when, how, and where dimensions, with selected representative methods and systems annotated at each leaf node.
4

---

Published in Transactions on Machine Learning Research (01/2026)
What to Evalve?
When to Evolve?
Where to Evolve?
Context
Task Completion
General Domain
Memory
Prompts
Generaa-purpeee Appliaations
Intra lest time Sell-evolution
POST
Inter lest Sime
Relrieve
Instract
S
Self-evolution
Specific Domain
Methods
ICL
SFT
RL
Agent
Coding
Financial
Medical
Education
Others
Plan, Reasion
Call/ Return
How to Evolve?
Models
Tools
Reward-based
Imitation Demonstration
Population-based
Evaluation
Agentic
Architecture
T
Textual
Leff-Generated
Single Agent
Goals & Metrics
Single Agent
Query
Mult-Agent
Query
External
Internal
Cross-Agent
Adaptivity
Retention
Mutti-Agent
deb Generalization
Efficiancy
Safaty
Impicitt
X Hybrid
Cross-cutting Evolutionary Dimensions
Paradigm
OnlineOflline
F OnVOIf-policy
Granularity
Shart-horizon
Long
Figure 3: A comprehensive overview of self-evolving agents across key dimensions. From left to right and top to bottom, the figure mirrors the organization of Sections 3
What to evolve (Sec. 3 decom- poses agent components: model, context, tools, and architecture, showing where evolution operates. When to evolve (Sec. distinguishes intra-test-ting versus inter-test- self-evolution, corresponding to ICL, SFT, and RL paradigms. How to evolve (Sec. 5 summarizes methodological families families-rewardd reward-based, imita- tion & demonstration, and population-based together with cross-cutting dimensions such as online/offling on/off-policy, and reward granularity. Where to evolve (Sec. G contrasis general-purpose and domain- specific deployments (e.g., coding, GUL, finance, medical, education). Evaluation (Sec. 7 outlines goals and metrics metrics-adaptivee adaptivity, generalization, efficiency, safety-and corresponding evaluation paradigms (static, short-horizon, long-horizon). Overall, the laxonomy maps the survey's 's reasoning flow: defining what, when, and how to evolve establishes the foundation for evaluating and advancing self-evolving agents.
7
2 Definitions and Foundations
Before delving into a comprehensive survey, we first present a formal definition of self-evolving agents and introduce a taxonomy of the key aspects in self-evolving agents. We also discuss the relationships between self-evolving agents and other renowned learning paradigms, such curriculum learning, lifelong learning, model editing, and unlearning, highlighting the adaptive, dynamic, and autonomous nature of self-evolving agents.
2.1 Definitions
Environment We first define the environment (including the user and the execution environment, e.g., Linux shell) of an agent system as a partially observable Markov Decision Process (POMDP), represented as tuple E S,T,
whore:
G is set of potential goals. Each G is task objective that the agent needs to achieve, e.g., user query.
S is set of states. Each S represents the internal state of the environment.
A is set of actions. Each action a E A can be a combination of textual reasoning, retrieval of external knowledge, and tool calls.
5

---

Published in Transactions on Machine Learning Research (01/2026)
STaR Self-Instruct
CREATOR
AFlow
MASS
CISC
ProTeGi
LIBRAi ToolGen
2022
Learn-by-interact
Voyager
DRAFT
2023
rStar-Math
WebRL
Math-Shepherd
1-8
GPTSwarm
Sirius
Anxiv Copilot
AdaPlanner
7-12
AgentOptimizer
ScoreFlow
RISE
Reflexion
2024
EvoAgent
Mobille-Agent-E
OS-Genesis
Tree-of-Thoughh
1-3
TextGuard
DigiRL
Alita
SCA
Generative Agents
IoE
STIC
GIGPO
EvolveSearch
PromptAgent
Expel
Gödel Agent
Reward Is Enough
PromptBreeder
QuantAgent
10-12
SOFT
AlphaEvolve
AutoWebGLM
AgentGen
ADAS
2025
Darwin Godel Machine
Richelieu
Agent Workflow Memory
1-2
3-4
ReMA
LADDER
REGEN
GUI-RI
EarthLink
ARIA
EvoAgentX
An evolutionary landscape of several representative self-evolving agent frameworks from 2022 to 2025. The figure chronologically organizes major research milestones in the development of self-evolving agents with capabilitics such as autonomous planning, tool use, and continnal self-improvement.
Figure 4:
is the state transition probability function which takes a state-action pair (s, a) and outputs the probability distribution T(s'ls, a) of the next state.
R: S x Ax G + R is the boodback/rewarr function, conditioned on the specific goal E G. The leedback r = R(s, a,g) lypically takes the form of a scalar score or lextual feedback.
Q is set of observations accessible to the agent.
O the observation probability function which takes a state-action pair (s, a) and outputs the probability distribution O(o'|s, a) of the next observation for the agent.
7 the discount factor.
Agent system We define a (multi-)agenn system as II (T, (w.). (MW)). The architecture deter- mines the control flow of the agent system or collaborative structures between multiple agents. It is typically represented as sequence of nodes (N1, N2,... organized by graph code structures. Each node N₂ consists of the following components:
dhy the underlying LLM/MLLM.
Cj the context information, e.g., prompt P. and memory Me.
Wi: the set of available tools/ APIs.
At each node, the agent policy is function No, (-10) that takes an observation and outputs the probability distribution of the next action, where 02 = (wh, C,). The actual action space here is the union of the natural language space and the tool space W..
For given task = (F,g), represented by an environment E and a corresponding goal g E 9, the agent system follows the topology I' generate trajectory 7 = (Ou, ag. 01, d1...., and receives a feedback r either from the external environment from internal signals (e.g., self-confidence feedback from evaluator).
6

---

Published in Transactions on Machine Learning Research (01/2026)
Self-evolving strategy A self-evolving strategy is transformation, f that maps the current agent system to new state, conditioned on the generated trajectory and the external/interin feedback r:
= = (T", (WA) (CC)(W))
(1)
Objective of self-evolving agents Let U be a utility function that measures the performance of an agent system II on a given task T by assigning scalar score TI,, R. The utility may be derived from the task-specific feedback 7, such reward signal textual evaluation, possibly combined with other performance indicators (e.g., completion time, accuracy, or robustness). Given sequence of tasks (To, .i.... Tn) and an initial agent system lo, self-evolving strategy f recurrently generates an evolving sequence agent systems (II1, Il.,..., IIa) via
= f(II,,
(2)
where Ty and ry the trajectory and feedback on task Tz.
The overarching objective in designing self-evolving agent is to construct strategy f such that the cunulative utility over tasks is maximized:
max E UI, Tj)
(3)
Operational definition of self-evolving agents To provide a conceptual boundary, we introduce an operational definition of self-evolving agents. A self-cvolving agent is the agent that modifies its internal porameters, contextual state, toolset, or architectural topology based on its own trajectories or feedback signals, with the explicit objective of improving future performance.
This definition entails three inclusion criteria: (i) updates must be seperience-dependeen driven by trajec- tories, self-generated data, or environment feedback, specifically targeting the agent's policy limitations capability boundaries rather than generic data synthesis; (ii) updates must produce persistent, policy- changing effect rather than a transient instruction-followingg behavior; (iii) the system must possess mecha- nisms for autonomous exploration or self- inff-initiated learning, even if it also leverages pre-collected data. For clarity, we use 'passive' to denote learning triggered exclusively by externally provided data or schedules, and 'active" to denote self-initiated exploration, reflection, or structural modification (i.c., using self-reflection to collect data), explicitly excluding static pipelines (e.g., standard distillation) where data generation is agnostic to the agent's 's interaction history.
As this field is rapidly forming, fully autonomous self-evolution without human intervention represents an aspirational goal rather than the current norm. In this survey, we do not impose a rigid exclusion threshold that would disregard early-stage developments. Instead, we analyze the mechanisms contributing to the self-evolving paradigm ranging from proto-evolution (e.g., iterative bootstrapping or feedback- driven prompting) to strong self-evolution (fully autonomous diagnosis and reconfiguration). allowing us to provide comprehensive view of how diverse methods contribute to the 'What, When, and How" of the paradigm's progression toward full autonomy.
2.2 Relationships with Other Works
Table I summarizes the key distinctions between sclf-cvolving agents and other paradigms (including cur- riculum learning, lifelong learning, model editing, and unlearning). We provide a brief introduction to each paradigm below, highlighting the differences among these paradigms, as well as the differences with self-evolving agents.
Curriculum Learning Curriculum learning a training strategy in which data are presented in order of increasing difficulty (Bengio et al. 2009 Wang et al. 2021 This stratogy resembles human curricula where concepts are introduced progressively from simple complex. Curriculum learning has been widely adopted
across diverse domains, including computer vision (uo et. al. 2018 Jiang et. al. 2014 Liu et al. 2023a

---

Published in Transactions on Machine Learning Research (01/2026)
natural language processing Platanios al. 2019 Tay et al. 2019 speech recognition Braun et al. 2017
Lotfian & Busso
2019). etc. Recently, several curriculum learning based methods have been proposed to fine-tune LLMs during the post-training phase (Wang et al., 2025p; Zhang et al.) 20250; Parashar et al, 2025 Zhang et al. 2025a Li et al.) 2025b). The framework for curriculum learning generally comprises
two key components: dilficulty measurer that quantifies the difficulty level of each training data point, and a training scheduler that reorganizes the order of data points received by the model according to the difficulty level. Unlike curriculum learning, which operates on a static dataset, self-evolving agents aim to handle sequential tasks in dynamic environments. Additionally, curriculum learning updates only model parameters, whereas self-evolving agents are able to adjust non-parametric components like memory and tools.
Lifelong Learning Lifelong learning refers to the ability of Al models to continuously and adaptively learn when exposed to new tasks and environments, while retaining previously acquired knowledge and abilities. This learning paradigm, also known as continual learning or incremental learning, is crucial for AT models
to operate in dynamic and complex environments [Wang et al. 2024c; Zheng et al. 2025c Parisi et al. 2019 Shi et al. 2024 Yang et al.) 2025d Zhou et al. 2024a). The primary goal of lifelong learning for Al
models to achieve a balance between preserving existing knowledge (stability) and acquiring new knowledge
(plasticity) when exposed to new data tasks (McCloskey & Cohen 1989 Zheng al. 2025c Ratcliff
1990
Rolnick et al.) 2019). Though it shares the sequential task setting with self-evolving agents, lifelong learning differs in two fundamental ways: (1) Memory funclionality and usage liming: While continual
learning methods extensively employ memory mechanisms (e.g., experience replay buffers (Rolnick et al, 2019), episodic memory (Lopez-Pat Pazz & Ranzato 2017]) to mitigate catastrophic forgetting, these mechanisms primarily serve as training- time tools for parameter optimization through gradient computation. In rontrast, self-evolving agents leverage runtime contect (prompts, working memory, conversation history) that directly influences action generation at test-time without requiring parameter updates. The distinction lies not in the presence of non-parametric components, but in their functional role: training- replay vs. test- time state adaptation. (2) Learning iniliative: Lifelong learning primarily acquires knowledge passively through externally provided task sequences, whereas self-evolving agents actively explore their environment and incorporate internal reflection or self-evaluation mechanisms to guide their own learning jrajectory. Recent self infrovipoo LLM methods (Huang et al. 2022: Yuan et al. 2024d), which iteratively refine models through self-generated data and self-critique, can be viewed as instances of lifelong learning focused on model-centric improvement. Self-evolving agents extend beyond this paradigm to encompass system-wide evolution including tool acquisition, architectural reconfiguration, and environmental exploration.
Model Editing and Unlearning Model editing and unlearning aim to efficiently and precisely modify specific knowledge in AT models while preserving irrelevant knowledge and avoiding full retraining (Wang)
et al.
2024f 2025i Zhang et al. 2024d Wang et al. 2025i Nguyen et al. 2022 Geng et al. 2025a). A canonical application of model editing is to perform efficient and precise localized factual updates (e.g.. modifying the answer to "2021 Olympics host city' from 'Tokyo' to 'Paris"). Early methods focused on
triples of atomic knowledge and later expanded into various trustworthh--reltee tasks (Fang et al. 2025a Huang et al.] 2025aa). Recent studies also propose lifelong model Baiiting(Cher et al. 2024c) that sequentially performs model editing. For model unlearning, early efforts mainly focus on the removal of privacy-related information (Chen et al., [2021). With the rapid development of LLMs, model unlearning also used to enhance LLMs' safety (Zhang et al. 2024j Li et al.) 2024c Zou et al., 2024 Lu et al. 2025). Compared to lifelong learning, model editing shares an aligned objective: both aim to acquire new knowledge or capabilities while mitigating catastrophic forgetting. However, lifelong learning typically relies on extensive gradient- based finc-tuning across all model paramcters, whereas model editing often modifies only a small subset of parameters in a targeted manner. Compared to self-evolving agents, model editing (1) cannot modify non- parametric components such as memory or tools, and (2) relies on a pre-defined pipeline from the algorithm designer, whereas self-evolving agents can spontaneously employ more diverse and flexible strategies based the observation of the environment or internal feedback signals.
Positioning Self- Self-Evolving Agents To clarify the relationships among these paradigns and to motivate the role of self-evolving agents, we examine them through two complementary perspectives: a problem-setling
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Comparison between self-evolving agents and other renowned paradigms
Runtime Context
Evolving Toolsct
Dynamic Tasks
Test-time Adaptation
Active Exploration
Structural Change
Self-reflect
Paradigm
& Eval
Curriculum Learning
X
x
X
X
X
Lifelong Learning
X
x
X
X
X
X
Model Editing
X
X
X
Self-evolving Agents
V
V V
V
V
V
lens and a celuution-parddien lens. This distinction clarifies the basis of each paradigm whether it emerges from constraints and challenges inherent to the lcarning sctting, or from mothodological proposals for how the model or agent itself can be updated.
Problem-sctting view. Curriculum learning and lifelong learning arise from concrete learning problems. Curriculun learning addresses how to structure training examples varying difficulty so a model can handle complex samples more effectively; lifelong learning focuses on acquiring new abilities over time while mitigating catastrophic forgetting. These paradigms are therefore driven by the problems they aim to solve and primarily specify how experience is organized for the learner, rather than how the agent itself may adapt beyond parameter updates.
Solution-paradigm view. Model editing and self-evolving agents, in contrast, originate as so- lutions: they propose mechanisms for updating or modifying system. Model editing provides largeted procedures lypically localized parameter adjustments to correct or insert. knowledge. Self-evolving agents generalize this idea by treating adaptation as a first-class capability, allowing not only parameter updates but also changes runtime context, memory, tools, and workflow structures, driven by the agent's own trajectories and foedback signals.
Viewed through this two-lens framework, curriculum and lifelong learning are anchored in the nature of the learning problems they address, whereas model editing and self-evolving agents are defined by the methods they provide for effecting change. Self-evolving agents thus represent system-level solution paradigm: they include parameter-level editing as one update pathway while enabling broader, persistent, and interaction- driven evolution across multiple components of an agent.
3 What to Evolve?
A self-evolving agent differs from static agent not by what components contains, but by which in- ternal states can be autonomously modified based on its own trajectories, reflections, and feedback sig- nals. Thus, the key question of this section is to identify the evolutionary loci within an agent system II (T, (vi)), (C.,, (W,)) parts of the system whose states can be rewritten in experience-driven and persistent manner, enabling cumulative self-imprvvement.
Following the formulation in Section 2.1 these evolutionary loci align with four major pillars of an agent system. Our investigation starts at the agent's cognitive core, namely the Models { whose parameters can be continuously updated through self-generated supervision, execution traces, or environmental feodback (Zhou al. 2025e Wang et al.) 2025g). We then consider the Context (C,) including instructions (Xiang
et al. 2025 Khattab et al. 2023) and long-term memory (Chhikara al. 2025 Wang et al.
2024j -which
evolves as agents reflect, store, and retrieve experience in ways that shape future decision-making, From this
internal foundation, we examine the evolution of Tools (W.), where agents autonomously create (Qiu et al. 2025b), refine (Qu et al., 2025),, and managing (Wang et. al. 2025k) executable skills based on verifiable interaction signals Finally, we scale to the Agentic Architecture, where the system's 's architecture (Hu ct al. 2024c Zhang al.) 2024ch and collaborative structures (Wan al. (2025) are optimized over time, enabling structural adaptation beyond individual components. We present representative examples of these evolutionary loci in Table
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Representative self-evolving agent methods positioned along four evolutionary pillars; filled bullet marks dimensions where the approach actively evolves.
Method
Model
Context
Tool
Architecture
Policy
Experience
Prompt
Memory
Creation
Mastery
Selection
Single
Multi
SCA Zhou et. al.
20225
RAGENT (ang 20255g
AgensGen Ho 20116
Promptbredder Ferronnda et
2023
Expel Zhuo
20
Agen: WorkHow Memory Wang et al. 2024]
Memo Chhikara nl.
MAS Zero Ke et aue 4144
Multi-Agent Desten Zhou al. 2025p
SPO Xlang al_ 20726
Alita Qui al
(2/20
Text Vollanra,
2024
DGM Zhiing e6 al 20255
AlpliaEyolve Novisov
2025
ADAS Hu ot ml. 20724c
AFlow Ziiang el 21/2
ReMA
Skill Weayer
Zhenng ot al 2025n
LearnAAt Zhao es al. 2221
DRAFT Lt cs al 21
ToolGen
Wiang rS Al.
CRAFT
Uuu al
2172kn
SEEATOR[QiaT 05
21
Voyager Wang
3.1 Models
Models constitute a primary locus of self-evolution, as their parameters can be autonomously rewritten based on the agent's own trajectories, reflections, and interaction outcomes. The ability of these models to evolve by continually adapting their internal parameters and expanding their functional capabilities is essential for the development of autonomous, general-purpose agents. Unlike static systems that rely heavily human- annotated datasets and fixed training regimes, self-evolving models can improve through interaction, self- supervised data generation, and dynamic learning loops, thereby achieving greater efficiency, adaptability, and scalability.
In detail, we outline the principal axes along which model evolution unfolds. These include learning from self-gencratod supervision to refine model weights, evolving through interaction with constructed or external environments, and integrating leedback signals that directly reshape future reasoning behaviors. Together, these strategies represent shift from passive learning paradigms toward active self-imprrvement.
Policy A self-evolving agent can refine its parameters to perform better on targeted tasks. Traditional methods of data collection for training agents on tool-use benchmarks are costly and often yield limited coverage, while purely synthetic data-generation pipelines typically suffer from inadequate quality. Con- sequently, recent studies emphasize enabling agents to autonomously generate data to improve their own model weights. One representative approach is the Self- Self-Challenging Agent (SCA) (Zhou et al. 2025e), where a language model alternates roles between a challenger generating executable Code-as- Task problems and an executor solving them. The model then fine-tunes its parameters using trajectories derived from suc- cessful solutions, resulting in significant. performance gains on complex, multi-step tasks. Similarly, the Self-Rewarding Self- relf-lmproving (ramons et al., 2025) implements an internal self-judging mecha- nism, allowing the model to autonomously generate problems, solve them, and assess its performance, thus producing self-contained fine-tuning data without external annotations. This method demonstrated notable improvements, particularly in complex reasoning tasks. Beyond task crcation, another promising rescarch direction involves leveraging interaction feedback directly for parameter updates. For instance, SELF [Lu et al. 2023), SCoRe(Kumar et al.) 2024), and PAG (Jiang et al.) 2025b) interpret execution traces or natural- language critiques as reward signals within an online Supervised Fine- Tuning (SFT) combined with Rein- forcement Learning (RL) framework, enabling continuous policy improvement. TextGrad TestGrad(Yellamaaj et al., 2024) further extends this concept by treating unstructured textual leedback differentiable training sig-
10

---

Published in Transactions on Machine Learning Research (01/2026)
nal capable of directly influencing both prompt design and model parameters. Additionally, AutoRule Wang & Xiong 2025) converts language-model reasoning traces and preference feedback into explicit rule-based training rewards, enhancing the quality of model outputs through structured reward signals. Collectively, agents autonomously crafting their training tasks to di-
these advancements chart a clear trajectory-free rectly refining their parameters based on execution leedback, highlighting the capacity of models to evolve continuously by learning from the data they produce.
Experience Agents can evolve not only by adjusting their internal parameters but also by actively in- teracting with or even constructing their environments, capturing experiences, and transforming them into learning signals that drive iterative improvement. This envirommental loop provides agents with the com- plexity and diversity required for scalable aelf-adaptatiin. The Self- SefffChallnnging Agent (SCA) et al, 2025cp exemplifies this dynamic the task level, where the agent autonomously generates novel Codc-as- Task problems, executes them, and then filters successful trajectories for retraining itself. AgentGen et al. 2024b) extends this concept to full-environment generation, synthesizing diverse simulation worlds (in PDDL or Gym-style formats) derived from an initial corpus. It implements a bidirectional evolution loop that progressively adjusts task difficulty, enabling the agent to continuously grow within a dynamically structured currieulum. Deflexion(Shin et al. 2023) complements this by introducing self-reftective mech- anisms, where agents iteratively record natural-languee critiques of their previous actions, guiding future behavior to avoid recurring mistakes. Additionally, AdaPlanner (SannerrSSunnrr et al., (023) introduces closed-loop adaptive planning, allowing agents to refine their strategies on-the-fly based on environmental feedback, effectively reshaping action sequences in response to immediate outcomes. Similarly, Self-Refine(Mablle
et al. 2023b) employs an iterative refinement loop in which the agent repeatedly critiques and revises its initial outputs, significantly improving task accuracy without explicit retraining. SICA (Self-Improvil Improving Coding (Rentt)Robyyss et al. 2025b) further pushes the boundary by enabling agents to autonomously edit their underlying code and tools, iteratively enhancing their core reasoning abilities through direct self- modification. From a reinforcement learning perspective, frameworks such as RAGEN (Wang et al. 2025q) and DYSTIL(Waar) (Wang et al., 2025c) conceptualize multi-step tool-use tasks as Markov Decision Processes, op- timizing agent policies through rich environmental rewards and strategy induction loops. RAGGN leverages dense feedback from the environment to iteratively fine-tune action policies, while DYSTIL utilizes high-level strategy advice generated by language models to progressively internalize complex decision-making skills into reinforcement learning agents. Collectively, these approaches highlight a compelling paradigm where self- evolving agents not only leverage self-generated data but actively reshape their environments and internal mechanisms to fuel ongoing learning. Such dynamic interaction loops point toward autonomous, open-ended improvement cycles dccply grounded in experiential adaptation.
3.2 Context
An essential component of an LLM agent to be evolved is the context, which shapes how an agent behaves. To start with, we want to interpret two terms, 'prompt optimization" and 'memory evolution', which have been used in different literature. In most cases, these two terms can be used interchangeably because they both refer to what is included in the context window. Prompt optimization asks "how can phrase structure the instructions so the LLM behaves better?", and attends to details such as the wording, ordering. On the other hand, memory evolution asks 'how should we store, forget, and retrieve context so that the agent can stay informed and perform better?", which focuses on what past information to surface or archive.
3.2.1 Memory Evolution
LLM- agents are increasingly designed with long-term memory mechanisms that grow and adapt as the agent continues to solve tasks and interacts with its mavironeen((Shan et al.] 2025 Qian et al. 2023a). An evolving memory enables the agent to accumulate knowledge, recall past events, and adjust its behavior based on experience. Many works stress that effective memory management is crucial for agent performance (Zhong ct al. 2024 Zhang et al. 2025c Yan et al., 2024 SAGE et al. 2024) uses the Ebbinghaus forget curve to decide what remember or forget. A-mem (Xu et al.] 2025aa) updates the agent memory struc- ture to create interconnected knowledge networks through dynamic indexing and linking, following the basic
11

---

Published in Transactions on Machine Learning Research (01/2026)
principles of the Zettelkasten method. Chhikahh et al. 2025) introduces two-phase pipeline where the agent first extracts salient facts from recent dialogue and then decides how to update the long-term memory: the agent can ADD new facts, MERGE/UPDATIE redundant ones, or DELETE contradictions. Furthermore, Memory-R1 (Yan et al. 2025) presents a reinforcement learning framework to train a dedicated Memory Manager agent that learns to select structured operations like ADD, UPDATE, and DELETE. Such a mechanism ensures the agent's 's long-term memory is coherent and up-to-date. MemInsight (Salama et al., 2025) augments raw memories with semantic structure, which summarizes and tags past interactions for retrieval later. REMEMBER Zhang et al.) 2024a) combines an LLM with memory of experiences and uses reinforcement learning signals to decide how to update that memory after each episode. Memento (Zhou et al. 2025b) enables continual adaptation without fine-tuning the LLM's parameters by employing online reinforcement learning to optimize a case-retrieval policy, which allows the agent to learn from past experi- ences stored in an evolving memory bank. MemGen (Zhang et al. 2025t) introduces a dynamic generative memory that operates in latent space. It uses a Icarned memory trigger to decide when to invoke memory and a weaver to construct latent token sequences, enabling fluid interweaving of reasoning and memory.
A critical aspect of memory evolution is enabling agents to learn heuristics or skills from past experiences. Rather than only retricving exact past instances, advanced agents distill experiences into more general guidance (Zhao et al. 2024a Fu et al.) 2024). Expel(Zhao et al.] 2024a] processes past trajectories to generate insights and rules to guide further interactions. This experiential knowledge accumulation leads to measurable gains, as the agent steadily performs better with more experience. ReasoningBank (Ouyang et al. 2025) further develops this idea by distilling generalizable reasoning strategies from both successful and failed experiences into a structured memory. It also introduces memory-aware test-time scaling to generate each task. Other systems focus on storing higher-level building blocks of problem- solving. For instance, Agent Workflow Memory KeeorryWangg et al.) 2024j) records common sub-task sequences (workflows) so that an agent solving complex task can retrieve and reuse a proven sequence of actions rather than plan from scratch. Similarly, MUSE (Yang et al.] 2025a) introduces an experience-driven agent for long-horizon tasks, centered on a hierarchical memory module that organizes experience into strategic, procedural, and tool-use memories. The agent populates this memory through a Seann-Excctt--eeleee Memorize loop, enabling it to learn on the job. In the Richelieu diplomacy agent, the system improves its negotiation strategies by augmenting its memory through sclf-play games, storing the insights from simulated interactions to refine future decisions et al. 2024). By generalizing from specific episodes to reusable knowledge, these approaches illustrate how memory evolution turns an agent's one-time experiences into long-term competencies, which leads to agents evolving.
diverse experiences
3.2.2 Prompt Optimization
While memory evolution focused on what knowledge an agent retains, Prompt Optimization (PO) enables LLM agents to self-evolve by refining the instructions it feeds to the backbone model, which directly alters the model's behavior without modifying model weights weights(Ramnal)) Rammath et al., 2025). Farly research treats instruction design as a search problem. APE et al. 2022) generates candidate prompts, scores them on validation examples, and selects the best. ORPOOYang et al. 2023) extends this idea by letting the model iteratively rewrite its own prompt., guided by leedback on prior outputs. ADO et al.; 2025a) introduces DSP that imposes semantic constraints on iteratively proposed prompts to facilitate finding the optimal prompt. Pro- TeGi(Pryzant et al. 2023) generates natural language "corrections" that are applied as edits to the prompt. forming a textual analogue of gradient descent. PromptAgent /Wang et al. 2023d) casts prompt discovery as Monte-Carlo Tree Search, exploring instruction space strategically, while evolutionary approaches like PromptBreeder ererrrra et al. 2023) maintain a population to discover increasingly effective instructions. REVOLVE (Zhang et al. 2024e] further stabilizes long optimization runs by tracking the trajectory of model responses and applying smoothed updates. Pushing this autonomy to its limit, SPO et al. 2025) creates fully self-contained loop where the model generates its training data and uses pairwise preference comparison on its outputs to refine the prompt, eliminating the need for any external labeled data or human feedback. Collectively, these techniques demonstrate that an agent can autonomously improve its prompting policy, turning prompt text into a Icarnable component that co-evolves with the agent's experience. To ad- dress the brevity bias and context collapse of some optimizers, Agentic Context Engincering (ACE) (Zhang et al. 2025ml treats contexts as comprehensive playbooks that accumulate strategies over time. Tt. uses
12

---

Published in Transactions on Machine Learning Research (01/2026)
modular agentic process with incremental updates to evolve contexts for both offline prompt optimization and online memory adaptation.
In complex systems, an agent often orchestrates a sequence of LLM calls or collaborates with other agents, making prompt design a molti-node problem. Frameworks such as DSPy represent an entire workflow as a graph whose sub-prompte are jointly tuned for global objective(Khattal et al. 2023). Trace(Wang et all 2023c3, Yellamara et al. 2024), and LLM-AutoDift( & Wang, 2025) generalize this idea by treating each prompt as parameter in differentiable program and propagating natural-language "gra- dients" to refine every step. In collaborative scenarios, Multi-Agent System Search (MASS) (Zhou et al, 2025a) first optimizes individual role prompts and then refines inter-agent communication patterns, while MAS- MAS-ZERO(Ke et al., 2025) dynamically proposes and revises role prompts to assemble an effective team for each new problem. Evolutionary systems such as EvoAgent (Yuan Yuan et al. 2024b) and AgentSQquaee(Saang et al. 2025) treat each agent along with prompts as the modules and use mutation and selection to dis- cover specialized teams that outperform hand-crafted designs. These approaches extend PO from a single instruction to the language that defines whole workflows or societies of agents.
3.3 Tools
An agent's capabilities are fundamentally defined by the tools it can wield. The trajectory of agent devel- opment is marked by crucial evolution: from being mere tool users to becoming autonomous tool makers. This transition from relying on predefined, static toolsets to enabling agents to autonomously expand and refine their own skills is a critical leap towards cognitive self-sufficiency. This paradigm, where agents dy- namically adapt their capabilities, allows them to solve a long tail of complex problems not envisioned by their initial designers. This evolution unfolds across three interconnected fronts: tool discovery, mastery, and management, as detailed in the subsections below.
Autonomous Discovery and Creation The primary impetus for autonomous tool creation is to over- come the inherent limitations of fixed toolset, granting agents the flexibility to innovate on demand. Methodologies for this now span a spectrum from opportunistic discovery to formalized synthesis. At one end, agents like Voyager build an over-expanding library of skills through emergent trial-and-error, driven by an intrinsic motivation to explore complex, open-ended environments like Minecraft /Wang et al., 2023a This exploratory approach is powerful for generating wide array of skills but may lack precision. In con- trast, systems like ATLASS, Alita, and Live-SWE-Agent take more reactive approach, often creating new tools from scratch or employing rotrieval-augmenner generation (RAG) to scarch open-source code reposi- tories the moment a capability gap is identified (Haque el al. 2025 Qiu et al.) 2025b Xia et al.] 2025a At the other end of the spectrum lie highly structured frameworks that treat tool creation as a deliberate enginecring process. CREATOR, for example, discntangles abstract tool creation (c.g., reasoning about the general structure of a reusable function for averaging temperatures over N days) from concrete tool usage (e-g., deciding how to apply that function to a specific city and time range). which enhances modularity and reusability et al. (202b)). Even more formally, SkillWeaver analyzes successful human or agent task trajectories to propose, synthesize, and hone new skills into robust, reusable APIs, ensuring higher degree of initial quality(Zheng et al.] 2025a). Furthermore, frameworks like CRAFT demonstrate that creating specialized toolsets for specific domains is essential to complement general-purpose models, enabling expert- Saaabilltyyyaannnna
level performance without sacrificing adaptability Yuuan et al., 2024a). RL-GPT et al.) 2024) integrates generated code implementations into the RL pipeline, leveraging these as tools to tackle complex tasks while addressing simpler ones directly using a Mode-as-Policy approach. This integration dynamically adapts and evolves in response to environmental foedback, cnabling continuous improvement. However, this burgconing autonomy introduces significant challenges, particularly around safely and security. The unconstrained gen- eration of code risks creating tools with exploitable vulnerabilities unintended harmful behaviors, making automated verification and sandboxing critical arcas for future rescarch.
Mastery Through Iterative Refinement The proliferation of self self-created tools necessitates a robust mochanism for their mastery; a newly generated tool is often a brittle script, not a reliable function. This is where iterative refinement becomes essential. Frameworks like LearnAct and From Exploration to Mastery establish a critical self-correction loop where the agent learns from its own experience et al. 2024b:
13

---

Published in Transactions on Machine Learning Research (01/2026)
This involves tackling the difficult "credit assignment" problem: determining precisely which line of code or which parameter was responsible for failure. To do this, the agent analyzes rich variety of feedback signals including compiler errors, unexpected API return values, environmental state changes, or even implicit signals from a user's subsequent actions. The goal is not only to debug the tool's underlying code but also to refine its documentation (e.g,, its doostring and argument descriptions), which is crucial for improving the agent's ability to understand and correctly use the tool in the future. This refinement process also opens the door for valuable human-agent collaboration While full autonomy is the ultimate goal, many systems can be designed with 'human in the loop," where human expert can provide corrections, offer high-level suggestions, or validate newly created tool. This collaborative approach can significantly accelerate the mastery process and ensure that the agent's is skills align with human intentions and safety standards. Ultimately, this self-honing process is what elevates a nascent skill into a dependable capability, ensuring the agent's growing skill library increases not just in quantity, but more importantly, in quality and robustness.
Qu et al. 2025).
Scalable Management and Selection As an agent's mastered skill library grows into the hundreds or thousands, it faces a "curse of abundance." The challenge shifts from creating tools to efficiently managing and selecting from them. A large library creates a massive search space, making traditional retrieval methods slow and inaccurate. To overcome this, ToolGen (Wang et al. 2025k) represents a fundamental paradigm shift by encoding tools as unique tokens within the language model's vocabulary. This elegantly reframes tool retrieval as generation problem, leveraging the transformer's immense pattern-recoguitinn capabilities to predict the most appropriate tool natural continuation of its thought process. TOOLMEM (Xiao 2025b) enables agents to learn and store the strengths and weaknesses of different tools in dedicated
et al. memory. At inference, the agent retrieves this knowledge to make more informed decisions, optimizing tool selection for specific task requirements. Beyond selecting a single tool, advanced agents must also excel at tool composition-leanee learning to chain multiple tools in novel sequences to solve multi-step problems. This is a higher-order management task. Architectural approaches like AgentSquare engage in a form of mela-learning, automatically searching the modular design space of an agent including its planning, memory, and tool-use components-to find an optimal configuration for complex task exccution (Shang al.) 2025). As logical endpoint to this evolutionary trend, visionary concepts like the Darwin Godel Machine propose framework for open-ended evolution, where the agent can fundamentally rewrite its own core code. In this vision, the distinction between the agent and its tools blurs, leading to a recursive cascade of self-improvement that transcends tool enhancement alone[Zhang et al.] 2025h). In essence, this entire evolutionary path aims to establish a closed and virtuous cycle: a truly autonomous agent. that can perceive gaps in its capabilitics, create novel solutions, master them through practice, and scamlessly integrate them into a coherently managed and ever-expanding repertoire.
3.4 Architecture
The defining feature of next-gencration agentic systems is their intrinsic capacity for self-improvement. This marks fundamental shift from systems with fixed capabilities to those that can autonomously enhance their performance (Liu et al.] 2025bj. By treating their own internal logic and collaborative structures as optimizable components, these systems can adapt their behavior and design in response to feedback, achieving a level of efficiency and effectiveness that static designs cannot match. This section details how this self-optimization is realized, first by examining improvements within single-agent systems and then by exploring the co-evolution of complex multi-agent systems.
3.4.1 Single-Agent System Optimization
LLM-Invoking Node Optimization Optimizing a single LLM call is straightforward in isolation, but within an agentic system, it becomes a difficult credit assigument problem, as the effect of any single change is obscured by subsequent steps. Research addresses this by making node-level components optimizable, following two main strategics. The first focuses on refining nodes within a fixed agentic topology. A prime example Trxt (Yellamraju et al. (024). which, inspired by backpropagation, uses 'textual gradients" to propagate feedback from the final output backward through the workflow, guiding systematic, local
14

---

Published in Transactions on Machine Learning Research (01/2026)
refinements at each node without altering the system's overall structure. The second, parallel strategy integrates this component-level optimization directly into the search for the system's 8 architecture itself. Under this approach, node characteristics become tunable parameters in a larger search space. For instance, frameworks can embed prompt engineering directly into the search loop, allowing the system to discover not just. the optimal workflow but also the most effective instruction for each agent simultaneously (Zhou el al. 2025a). Similarly, EvoFlow (Zhang et al.) 2025c) uses evolutionary algorithms to construct heterogencous workflows by selecting the most suitable LLM for each task from a diverse pool. This holistic strategy enables the discovery of systems that are co-optimized for both their structure and individual agent capabilities, effectively balancing metrics like overall performance and cost. Yee et al. 2025a
Autonomous-Agent Optimization Building upon the optimization of individual LLM-invoking nodes, a more profound level of self-improvement targets the autonomous agent as a holistic entity. This evolution proceeds along two main fronts: optimizing the agent's 's high-level architectural design and enabling the agent to directly modify its own source code. The first approach focuses on discovering the optimal agent structure. AgentSquare (Shang et al. 2025 exemplifies this by defining a modular design space of components like planners and memory modules, then using an evolutionary algorithm to find the most effective combination for a given task. The second front involves agents that dynamically rewrite their own operational code. This is seen in radical systems like the Darwin Gödel Machine (Zhang et al.) 2025h), which recursively modifies its own Python codebase, and AlphaEvolve (Novikov al., 2025), which uses evolutionary coding to im- prove specific algorithms. Similarly, Gödel Agent (Yin et al. 2025 provides a self-referential framework for agents to analyze and alter their logic. Together, these two directions (optimizing the agent's architectural "blueprint" and its functional code) demonstrate a key trend toward turning the agent's 's fundamental struc- ture and logic into learnable components. MemEvolve (Zhang et al. 2025g) introduces a meta- evolutionary framework that evolves not just the agent's experiential memory, but the memory system's 's architecture it- self. Through a bilevel optimization process, it adapts the mechanisms for encoding, storing, and retrieving information to better suit specific task doomiin.
3.4.2 Multi-Agent System Optimization
How agents are organized and communicate within a system (its topology) fundamentally determines its ca- pacity for solving complex problems. The field has evolved from using fixed, human-designed communication structures to creating dynamic systems that automatically adapt their organization to a given task, allowing them to discover and exploit the most effective collaboration patterns. This evolution is explored along two major fronts: the optimization of static, explicit workflows and the co-evolution of dynamic, internal policies.
Agentic Workflow Optimization The optimization of agentic workflows focuses on finding the most effective, often static, structure of communication and lask delegation for a given problem. Early research established important foundations, with studies like AutoFlow (Li al. 2024d) demonstrating the automated creation of linear workflows from natural language, and GPTSSarm (Zhuge et al. 2024) proposing a unifying graph-based framework. Concurrently, other foundational work explored how agents could evolve by using symbolic learning to dist.ill their interaction experiences into explicit, interpreiable set of logical rules to guide future decisions (Zhou et al., 2024b). This abstraction of systems into tunable components whether nodes, edges, or symbolic rules was crucial. However, these early systems often lacked a formal method for efficiently navigating the vast space of possible configurations and interactions.
The major breakthrough came when ADAS (Hu et al. 2024ch and AFlow (Zhang et al. 2024c) formally defined this challenge as search and optimization problem. ADAS set a theoretical vision by framing system design as search through a coriingeomppete space of code-based configurations, Building on this, AFlow made it practical by introducing reusable operators that represent common agentic patterns and by employing Monte Carlo Tree Search (MCTS) to efficiently navigate the enormous design space. Together, these works established a core methodology for treating agent system design as a tractable optimization problem, proving that automatically discovered workflows could outperform human-designed ones.
Following this formalization, research rapidly diversified toward creating customized agent systems for each specific query. Two primary strategies emerged: search-based and learning-basee generation. Saarch-
15

---

Published in Transactions on Machine Learning Research (01/2026)
methods, such MaAS (Zhang et al. 2025d create "supernet" of potential architectures and then sample specialized system from it. In parallel, learning-basel methods train models to generate effective topologies directly. ScoreFlow (Wang et al.
2025n), for instance, trains a generator using a novel preference optimization method, while FlowReasoner (Gao et al. 2025 uses reinforcement learning to train a meta-
agent that constructs a bespoke workflow on the fly. T This line of query-specific generation continues to be
2025). Furthermore, it is important to note that this process is not limited to the topology alone; many of these frameworks also perform node-level optimization in tandem, such as co-optimizing prompts selecting heterogeneous models as integral part of the architectural generation process (Zhang et al. 2024c Zhou et al. 2025a Zhang et al. 2025c).
an active area of rescarch (Ye ct al. 2025b Ke al.]
A key challenge for all search and learning methods is the computational cost. of evaluating each potential workflow (Shang et al. 2025). To address this, researchers have developed lightweight prediction models. Agentic Predictor (Trirat et al. 2025) a prime example, training a model to accurately estimate a workflow's performance basod on its structural and semantic features without a full exccution. By providing a fast and inexpensive evaluation proxy, these predictors significantly accelerate the optimization process, making the exploration of vast design spaces feasible (Zhang et al. 2025s).
Mluti-AAtonomonne Agent Optimization Distinct from optimizing a system's ''s explicit workflow struc- ture, this line of research focuses on how multiple autonomous agents can co-evolve their internal behavioral policies through interaction. This approach enables emergent capabilities like coordination, task delegation, and beneficial competition. For instance, ReMA (Wan et al.) 2025 uses multi-agent reinforcement learning (MARL) to collaboratively train a high-level meta-thinker and a low-level executor, significantly improv- ing performance on reasoning benchmarks. Building on this, GiGPO (Feng et al. 2025b enhances MARL training by aggregating trajectories to provide more precise credit assignment, boosting success rates on long- horizon tasks. To support this direction, platforms like MARTI (Liao al. 2025) provide open-source infrastructure for orchestrating and scaling the training of these language-model collectives. Collectively, these studies underscore multi-agent reinforcement learning as promising route for cultivating group-level competencies unattainable by individual agents alone.
4 When to Evolve
The temporal dimension of self-evolution in LLM-based agents mainly concerns the relationship between learning processes and task execution. Therefore, the second key aspect of self-evolving agent is identifying the evolving timing, i.e., at which stage the self-evolving strategy f is invoked and applied to the agent system. To this end, we propose a taxonomy that distinguishes between two temporal modes of self-evolution: Intra- test-time self-cvohution and inter-test-time self-cvolution.
Intra-test-time self-evolution refers to adaptive processes that oceur during task execution, where agents recognize their limitations on a specific problem and initiate targeted learning mechanisms to enhance their capabilities in real-time (Xi et al.) 2024 Bi et al.) 2024). This mode of evolution is characterized by its immediate coupling with the task al hand: the agent improves its problem-solving abilities for specific problem encountered, creating a dynamic interplay between performance and adaptation.
Inter-test-time self-cvolution refers to learning processes that occur between task completions, leveraging accumulated experiences to improve future performance. This category encompasses diverse methodological approaches: offline learning paradigms that extract knowlodge from pre-collected datasets through itcrative refinement (Zelikman al.) 2022 2024), and online learning paradigms that continuously adapt based on
streaming interaction data [Qi et al. 2024 Qiu et al. 2025b Qian et al. 2024b Wang et al. 2025o
The implementation of self-cvolution across these temporal phases leverages three fundamental learning paradigms in LLMs: in-conlext learning (ICL) (Dong el al.) 2022 Min et al. 2021 Wies et al. 2023). which
adapts behavior through contextual examples without modifying parameters; supervised fine- sine-tuning (SFT).
which updates model weights through gradient-based optimization on labeled data (Devlin et al. 2018 Shen
2024. Dong al. 2023); ; and reinforcement learning (RL), which shapes behavior through reward-driven
policy optimization (Kaelbling et al. 1996 Sun et al. 2024a Zhang et. al. 2025m). While these learning
16

---

Published in Transactions on Machine Learning Research (01/2020)
Variant Generation
Verilication
Policy Update
Intra-test-time Self-evolution
Agent
Task
Lumden
Inter-teat-time Self-evolution
Policy Update
Trajectory
Rollout
Figure 5: An overview of when to evolve, The top pathway illustrates intra-test-time self-evolution, where adaptation (e.g., variant generation, verification, and policy update) occurs within task execution. The bot- tom pathway depicts inter-test-time self-evolution, where learning happens retrospectively through rollout, trajectory analysis, and policy updates.
paradigms remain conceptually consistent across temporal contexts, their instantiation differs terms of data availability and learning objectives:
intra-test-time is characterized by its online nature: learning data emerges dynamically during task execution, with optimization directly largeting performance enhancement the immediate problem instance. This real-time coupling necessitates rapid adaptation mechanisms that can process learning data and feedback signals and modify behavior within the temporal constraints of active task-solving. On the other hand, inter-test-time characterized by its retrospective nature: learning algorithms operate on historical data, whether from curated datasets or accumulated behavioral trajectories, with optimization objectives oriented toward improving expected performance across the task distribution rather than maximizing success on any specific problem instance. This temporal decoupling enables more sophisticated learning procodures that identify cross-task palterns, consolidate diverse experiences, and develop generalizable capabilities without the immediacy constraints of active task execution.
4.1 Test-Time Self Eefl-Evolution
In intra-test-time self-evolution, agents engage in self- gelf-improvement processes that are intrinsically coupled with solving the immediate task at hand. The distinguishing characteristic of this temporal phase is its synchronous nature: feedback signals are generated and processed during task exccution, with optimization objectives specifically targeted at improving performance on the current problem instance rather than gen- eralizing to future tasks. Here, we introduce how the three learning paradigms are realized in this temporal phasc.
In-Context Learning Intra-test-timo ICL methods leverage the model's context window dynamic memory system for immediate adaptation without parameter modification. These approaches typically em- ploy self-reflective mechanisms where agents analyze their own performance, generate verbal critiques or insights, and maintain these reflections in episodic memory buffers to guide subsequent decisions within the same task context (Shinn et al. 2023 Madaan et al. 2023a) Some methods extend beyond simple reflection to include dynamic planning revision, where agents can modify their entire approach based on environmental feedback, switching between action execution and plan modification as needed. For instance, AdaPlanmer (Sun et al.] 2023 decomposes tasks into manageable sub-goals and predicts environmental feedback for each. During execution, its refiner component distinguishes between in-plan feedback (observations aligning with predictions) and out-of-plan feedback (deviating obscrvations). For in-plan feedback, the refiner dynamically queries the LLM through a specialized ask LLM() action to parse observations and extract pertinent infor- mation. For out-of-plan feedback, the refiner proactively revises the entire plan and resumes solving from an intermodiate point, rather than restarting from scratch. This adaptive closod-loop framework climinates the need for prior knowledge about feedback structures and enables more efficient Docision-making Similarly, Trust Agent. (Hua et al. 2024) employs rule-based plan revision during execution, modifying its approach
17

---

Published in Transactions on Machine Learning Research (01/2026)
based on language feedback to evolve toward safer planning strategies. These ICL methods demonstrate how test-time adaptation can achieve sophisticated behavioral modification ion without, permanent model changes, maintaining flexibility while preserving the model's general capabilities.
Supervised Fine-Tuning Intra-test-time SFT represents a paradigm shift where models perform imme- diate melfmmoiiication through learned meta-adaplation strategies. Self-adaptive language modeling (Zweiger ct al. 2025 exemplifies this approach by generating "self-edits", which are meta-level instructions that can restructure information representations, specify optimization hyperparameters; or invoke tools for data aug- mentation and gradient computation. These self-edits trigger immediate supervised fine-tuning, resulting in persistent weight updates that adapt the model to the current task. The key innovation lies in the meta- meta-learning phase, where reinforcement learning trains models to produce effective self-edits by using the downstream performance of the updated model as the reward signal, essentially teaching models how to teach themselves. Acikgoz et al. (2025) introduce a Test-Time Self-Improvement (TT-SI) framework that en- ables agents to adapt on-the-fly fhe by first identifying uncertain test samples through self welf-awareness. For these challenging inputs, the agent then generates a single synthetic training example and performs a temporary, lightwcight parameter update to improve its immodiate performance before resetting its weights.
Reinforcement Learning. Intra-test-time RL enables models to develop new capabilities on-demand when encountering problems beyond their current competence. LADDER (Simonds & Yoshiyama 2025)
demonstrates this through its test-time reinforcement learning (TTRL) mechanism: upon identifying
particularly challenging problem, the system generates a focused set of related problem variants and conducts intensive, targeted reinforcement learning specifically for that problem class. This approach transforms insurmountable challenges into learning opportunities allowing models to expand their problem-solving repertoire during deployment rather than failing or providing suboptimal solutions. The method represents a form of just-in-time skill acquisition, where computational resources are invested precisely when and where they are needed most.
4.2 Test- Time Self-Evolution
Inter-test-time self-evolution represents the predominant learning process in autonomous agents, wherein adaptation occurs following task exccution rather than during it. In this temporal mode, agents complete a given lask, extract feedback siguals, including explicit rewards (Gao et al. 2024), gradients (Amari, 1993 Bottou, 2010), and performance metrics (Ge et al.) 2023), and subsequently leverage this information enhance their capabilities for future problem-solving. This retrospective learning process decouples task performance from capability improvement, allowing agents to consolidate experiences, identify patterns of success and failure, and systematically refine their behavioral policies without the computational constraints imposed by real-time task demands.
In-Context Learning. Inter-test-time in-context learning has emerged as a widely adopted approach for agent self-improvement. This paradigm leverages execution results and feedback from previous tasks as con- textual information for future problem-solving. Wang et al. (Wang et al.) 2024j) demonstrate this principle by inducing workflows from agent action histories and incorporating them into the context for subsoquent tasks. The field of in-context reinforcement learning (ICRL) (Mocini ct al.] 2025 Laskin et al. 2022 Lee ee al.] 2023) extends this concept by maintaining histories of observations and actions within the agent's 's context window. These methods exploit the hypothesis that pre-trained neural networks implement implicit reinforcement learning algorithms within their forward pass, processing contextual information to adapt behavior without parameter updates (Kirsch et al. 2023). A defining characteristic of ICRL is in- context improvement: the phenomenon whereby agent performance progressively enhances as task-relevant information accumulates in the context, enabling sophisticated adaptation through attention mechanisms rather than gradient-baat basod learning.
Supervised Fine- Fine-Tuning. Inter-test-time SFT (Chen et al.] 2025b) methods establish paradigm of iterative self-improvement through synthetic data generation and evaf-evalu.tion. SELF (Lu et al. (2023) pioneered meta-cognitive training, where models first. acquire self-feedback and self-refinmmet capabilities,
18

---

Published in Transactions on Machine Learning Research (01/2026)
then iteratively generate responses to unlabeled instructions and enhance them through self-critique. STaR (Zelikman et al. 2022) and Quiet-STaR (Zelikman et al. 2024) focus on reasoning improvement through rationalization models attempt problems, then generate explanations for correct answers they initially failed to solve, creating augmented training data that combines successful attempts with post-hoc reasoning. SiriuS (Zhao et al. 2025b) exlends this lo sequential problem-solvingg maintaining repositories of correct solutions while augmenting failures through multi-stage refinement involving feedback incorporation, regeneration, and rephrasing. These methods share a core insight: models can bootstrap their own improvement by learning to evaluate and enhance their outputs, creating high-quality training signals from initially imperfect attempts without extensive human supervision. Recent frameworks such as ARIA (He et al. 2025a) further extend this paradigm by incorporating human-in-the-loop guidance into test-time adaptation, allowing agents to proactively identify knowledge gaps and request expert leedback.
Reinforcement Learning. Inter-test-time RL leverages unconstrained computational resources to op- timize agents through extensive environmental interaction and sophisticated curriculum design. RAGEN (Wang et al., 2025q) and DYSTIL (Wang et al. 2025ch employ online reinforcement learning for multi-turn interactive tasks, continuously refining policies through on-policy learning in simulated dialogues. Learning Like Humans (Zhang et al. 2025a introduces cognitive-insireed training with adaptive difficulty progression, combining on-policy exploration with off-policy elficiency and expert demonstrations to accelerate learning. Domain-specifie applications demonstrate the versatility of inter-test-ting RL: WebRL (Qi et al. 2024] de- velops web navigation agents through self-cvolving curricula that automatically adjust task complexity basod on performance, while DigiRL (Bai et al., 2024) enables device-control agents to master in-the-wild inter- actions through autonomous reinforcement learning. These approaches exploit the pre-deployment phase to engage in extensive tria--and-error learning, developing robust policies through thousands of interactions that would be impractical during real-time deployment.
5 How to Evolve
The pursuit of self-evolution lies at the heart of building advanced, autonomous, and increasingly general artificial intelligence. For large language models (LLMs) and their agentic extensions, the question of how to continually, autonomously, and efficiently evolve their capabilities has become central challenge. Therefore, the third key aspect of a self-evolving agent is instantiate an effective evolving strategy f, i.c., how to transform agent system II = (T, [v.), (C.), (W:)) to its new stale II' = (I', (v1), (C)), (W])). Unlike traditional approaches that rely on static datasets or one-time supervised fine-tuning, self evolution empha- sizes an ongoing process where models learn from real-world interactions, actively seek feedback, self-reflect, generate or curate new data, and adapt their strategies in response to dynamic environments. This contin- uous evolution is not merely matter of scaling up data or computation; requires the agent to acquire a spectrum of meta-capabilities including self-correction, autonomous data generation, knowledge transfer, and multi-agent collaboration. As result, the landscape of self-evolution has become increasingly rich and multi-faceted with each methodological branch exploring different axes of feedback, learning paradigms, data sources, and evolutionary scales.
Over time, research on self-evolving agents has progressed through three major paradigms- maradigns-reward- sewarddbaseed imitation-based, and population- evolution each emerging to address the limitations of the previous one. Reward- methods first closod the foodback loop through explicit signals but suffered from brittle- ness and high cost. Imilation-based learning stabilized evolution by leveraging high-quality demonstrations, though sometimes at the expense of exploration. Population- evolution then extended adaptation to collective scales, emphasizing diversity and emergent coordination. Together, these paradigms outline a coherent trajectory from individual self-improvement toward collective intelligence.
This chapter aims to systematically map and analyze the major families of self-evolution methods, providing a unified framework for understanding their principles, mechanisms, and interactions. We begin with reward- based evolution, which centers on the design of reward gegnals-ranghn from natural language feedback and internal confidence metrics to external implicit signals to guide iterative self-improvement.
19

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Overview of Reward-based, Imitation Daniaattonn/noootaaaittoonno and Population-based Learning Methods for
Self-Evolving Agents. This table categorizes key approaches basod on the following criteria: (1) Foodback
Type: the type of leedback used, including language- based rationales and numerical rewards. (2) Feedback
Source: the origin of the feedback, either internal (model-generated) Learning Method: the learning paradigm applicd, such as in-context learning (TCL., supervised fine-tuning (SFT), reinforcement learning (RL), and evolutionary algorithms; (4) Updated Components: which parts of the model are updated, either full parameters or a subset of the model. (5) Update Timing: the stage
external (provided externally). (3)
during the agent's 's evolution when updates are applied, such as pre- gre-training, pre-test, or test-time.
Method
Feedback Type
Feedback Source Learnnng Method
Updated Componeuts
Update Timing
Revord baald Essiation Methods
Rellesson Shiun ml
3045
langunge
Intertal
ICL
tist-lime
AdaPhamns
language
esternal Intermal
ICL
test time
langaaye
ICLL
Loel-line
lutguage
esternal + Internal
SFT
perams
pre-test time
SFI.-RRFFIN
20734
language
Interne)
[CI
conteett
trat-time
SCoRe
esternal
RL
full paroms
pre test time
PAG
nunerial
teclorsed
RL
full parune
pre-isot time
Teatiesd Tellonng
3124
language
cocerraal
[C1
pro-test Lime istt-lime
SRS1
langunge
RT.
full perame
presteed time
SelCTimin
R1.
pre-tast time
MM
numerica)
porams
pro fest time
CoNo Ziiing
nunetical
Intennal
parums
pre-test time
TWW.assndinn Mil m
KW/4
languae
external
ICL
contestt
test time
SICI Rakenga De 21 kt
nunerical
seteried
ICLL
curdlenoen todds work lowe, prompta)
toa--line
Peedhack
language
osternal
[CL
content
test time
USFagen
Anplle
language
icternal
ICT
conttrct
teet-lime
DYSTIL
language mumerical
external Internal
SPT+RL
full
test time
OTC-PO
RL
pre-tsst
AUTORULE Wott X2a2
lunguage mumerical
external +
Internal
RL
parums
pre-trst time
Frest:
numerica)
neternal
RL.
pre-tret time
LADDER Shopants A Yelllyiiin
numerical
external
RL
full paroms
pre test time
RACES Jinn UP PJMM0
numerical
esternal
ILI
full porams
test time
SPIRA EE
annerical
Incernel
RL
full
pre-test time
ICRL. Prong Sumee 3125
exterral internal
RT.
fill
trat-iime
MATH-SHEPHESS
RL.
pre-tsst Gime
Apon PRM
nsternal
SET+RI
porame
pre-teet time
Agett
reterral
RT.
parma
pre-tret time
GIGPO
uuncical
external
R11.
full paroms
pre test time
SPA-BL
socoried
RL
Fall parma
pre-irsi Gime
Sult 1i
Wiine
languags
SFT
Full
pre-test time
Wioorlf.M M
Alr
langunge
SFT
fill
pre-taed time
08 Genostz
osternal
SPT
prc time
UT-Gemse
Xito
mumerica)
esternal
SFT
partial parame
pre-tret time
GUL-R
53490
munerica
ooternal
SFT+RL
params
pre-test time
numerical
rcternal
SFT+RR.
parmm-
pre-teet time
Vigsgge
W
90
Iangunge
seleriaa
ICLL
tusl-tine
SwitsSng
ml
language
esternal
[CE
content
test time
AutoWehGl.
6124
language
oxternal
SPT RL
full params
fest time
DigiRL
Do
language
esternal
RL
portial porams
time
Webkl
lazgragg
SPT+RL
pre-istt Gime
Lat's Verity Stop-le
2023
language
external
SFT
parums
pre-test time
AlphaMath
Chee
numerical
rcternal
SFT
param-
pre-tret time
1Star Mothl Cineec [jp
numerical
esternal
SFT
full params
pre test time
Deeli Warn
2e mi 10/1
language
esternal
RL
full porams
pre-test time- tret-time
Mo Stn mi 204ba
languag
oxternal
RL
full parums
pre fest time
Ausiution and Demossvvtiinn Learning Methads
STAR
Zellkmmss el m 2123
lungange mumerical
intesual
SPT
full params
pre-tist lime
V-8 Tw Miniooon
numerical
external + Internal
SPT RT.
partial
time
AdaSTar
SFT
param-
pre-tast
ST1C
language
Inteenal oxternal
RL SPT
portial parame
pic time
GENIXEE
language
SFT
params
pos-tainiis,
Salus 2hoo
Junguage mumeriicl
incernal
SFT
full paroms
pee time
SOFT 1 Tell 4
langunge
internal
SFT
ne squcilled
pre-tred time
RISE 04
langnage numerical
Internal + esternal
SPT
full params
pre test time
ToR
numerical
Inceenal
tret-time
Popsiction dased and Brolitionoyy Methods
DGM Zhing ct al
numerical lazguage
esternal
[CL
codebase (tools, woekfiouss promptes loun compudtion, workilow, prosmpts
test time tist-tiue pre-test time pre-tred time
EvaM
ICL
SPIN
language
incernal
parums
GENDMI
numerical
nnternal
Eoodution Alg-
partial perrams
SPC Pir MBMOI
numerica)
incernel
SET+RL
critic paromss
pre test time- teet time
Puppetre: Tinie mF 57.3
numerical
octernal
RL.
planeer pulley
pro-tess time between tacks test time pre-teed time List-time
ModdgessSim Ammasooo
2120h
langugg:
oxternal
[CL
contistt (knowlodge basc)
STI Meirin Balor
langoot munerical
intrenal
SFT
valta murell
lazguage
osternal
IUL
(kuoshalge
Next, we examine imitation and demonstration learning, where agents learn by mimicking complete, high-quality behavioral exemplars (i.e., demonstrations). While traditionally sourced from human experts, in the context of self-evolving agents, these exemplars are often generated by the agent itself or by other agents. This paradigm is particularly powerful when demonstrations are abundant or can be autonomously synthesized, and it has driven significant progress in both reasoning and multimodal domains.
20

---

Published in Transactions on Machine Learning Research (01/2020)
Textual Feedback
Internal Reward
T
Natural language: My plan was to However, the task says to... I should have
Model's own probability estimates or certainty
Reward-Based Evolution
Implicit Reward
External Reward
In-context RL using simple scalar signals
Environment, majority vating, explicit rules
Figure 6: Overview of reward-based self-evolution strategies, categorized into textual, implicit, internal, and external rewards, each associated with distinct feedback sources and mechanisms.
Finally, we introduce population-based and evolutionary methods, which draw inspiration from bio- logical evolution and collective intelligence. These approaches maintain populations of agent variants or col- laborating agents, leveraging mechanisms such as selection, mutation, crossover, and competition to explore the solution space parallel, foster diversity, and enable the emergence of novel strategics or architectural innovations.
5.1 Reward based Self-Evolution
The capacity for ielf-improveeent is a cornerstone of advanced intelligence. In the context of Large Language Models (LLMs), this manifests as a dynamic process of reward-driven evolution, where models iteratively learn from their own outputs and interactions to refine their capabilities. The design of the reward signal, which serves as the guiding feedback, crucial; it determines the nature, efficiency, and effectiveness of the learning process. In this section, systematically review the main methodologies for reward design, categorized by the nature of the feedback: textual feedback, internal confidence, external rewards, and implicit rewards.
Textual Feedback Textual Feedback leverages the native modality of LLMs natural language to pro- vide detailed, interpretable instructions for refinement. Unlike scalar rewards, textual feedback encapsulates nuanced critiques and actionable suggestions. Recent frameworks such as Reflexion (Shinn et al. 2023). AdaPlanner (Sun et al. 2023), AgentS2 (Agashe et al.) 2025), SELF (Lu et al. 2023). Self-Refine (Madaan et al. 2023a SCoRe [Kumar al. 2024), PAG (Jiang el al. 2025b), and TextGrad (Yellamraju el al. 2024) exemplify this direction. For instance, Reflexion proposes "verbal reinforcement learning," where agents reflect in natural language their past trials, storing these reflections episodic memory to guide future decisions. AdaPlanner enables closed-loop adaptive planning by allowing LLM agents to revise their plans based on both in-plan and out-of-plan feedback, while also mitigating hallucination via. code-style prompts and leveraging skill discovery. Self-Refine and SELF further explore iterative self-feedback and self- correction, demonstrating that even saate-of-the-art models can be improved via multi-turn, language-basee based self-critique, without additional supervised data or external reinforcement. Such frameworks highlight the power of language as a reward channel, enabling nuanced, flexible, and sample-cfficicnt self-improvement.
Internal Rewards Internal Confidence-based rewards move away from external signals and instead exploit internal metrics such the model's probability estimates or certainty. This paradigm leverages the model's intrinsic understanding to guide improvement without relying external supervision. Methods such
Sanfideenc-IIffrmeer Self- Self-Consistency (CISC) Taubenfeld et al. 2025), Self-Ensembbe (Xu et al. 2025b).
Solf- LaffRRearddig Sclf-Improving (Simonds et al. 2025), scalable best-of-N selection via self-certainty (Kang el al. 2025), and Self-Rewarding Language Models (Yuan el al. 2025b) allow models to self-evaluate and calibrate their responses based on internal confidence metrics. For example, CISC weights reasoning paths by confidence scores to improve both accuracy and computational efficiency, effectively filtering high-quality solutions from multiple candidates. Self-Ensenn Ensembeee mitigates confidence distortion by dividing choices into smaller, more manageable groups and aggregating predictions reduce overconfidence bias. Self- Relf-Rewarding
21

---

Published in Transactions on Machine Learning Research (01/2026)
Language Models demonstrate that models can act as their own reward function, generating training data through sell-instruction and self-evaluation cycles. These approaches can reduce reliance on human labels and external evaluators, enabling scalable and autonomous self-improvement loops that can operate continuously without human intervention. AgentEvolver (Zhai et al., 2025) proposes a comprehensive framework to improve agent. training efficiency through three synergistic mechanismss: sell-questioning for autonomouss task generation, self-navigating for experienco-guidee exploration and self-attributing for fine-grained credit assignment. In particular, its self-attributing mechanism uses an LLM's reasoning to retrospectively assign step-wise rewards that are dense and semantically grounded for policy optimization.
External Rewards External Rewards are derived from sources outside the model, such as the environ- ment, majority voting, or explicit rules. Majority voting (Shafayat et al. 2025 Wei et al.) 2025c Zhang et al. 2025jj uses consensus among multiple model outputs as a proxy for correctness, providing a self-generated but grounded reward signal. Environment foodback, including tool-basod signals, is central agentic LLM research (e.g., SWE-Dev (Du el. al. 2025), SICA (Robeyns et. al., 2025ap, Feedback Friction (Jiang et al., 2025a), USEagent (Applis et al.) 2025), DYSTIL (Wang et al.) (025cl)), where agents learn through direct interaction with real-worly environments and tools. Rule-based rewards (Wang et al. 2025h Wang & Xiong, 2025 Zhang et al.) 2025a Simonds & Yoshiyama 2025 Wang et al. 2025q; Liu et al., 2025d) use explicit
constraints or logical rules as verifiable signals, particularly effective in the domains of hetmatiicall rea- soning, game play, and structured problem solving. These methods offer objective, reliable supervision but may require significant engincering or be limitod expressiveness.
Implicit Rewards Implicit Reward frameworks hypothesize that LLMs can learn from feedback signals even when not explicitly labeled as rewards. For instance, "Reward Is Enough" (Song et al., 2025) demon- strates that LLMs can perform in-context reinforcement learning using simple scalar signals embedded in the context window, improving their responses over rounds without explicit RT fine-tuning or supervision. This reveals an inherent capacity for models to interpret and learn from implicit feedback cues present in their input context. Recent work has expanded this concept by showing that LLMs inherently encode reward- like signals through their standard training objectives. Endogenous reward (Li al. 2025e) reveal that standard next-token prediction implicitly learns generalist reward function, which can be extracted from model logits without additional training. Moreover, ImPlicit Self-Improvemenn1 (PIT) framework (Wang et al. 2024i implicitly learns the improvement goal from human preference data without extra human ef- forts by maximizing the quality gap of the response conditioned on a reference response. Unlike rule-based or environment-dertn
external rewards, implicit reward mothods offer unique advantages by discovering and utilizing reward signals that are inherently present in language modeling.
In summary, reward-based evolution provides explicit optimization and strong autonomy but remains sensi- tive to reward design, often trading stability and safety for adaptability and openness.
5.2 Imitation and Demonstration Learning
Imitation and demonstration learning traditionally involves agent that learns to mimic the behavior of an expert (typically human) from a set of demonstrations. In the context of self-evolving agents, this paradigm is adapted and generalized, which is the focus of our survey. Here, the role of the "expert" is not necessarily a fixed, external entity (e.g., human) but rather any source of high-quality demonstration. In self-evolving agents, these "expert exemplars" are typically generated by the agent itself (e.g., a past successful trajectory), by other more capable agents, or synthesized from environmental interactions.
The key distinction between imitation learning and reward-based methods lies in the nature of the feedback. Imitation learning is prescriptive and cempplar-buse:: the agent is provided with complete, successful guide (e.g., a full reasoning trace) and learns to reproduce this behavior. In contrast, reward-based methods are evaluative and sigmal-based: the agent explores on its own and receives a scalar or textual critique, forcing it to infer the path to improvement through trial-and-error and credit assignment.
Furthermore; the evolutionary mechanism differs fundamentally from population- based methods that will be introduced later. While both imitation and reward- reward-based learning typically focus on optimizing a single agent's 's improvement through iterative refinement, population-based methods evolve collection of agents
22

---

Published in Transactions on Machine Learning Research (01/2026)
in parallel. Their progress typically comes from selection pressure across a diverse gene pool, rather than the direct knowledge transfer from an exemplar to an individual. Therefore, imitation learning occupies a unique niche: it relies on the availability of high-quality solutions to directly guide and accelerate the evolution of an individual agent, making it exceptionally powerful when such demonstrations can be reliably and autonomously generated.
5.2.1 Self-Generated Demonstration Learning
Self-gencrated demonstration learning involves agents creating their own training data through iterative refinement processes, where the models learn to improve by generating and selecting high-quality examples from their own outputs.
Bootstrapping Reasoning Capabilities. Zelikman et al. (0222) intrrouucss the foundational framework for self-generated demonstration learning, enabling language models to bootstrap their reasoning capabilities through iterative self-training. This process involves generating reasoning chains for problems, fine-tuning on correct solutions, and repeating this cycle to progressively improve performance without the need for ground- truth reasoning paths. Building on this framework, recent advancements have refined the bootstrapping process through more sophisticated training strategies. For instance, Hosseini et al. (2024) proposes a verifier-guided self training approach, where separate verifier models assess the quality of generated reasoning chains before they incorporated inlo the training data, enhancing the reliability of sell-improvement. Additionally, Koh et al. (2025) introduces adaptive data sampling strategies that dynamically adjust the composition of training data based on model performance across various reasoning tasks, thereby mitigating overfitting to specific problem types. The "Explore to Evolve" paradigm (Wang et al. 2025| extends this concept to deep research web agents by proposing an automated pipeline for generating complex, verifiable training data. The framework directs an agent to first perform proactive online exploration to gather grounded information from the live web, and then to evolve sophisticated aggregation logic to synthesize question-ansee answwrr pairs that require both information-seekinng and deep reasoning.
Multimodal Telf-iraining. Extending self-training to multimodal domains presents unique challenges in generaling high-quality demonstrations that span both visual and textual modalities. Deng et al. (2024) demonstrates how vision-language models can improve iteratively by training on their own generated image descriptions and visual reasoning chains. The approach leverages the model's cxisting visual understanding to generale detailed image descriptions, which are subsequently used to fine-tune the model's visual perception in a bootstrapping manner. Zhhao et al. (2024c) builds on this concept by empowering multimodal large language models serve as powerful data generators, producing diverse training examples across different modalities and tasks through advanced prompt enginecring and quality filtering mechanisms.
5.2.2 Cross-Agent Demonstration Learning
Cross-agent demonstration learning involves agents learning from demonstrations provided by other agents, either within the same system or from external sources, enabling knowledge transfer and collaborative im- provement.
Multi-Agent Bootstrapped Reasoning. Zhhao et al. (2025b) presents a framework for multi-agent sys- tems to learn from each other's successful demonstrations through bootstrapped reasoning. The system maintains an experience library containing successful interaction trajectories generated by different agents, facilitating efficient knowledge sharing and collaborative improvement. Each agent can leverage the collec- live experience of the entire system, thereby accelerating the learning process and enabling the discovery of diverse solution strategies. This framework illustrates how agents can specialize in different aspects of complex tasks while benefiting from the accumulated knowledge of the entire system.
Domain-Specific Demonstration Learning. Domain-specific applications of demonstration learning have proven especially effective in specialized fields where expert knowledge can be effectively transferred through demonstrations. In recommendation systems, techniques such as self-optimized fino-tuning (Tang et al., 2025) enable LLM-based recommender systems learn from their own successful recommendation patterns, creating feedback loop that enhances personalization over time. The system generates high-
23

---

Published in Transactions on Machine Learning Research (01/2026)
quality recommendation demonstrations from successful user interactions and uses these to fine-tune the underlying language model, ultimately leading to more accurate and personalized recommendations.
5.2.3 Hybrid Demonstration Learning
Hybrid demonstration learning combines both self-generated and external remonstrations to create more robust and diverse training regimens that leverage the strengths of each approach.
Recursive Sclf-Imprrvement. Qu et al. (2024b) demonstrates how agents can be trained to systematically improve their behavior through structured self-reflection and demonstration generation. This approach enables language model agents to introspect on their reasoning processes, identify areas for improvement, and generate corrective demonstrations to address these weaknesses. This recursive process establishes a continuous improvement loop, where agents become increasingly skilled at self- celf-diagnosis and self-correction, leading to more robust and adaptable behavior.
Confidence-Guideer Demonstration Selection. Recent developments have focused on more sophisti- cated mechanisms for selecting qugh-tuality demonstrations from both self-generated and external sources, confidence-based approaches (Li et al. 2024b) utilize the model's uncertainty estimates to determine which demonstrations are most likely to contribute positively to learning, filtering out potentially detrimental or low-quality examples. This method addresses a critical challenge in demonstration learning: poor-quality demonstrations can degrade performance. By ensuring that only high- high-confidence, high-quality examples are used for training, this approach helps to maintain the integrity of the learning process.
The effectiveness of imitation and demonstration learning approaches is highly dependent on the quality and diversity of the available demonstrations. While these methods can yield impressive results when high- quality exemplars are present, they face challenges in domains where good demonstrations are scarce where the optimal behavior is not well-represented in the available data. Future research directions include developing more sophisticated demonstration selection and generation strategies, improving the robustness of learning from imperfect demonstrations, and creating better mechanisms for combining demonstrations from multiple sources.
Overall, imitation-based evolution stabilizes learning through high-quality exemplars but often trades explo- ration and generalization for reliability and sample efficiency.
5.3 Population-basee and Evolutionary Methods
Population-based and evolutionary methods are paradigm with long history in improving agent behavior that complements modern learning- based techniques. This approach, drawing inspiration from biological evolution, has deep roots in AI. The concept was formalized into a practical computational tool by John Holland, whose seminal work on the Genetic Algorithm (GA) established the core operators of selection, crossover, and mutation for refining population of solutions (Holland, [976). Building on this, John Koza pioneered Genetic Programming (GP), a powerful extension that directly evolves exccutable programs symbolic expressions, which makes it appropriate for generating agent logic. This paradigm's power was demonstrated by automatically synthesizing novel, human-competitivy results, such as patented analog electrical circuits (Koza) 2010). This success extended into diverse domains, from evolving competitive agents for strategic games like backgammon and chess (Hauptman & Sipper 2007 Azaria & Sipper) 2005) to discovering complex, interpretable policies in agent-based simulations, such as evolving dynamic taxation rules that outperformed static, human-designed strategics (Garuccio 2016).
This paradigm led to landmark achievements in evolving agent systems. For example, the classic 'Evolved Synthetic Creatures' co-evolved agent morphology and neural controllers in simulated 3D world, leading to the discovery of a wide variety of novel and effective locomotion strategios (Sims) 1994). Later, the influential NEAT algorithm addressed a critical challenge by demonstrating how to evolve not just the weights but the entire topology of a neural network (Stanley & Miikkulainen 2002). This enabled the autonomous discovery of complex agent "brains" from simple initial structures, a principle that has had an enduring impact on neuroevolution. These seminal works illustrated that evolution could construct both an agent's physical form and its complex control systems, establishing powerful alternative to manual design.
24

---

Published in Transactions on Machine Learning Research (01/2026)
Building on this foundation, these methods represent different paradigm for agent evolution compared to the reward-based and imitation-based approaches discussed in previous sections. While reward-based meth- ods typically optimize individual agents through iterative reward signals and imitation learning relies on learning from demonstrations, population- based methods maintain multiple agent variants simultancously. This allows for parallel exploration of the solution space and the emergence of diverse capabilities through mechanisms such as selection, mutation, crossover, and competitive interaction (Zhang et al. (025r). By leveraging parallel search and genetic variation, these methods enable broader search coverage and the dis- covery of novel solutions that might be missed by gradient- based optimization. This approach is particularly valuable when the solution space is complex, multimodal, or when the optimal strategy requires fundamental architectural changes rather than parameter fine-tuning,
5.3.1 Single Agent Evolution
Single-agent evolutionary approaches focus on evolving individual agents through population-based mecha- nisms, where multiple variants of an agent compete and evolve over time. These methods can be broadly categorized into two main paradigms: learning from evolution and sclf-play from multiple rollouts.
Learning from Evolution. This paradigm draws directly from biological evolution, maintaining popula- tions of agent variants and applying evolutionary operators to discover improved capabilities. The Darwin Gödel Machine (DGM) (Zhang et al., 2025h) exemplifies this approach through open-ended cvolution of sell-improving agents that maintain an archive of all historical versions, enabling branching from any past 'species' rather than linear optimization. The system achieves referenttll improvement by allowing agents to directly modify their own Python codebase, with evolution driven by empirical performance on coding benchmarks and parent selection balancing performance scores with novelty rewards for diverse exploration. Recent work has further explored using L.MSs themselves to implement core evolutionary operators. For in- stance, LLM GP (Hemberg et al. 2024) uses the LLM to perform mutation, crossover, and selection directly on programs represented as text, leveraging the model's innate knowledge of code to inform the evolution- ary search. Similarly, open-source frameworks like CodeEvolve (Assumpção et al. 2025 have demonstrated that ovolutionary coding agents can achieve state-of-the-art results on mathematical benchmarks, sometimes outperforming proprietary systems like AlphaEvolve (Novikov et al., 2025) by using an island-based genetic algorithm and inspiration- basee crossover. This principle is also explored in Self-Referential Graph Hyper- Networks (Pedersen et al., 2025), where networks learn to generate their own weight mutations, allowing the rate of cvolution itself to become selectable and adaptable.
Beyond evolving code and architecture, this paradigm extends to evolving the model's parameters and in- ternal logic. The Insprre- Population- Based Evolution (GENOME) framework (Zhang et al. 2025r) directly applics genetic algorithms to language model parameter evolution, maintaining populations and using crossover, mutation, and selection operators on model weights. GENOME+ (Zhang et al.) 2025r ex- tends this with particle swarm optimization concepts, adding inheritance mechanisms and ensemble methods that demonstrate gradient-free evolutionary optimization can effectively improve model capabilities through parameter space exploration. EvoLLM-JP (Akiba et al 2025) takes this further by using evolutionary algorithms to optimally merge multiple foundation models into a single, specialized model with superior performance. Furthermore, some frameworks create a tightly integrated feedback loop where evolution helps model fine-tuning. SOAR (Pourcel et al. 2025), for example, alternates between an evolutionary search phase to generate candidate programs and a "hindsight learning" phase that uses all attempts (both suc- oessful and failed) to generate a rich datasct for fine-tuning the agent model, creating a virtuous cycle of self-improvement.
Self-play is a paradigm where agents improve through iterative interaction with versions of them- selves, creating a dynamic and self-sustaining learning process. Its principles were famously demonstrated by systems like AlphaZero (Silver et al. 2017), which achieved superior performance in complex games by learning entirely without human data. The core mechanism is co-evolutionary learning: as an agent im- proves, its opponents (past or concurrent versions of itself) also bocome stronger, generating a perpetual and adaptive curriculum of increasing difficulty. This avoids the stagnation that can occur when training against fixed environment and enables the discovery of novel, emergent strategies.
Self-Play.
25

---

Published in Transactions on Machine Learning Research (01/2026)
This powerful principle has been adapted for LLMs and LLM Agents, enabling them to bootstrap their capabilities from zero or minimal external data. A prominent approach involves a single model or two model instances adopting distinct, co-evolving roles. For instance, Absolute Zero (Zhao et al. 2025a] and R-Zero (Huang et al.) 2025b) employ a "challenger' or 'proposer' agent that generates problems at the frontier of "solver" agent's s capabilities. A more complex multi-agent dynamic is seen Soeratic-Zero Wang et al. 2025mp, where a Solver co-evolves with a powerful Teacher that creates challenges and a Generator that distills the Teacher's strategy for scalable curriculum creation. To address the instability of purely autonomous systems, R-Few (Yu et al. 2025b) introduces a guided approach where the challenger is grounded by a small set of human examples to prevent concept drift and diversity collapse. The system improves through a closed loop where the solver is rewarded for correctness (often verified by execution) and the challenger is rewarded for posing difficult yet solvable problems, thus driving continuous improvement without external labels. Similarly, Self- Challenging Language Model Agents (Zhou et al. 2025e] establishes a framework where an agent alternates between generating and solving complex, multi-step coding tasks, using successful trajectories to fine-tume ilself. The paradigm also exlends more specialized, collaborative roles, as seen in the Sol-Ves framework (Lin et al., 2025b), where an LLM co-evolves its ability to both generate code (solver) and create corresponding unit tests (verifier). Likewise, SPELL (Yang et al.) 2025e) applies this principle to long-context reasoning, with a single model cyclically adopting questioner, responder, and verifier roles to provide reliable reward signals in a domain where programmatic verification is difficult.
Across these approaches, improvement is driven by self-generated learning signals derived from the agent's own trajectories. Self-Play Fine-Tuning (SPIN) (Chen et al.) 2024f) establishes a foundational approach where current models compete against previous versions, creating evolutionary pressure for improvement. SPC (Chen et al., 2025c) advances this with a more sophisticated adversarial co-evolution, featuring 'sneaky generator' that creates deceptive errors and a "step critic' that learns to detect them. STL (Mendes & Ritter 2025) demonstrates self- aaff-teaching through iterative lookahead search, where value models generate training data from their own exploratory rollouts. Recent work, such as EvoTest (He et al.
2025b). extends these ideas by introducing a gradient-free, evolutionary framework that revises an agent's ' prompt, memory, and tools between episodes.
Self-play is distinguished by a unique self-immroveeent mechanism: agent learns through direct interaction with variations of itself. This typically manifests in two ways: model competes against its own past versions to drive iterative refinement (as in SPIN), or a single model adopts distinct, interacting roles, such as a 'challenger' generating novel problems for a "solver' (as in Absolute Zero). This principle of learning from dynamic interaction is different from imitation- based bootstrapping. While mothods like STaR Zelikman el al. 2022) also learn from an agent's 's own outpuls, they do so by filtering and training on static, successful trajectorics. Self-play, in contrast, generates its learning signal from the process of the game-like interaction itself, learning from relative success even when no perfect exemplar exists. This focus single agent's ' lineage also sets it apart from broader population- methods: instead of evolving a large, diverse population, self-play creates a highly focused evolutionary pressure between a minimal set of policies derived from the same agent.
5.3.2 Multi-Agent Evolution
Multi-agent evolutionary methods extend population-based approaches to evolving entire teams or networks of agents, focusing on optimizing collective behavior, coordination strategies, and collaborative architectures. These approaches can be categorized into two main paradigms based on their evolution mechanisms: System Architecture Evolution and Knowledge-Baad Based Evolution.
System Architecture Evolution. This paradigm focuses evolving the structural and coordination aspects of multi-agent systems, including team composition, orchestration strategies, and workflow opti- mization. EvoMAC (Hu et al. (2024d) introduces framework that mimics neural network training for multi-agent systems, implementing 'textual backpropagation" where compilation errors and lest. failures serve as loss signals to drive iterative modifications of agent team composition and individual prompts. A specialized 'updating team' analyzes textual feedback to identify problematic agents and generate modifica- tion instructions, effectively implementing gradient- based optimization in the space of agent configurations rather than model parameters. The FELA framework (Wang et al. 2025el applies this concept to practical
26

---

Published in Transactions on Machine Learning Research (01/2026)
industrial problem, using a multi-agent system with specialized 'Idea,' 'Code," and 'Critic" agents that col- laboratively evolve to generate high-performing features from complex data, guided by principles from both reinforcement learning and genetic algorithms. Puppeteer (Dang et al.) 2025) takes a different approach by focusing on coordination strategy cvolution rather than tcam composition changes. The system employs a centralized orchestrator that evolves its decision policy through reinforcement learning, dynamically select- ing which agents to activate at each step while balancing task performance with computational cost. This "puppeteer-puppe"" paradigm demonstrates how architectural evolution can occur at the coordination level, discovering efficient collaboration patterns and emergent behaviors such tighter coordination among core agents and sophisticated cyclic interaction patterns. AgentO (Xia et al. 2025b) introduces framework that evolves agents from zero data via a co-evolutionary loop between a curriculum agent and an executor agent. The curriculum agent is trained to generale frontier lasks that challenge the executor. Then, the improved tool-use capabilities of the executor in turn drive the creation of a more complex, tool-aware curriculum, establishing a virtuous cycle of self-improvement.
Knowledge-Band Based Based Evolution. This paradigm emphasizes evolving the collective knowledge and expe- rience of multi-agent teams through memory accumulation and case-based learning, primarily operating through in-context learning or in-context-lik, adaptation rather than paramcter updates. MDTeanGPT (Chen et al. 2025ch establishes the foundation for this approach through dual knowledge base system, im- plementing CorrectKB for storing successful cases and ChainKB for capturing failure reflections, enabling the system to learn from both successes and mistakes through structured case retrieval and reasoning enhance- ment. Extending this medical consultation framework, MedAgentSim (Almansoori et al., 2025b) demon- strates how such soowledge-bssed evolution can be applied to real-world diagnostic scenarios, accumulating experience from patient interactions and using retrioval-augmented generation to improve consultation qual- ity over time. Piflow (Pu et al. (0255) applies this paradigm to scientific discovery, maintaining trajectory of principle-outcome pairs and using them to steer hypothesis generation through information-theorete optimization.
In summaar, population based and self-play evolution enhance diversity and open-ended discovery, yet typ- ically incur higher computational cost and lower interpretability compared with single-agent paradigms.
5.4 Cross- cussicu Evolutionary Dimensions
After outlining the three core evolutionary paradigms, we now analyze their cross-cutting dimen- sions rrevealne how different design choices balance feedback type, data source, and learning stability.
Agent self-evolution is a multifaceted process characterized by a number of cross-cutting dimensions that shape how agents learn, adapt, and improve over time. Beyond any single learning algorithm or supervision signal, these dimensions define the core principles underlying the design and analysis of autonomous agents, In this section, systematically compare the major families of self-cvolution methods wethods--reward-- pennttiionddmoonsstttttoo-oooooo
and population- base--aldddngggg several key axes, such learning paradigm (online vs. offline), policy consistency (on-policy vs. off-policy), and reward granularity process-based. outcome-based, or hybrid). We further highlight additional dimensions, including feedback types, data sources, sample efficiency, stability, and scalability, as summarized in Table ! This comprehensive compari- son provides a unified perspective for understanding the strengths, limitations, and design trade-off's inherent in different approaches to agent evolution.
5.4.1 Online and Offline Learning
Another fundamental dimension in the design of self-evolving agents is the learning paradigm, which can be broadly categorized as either offline or online. This distinction depends on whether the agent's evolutionary updates are performed on a static, pre-collected dataset of experiences (offline) or through continuous, direct interaction with a live environment. (online).
Offline Learning In the offline learning paradigm, the learning phase is decoupled from live task exccution. The offine process typically involves cycles of offline data generation, filtering, and model fine-tuning, focus- ing on building a powerful and generalist. foundational model before deployment. A primary strategy in this
27

---

Published in Transactions on Machine Learning Research (01/2026)
Agent Evolution
Offline Learning
Online Learning
Learning Paradigm
Data
Fittering
Model Fine-tuning
Agent
Environment
Policy Consistency
On-policy Evolution
Off-policy Evolution
Replay Buffer
n1
Env
Traj
Human Demos
Other agents
process-based Reward
Hybrid Reward
Outcome-basee Reward
Reward Granularity
Step
Step 2
Step 3
Outcome
Figure 7: Illustration of the cross-cutting evolutionary dimensions underlying agent self-evolution. Learn- ing Paradigm: in offline learning, data are pre-collected and used for filtering and fine-tuning, while in online learning, agents continuously interact with their environments for real-time adaptation. Policy Consistency: on-policy evolution updates policies based on the agent's own trajectories, whereas off-policy evolution relies on replay buffers, human demonstrations, or experiences from other agents. Reward Gran- ularity: feedback can be process-base based (step-level rewards), outcome-based (final-result rewards), hybrid of both. Together, these three orthogonal axes define how agents generate data, adapt their policies, and receive feedback across reward-based imitation-based and population-baser evolution paradigms.
Table 4: Comparison of self-evolution method familics along key dimensions.
Dimension
Reward-based
Emitation/Demonsse Population-basedd
Feedback Type
Scalar reward, natural language, confidence, ex- ternal signals
Demonstration trajecto-
Fitness scores, task suc- cess, competitive signals
ries, exemplars, ratio- nales Self-generated or other agents, humans Usually
Data Source
Self-generated, environ- ment, external rules Outcome paaanee/prrees///e process/Me /hybrid (flexible)
Population generations, multi-agent systems
Reward Granularity
out- Often outcome-level,
come/process
(via sometimes process via
demo steps)
competition
Online/Offline
Both (reward learning, RL, DPO, SFT)
Typically offline, some- times online demo min-
Online evolution batch population up- dates
ing
On/Offfpolicy
Both (DPO, Reflexion, GRPO)
Primarily off-policy, but online variants can be on-policy
Off-polies (population);
self- peaf-play on-policy
Sample Efficiency
Moderate (depends on reward sparsity)
High (if demo quality is high)
Usually low (needs many trials)
Stability
Sensitive to reward do-
Sensitive to demo qual- ity ity/diversity
Sensitive to population diversityyyty
sign
Scalability
Good with automation
Limited by demo collec- tion
High but resource- intensive
domain LLM bootstrapping, where a model enhances own capabilitics using its self-generated content. For example, Self-Instruct/ Wenntu al., (202)) shows how language model can bootstrap its own instruction- following ability by generating new instructions, paired with its own responses, creating synthetic dataset
28

---

Published in Transactions on Machine Learning Research (01/2026)
for fine-tuning, Building on this, (izardLM(Xu et al. 2024 demonstrates how to progressively evolve the complexity of these self-generated instructions, pushing the model's capabilities on more challenging tasks. Although these methods primarily focus on broad capability expansion via synthetic heuristics, acting as a bootstrapping phase, they lay the necessary groundwork for closed-loop, experienco-driven cvolution de- fined in our Tramework. In the context. of GUT and Web agents, offine learning often involves leveraging pre-collected high-quality trajectories for supervised fine-tuning (SFT). OS-Genesis$Sun et al., 2024b) intro- duced a reverse task synthesis method for automatic trajectory creation. Similarly, Ul-Genie (iiao et all, 2025a) employs unified reward model for trajectory evaluation and self-improving loop to generate high- quality trajectories iteratively. Both approaches focus curating rich SFT dataset to enhance the agent's 's capabilities to solve complex tasks. Beyond SFT, offline methods also incorporate reinforcement learning performed on a static dataset of aggnt-environmeen interactions. For example, GUI-R1 (Luo et al. 2025b)
and InfiGUIRRI (Liu et al. 2025eh utilize mule-based rewards and apply Rl-style Guo et al. 2025b training on offinn GUI datasots.
Online Learning In contrast, online learning enables an agent to learn and adapt continuously while it interacts with live or simulated environment. Feedback from each action is used to update the agent's policy, plan, or knowledge base in real-time. This allows for greater adaptability to dynamic or unseen situations. Some agents evolve online not by updating their model weights, but by refining their plans and skill libraries on the fly. For example, Toyager(Wang et al., 2023a) presents an L.M--poweeed agent that learns to play Minecraft by continuously exploring, generating its own curriculum of tasks, and building a persistent skill library from direct experience. AdaPlanner (Sun et al.) 2023) focuses on adapting its plan within a task; it generates an initial plan, receives feedback from the environment, and refines the plan online. Similarly, Swift.Sago(Lin et al., 2023) operates with a fast-and-slow thinking process, where it can reflect failures of its fast, inluitive mode and switch to a more deliberate, tool-using slow mode, adapting its strategy online based on task difficulty. Reinforcement Learning serves as a fundamental mechanism for online learning, enabling agents to learn from environmental reward signals. DigiRL (Bai et al. 2024) demonstrates how to train device-control agents in the wild using autonomous RL, while DistRL (WanR et al. 2024g) proposes an asynchronous distributed framework to make such on-device training feasible. MobileGUl-RI, SSi et al., 2025b) addresses the specific challenges of training GUI agents in online mobile environments by introducing synthetic task generation pipeline combined with group relative policy optimization (GRPO) through trajeclory-aware rewards.
5.4.2 On-policy and Off-policy Learning
While the previous section examined the timing of data collection and learning (online vs oflline), this section focuses on the policy consistency aspect of agent evolution specifically, whether agents learn from experi- ences generated by the same policy they are trying to improve (on-policy) or from exporiences generated by different policies (off-policy). This distinction is crucial for auderstanding how agents utilize their experien- tial data and manage the trade-offs between learning stability and sample efficiency during the evolutionary process.
On-policy Learning. On-policy approaches require agents to learn exclusively from experiences generated by their current policy, ensuring policy consistency but often at the cost of sample efficiency. Reflexion (Shinn et al., 2023) exemplifies this approach through its iterative self-reflection mechanism. The agent generates responses using its current policy, receives feedback on failures, and immediately incorporates this leedback to update its reasoning process for the next. iteration. GRPO (Shao et al., (024b) and DAPO (Yu et al., 2025a continue this path and show the effectiveness of multiple rollouts. The agent always learns from its current behavior, maintaining striet policy consistency. In agent settings, ou-policy methods provide excellent learning stability and avoid distribution mismatch issues that plague off- off-policy methods, However, they suffer from low sample efficiency, as each policy update requires fresh data collection, making them computationally expensive for complex multi-step reasoning or tool use scenarios where generating high-quality trajectories is costly.
Off-policy Learning. Off-policy approaches allow agents to learn from experiences generated by different policies, including previous versions, other agents, or human demonstrations, significantly improving sample efficiency at the cost. of potential distríbution mismatch. Yuan et al. (2024c) demonstrates sophisticated
29

---

Published in Transactions on Machine Learning Research (01/2026)
off-policy approach where model M4+1 learns from preference data generated by the previous version Mc. The system handles distribution shift through DPO's built-in KT, divergence constraint with the reference policy, preventing the new policy from deviating too far from the data-generating policy. Yuan et al. 2023) showcases another powerful off-policy paradigm by learning from diverse response sources-incciie other models, humans; and different sampling strategies through ranking- supervision. The method elegantly sidesteps distribution shift by treating alignment as a ranking problem rather than requiring policy consistency. Zhao et al. (2025b) illustrates off-policy learning in multi-agent settings, where agents learn from 'experience library' containing successful interaction trajectories generated by previous policy versions, enabling efficient reuse of expensive multi-agent coordination data. In agent settings, off-policy methods excel in sample efficiency, allowing agents to leverage historical data, expert demonstrations, and cross-agent learning. They are particularly valuable for multi-step reasoning where successful trajectories are rare and expensive to generate, and for tool use scenarios where agents can learn from diverse execution examples without repeated environmental interaction. However, they face challenges with distribution shift, reward hacking (where agents exploit. inconsistencies between training and deployment policies), and the need for careful regularization to maintain training stability.
5.4.3 Reward Granularity
Another critical choice in the reward design is its gramularity, which determines at what level of detail the agent receives its learning signal. Reward gramularity ranges from coarse-grained outcome-based rewards, which evaluate the overall task completion, to fine-grained process-based rewards that assess each step of the agent's trajectory. Current self-evolution frameworks adopt these varying levels of granularity to tailor foedback mechanisms according to task complexity and the desired learning outcomes.
Outcome-based Reward Outcome-based Reward is a feedback mechanism that evaluates an agent based the successful completion of predefined tasks. This reward is determined solely by the final state of the agent's 's trajectory, regardless of the intermediate steps. A central challenge, particularly in dynamic environments like web or GUI navigation, is to effectively learn from both successful trajectories and the much more frequent failure trajectories. To address this, Direct Preference Optimization (DPO) (Rafailov et al. 2023 is designed to directly maximize the likelihood of preferred responses while minimizing the KL- divergence with the reference policy. Similarly, RRHF (Yuan et al., 2023 employs a ranking loss approach that aligns model probabilities of multiple responses with human preferences by ranking response probabilities without requiring auxiliary value models. Morcover, several works have developed specialized frameworks for agent self-evolution that are built upon outcome-based based rewards. A straightforward approach is rejection sampling finetuning, as used in AutoWebGLM et al. 2024). This method employs a pre- pre-designed reward model to evaluate trajectory outcomes, identify the successful trajectories, and update the model with this high-quality data. [aa et al. 2024) models the GUI navigation task as a Markov Decision Process (MDP) and obtains a final, sparse reward at the end of an episode using VL.Mbasedd evaluator. WebRL et al. 2024) develops a robust outcone-supervised reward model (ORM) to address the feedback sparsity inherent in dynamic web environments. The ORM evaluates task success within self-cvolving curriculum framework, enabling agents to learn from unsuccessful attempts and progressively improve.
Process-based Reward In contrast to outcome-basee rewards, which provide single, delayed signal, the process-based reward paradigm offers more precise and granular feedback by evaluating each step in an agent's trajectory. Process- supervised reward models (PRMs) have been demonstrated to be significantly more reliable than outcome-supervised reward models (ORMs), particularly in domains requiring complex reasoning like solving math groblems[ighhtnn et al. 2023). However, obtaining such fine-grained step-level feedback traditionally requires extensive human annotations, which are both time-consuming and expensive to scale. To address this annotation bottleneck, Math-Shepherd. (Wan et al. 2023b) proposes an automatic process annotation framework that utilizes Monte Carlo Tree Scarch (MCTS) to gather step-wisc supervision by assessing each step's potential to derive the correct final answer. Similarly, AlphaMath( 2024a) trains a value model to evaluate the step correctness in solution paths and updates both the policy and value model through exploration and exploitation within an MCTS framework. By leveraging process- based rewards, agents can improve their capabilities in a progressive, step-by-step manner. SStar-Math(Guaa et al. 2025) and Agent (Chouduurrrrryy 2025) both propose methods to iteratively evolve the policy and
el al.,
30

---

Published in Transactions on Machine Learning Research (01/2026)
the process reward model, generating progressively higher-quality reasoning paths without manual labels. Agent Q et al., 2024) integrates a step-wise verification mechanism into its MCTS process to collect high-quality trajectories, which are then used to iteratively refine the policy via DPO training.
Hybrid Reward The hybrid methods aim to provide more comprehensive learning signals by incorporat- ing both the clarity of final task success (outcome-based) and the granular guidance of intermediate steps brrocess-based). These methods overcome the sparsity of outcome-only signals while grounding the agent's 's step-by-step reasoning in the ultimate task goal. For example, GiGPO(Feng et al., 2025a addresses the instability of training long-horizon agents by introducing a dual-level reward mechanism. It provides episode-level reward based on the final success of entire trajectories, while simultancously assigning local- ized, step-level reward for intermediate actions. This dual signal provides both a high-level directional goal and low-level corrective guidance. Similarly, SPA-RL (Wang et al. 2025d) proposes a reward decomposition method that bridges the gap between sparse outcome signals and dense process feedback. Tt attributes in- cremental progress each step within multi-step trajectories based on the final task complction, effectively distributing the outcome-based reward across the process steps. This approach creates dense intermediate progress rewards that enhance reinforcement learning effectiveness while maintaining alignment with the ultimate task objectives.
5.5 Other Dimensions of Self-Evolution Methods
In addition to the core axes of learning paradigm, policy consistency, and reward granularity, Table highlights several other important dimensions that differentiate self-cvolution methods:
Feedback Type. The nature of leedback varies widely: reward-based methods leverage scalar rewards, natural language signals, or model confidence; imitation methods focus on demonstration trajectories and rationales; population- mothods use fitness scores or competitive signals. The foodback type funda- mentally determines what information the agent uses to improve.
Data Source. Reward-based methods typically generate data through agent-environment interaction engineered rules, while imitation learning often relies on human expert-generated demonstrations. Population-basee approaches draw from the collective experience of multiple agents or generations, enabling diverse exploration but requiring significant coordination.
Sample Efficiency. Imitation learning generally the most sample-effcient, provided high-quality demon- strations are available, agents can directly mimic expert behavior. Reward- based methods are moderately efficient, with efficiency highly sensitive to reward sparsity. Population-based evolution tends to be sample- inefficient, as it often requires evaluating a large number of agent variants through many trials.
Stability. soward-based learning is sensitive to the quality and design of reward functions, risking re- ward hacking or unintended bchaviors. Imitation learning depends heavily on the quality and diversity of demonstrations. Copulation-based methods are sensitive to population size and diversity, with small or homogeneous populations at risk of premature convergence.
Scalability, Scalability is determined by the feasibility of data or feedback collection and the ability to parallelize learning. Reward-based methods scale well when feedback is automated (e.g., via simulators). Imitation learning is often bottlenecked by the cost of collecting demonstrations. Population-basel ap- proaches can scale to large compute but are highly resource-intensive.
Together, these dimensions offer a more manced,, multidimensional view of self-evolution strategies, guiding practitioners in selecting and designing agent learning pipelines that are best matched to the challenges of their specific domains.
6 Where to Evolve?
Self-cvolving agents have facilitated advancements across a diverse array of domains and applications. Broadly, most of these applications can be systematically categorized into two groups: (1) general domain evolution, where agent systems evolve to expand their capabilities across wide variety of tasks, mostly
31

---

Published in Transactions on Machine Learning Research (01/2026)
General Domain
Memory Mechanism
Modal-Agent Co-Evolution
Curriulum-Driven Training
</>
aill
Specific Domain
Coding
GUI
Financial
Medical
Education
Others
Figure 8: Categorization of where evolve into major types: General Domain Evolution, which focuses broad capability enhancement across diverse tasks (e.g., memory mechanisms, co-cvolution, curriculum training), and Specific Domain Evolution, which largets domain-specific expertise in areas such as coding, GUI, finance, medical, education, and others.
within the digital realm, and (2) specialized domain evolution, which evolves specifically to enhance their proficiency within particular task domains. In essence, evolution in general-purpose assistants focuses on transferring learned experience to broader set of tasks, while evolution in specialized agents emphasizes deepening expertise within specific domain.
6.1 General Domain Evolution
The first category, general domain evolution, refers to self-evolving agents designed for general-purposs ap- plications, particularly as versatile digital assistants. These agents progressively enhance their capabilities to address a broad spectrum of user queries, especially in dynamic and diverse digital environments. Technically speaking, these general assistant agent enhance their abilities primarily via three mechanisms: memory op- timization, curriculum-driven training, and model-agent co-evolution. These mechanisms collectively enable the agents to contimuuusy adapt and effectively respond to increasingly complex user demands.
Memory Mechanism. The most common mechanism facilitating agent evolution is the memory mech- anism, wherein agents summarize historical success paacesssfaiiurre experiences (Wang et al. 2023a. Zhang et al. 2024f into memory representations (Zhang et al.] 2024g), anticipating that these distilled experiences will be beneficial when addressing previously unseen tasks. For instance, AgenttE (Wang et al. 2025o) em- ploys a long-term momory structure consisting of "Tips,' which provide general guidelines, and "Shortcuts," representing reusable action sequences derived from past experiences. This self-evolutionary module supports the continuous enhancement of performance on complex smartphone tasks. Another typical example is Mo- bileSteward (Liu et al., 2025f), which coordinates multiple app-specific Agents under central Agent, with specialized modules for task scheduling, execution, and evaluation. It also incorporates memory-basee self- evolution mechanism that summarizes successful executions to improve future cross-app instruction handling, Meanwhile, Generative Agents (Park et al. 2023h store episodic memories of their experiences, synthesize higher-level reflections, and condition future planning on this self-refloction. In these examples, memory serves as the foundation that enables agents to internalize past experiences, abstract high-level patterns, and refine their future behavior.
Model-Agent Co-Evolution. Another line of work is to perform Model-Agent Co-evolution for LLM agents. UT-Genie (Xiao al., (025a) constructs specialized image-text reward model that scores trajec- tories at both step and task levels. jointly fine-tunes the agent and reward model using synthetic trajec- tories generated by controlled corruption and hard-negative mining across multiple generations. WebE-
32

---

Published in Transactions on Machine Learning Research (01/2026)
volver (Fang et al. 2025b) introduces co-evolving world model LLM that simulates web environments. It generates synthetic training data by predicting next observations and enables look-ahead reasoning during inference, which greatly improves real-web task success. Absolute Zero (Zhao et al. 2025a) co-evolves a reasoning agent and its internal self-reward model through reinforced self-play. By adversarially generating increasingly challenging reasoning problems and optimizing the agent using internal self-certainly as a reward signal, the framework simultancously updates both the agent's 's policy and the self-rewarding mechanism. To- gether, these methods demonstrate the effectiveness of co-evolving agents and auxiliary models (e.g,, reward or world models) to achieve more robust, generalizable, and scalable learning in LLM agentic systems.
Curriculum-Driven Training. Curriculum-driven training also serves as a critical mechanism for build- ing a self-evolving general assistant. For example, WebRL (Qi et al., 2024 uses self-evolving curriculum: when an agent fails, similar but manageable tasks are automatically generated. Coupled with learned reward model and adaptive policy updates, this yields a success rate uplift on WebArena benchmarks. Voyager (Wang et al., (023a) similarly leverages an automatic, bottom-up curriculum in Minecraft, where GPT-4 proposes appropriate next tasks based on agent progress, building a growing code-based skill li- brary through iterative prompting and environmental feodback. These approaches highlight how curriculum learning enables agents to autonomously expand their capabilities through iterative task adaptation.
6.2 Specialized Domain Evolution
In addition to general digital agents, self-evolving agents have also been effectively applied within specialized domains, where their evolution is tailored to significantly enhance performance within narrower task sets.
Coding. The power of self-cvolving agents extends directly to practical applications like coding, where their ability to autonomously adapt and improve offers a transformative approach to software development. SICA (Robeyns et al., 2025a demonstrates that a self-improving coding agent can autonomously edit its own codebase and improve its performance on benchmark tasks. EvoMAC (Hu et al.] 2024d) introduces a self-evolving paradigm on multi-agent collaboration networks, which automatically optimizes individual agent prompis and multi-agent workflows, significantly improving code generation performance by over- coming the limitations of manually designed systems. AgentCoder (Huang et al. 2024] also focuses multi-agent codc generation framework that self-cvolves through iterative refinement. A programmer agent continuously improves code based on feedback from test executor agent, validated against independent test cases from a test designer, significantly boosting effectiveness and efficiency. Zhang et al. (Zhang et al., 2025b) enable LLM agents to continuously evolve by filtering high-quality answers, stratifying earned expe- riences by difficulty, and adaptively selecting demonstrations from gelf-generated data, leading to significant performance improvements and the construction of ML libraries. While these instances differ in their spe- cific wawtaaiiiss--aniiinn from singlo-agent self-oditing to complex multi-agent collaborative networks and experience-basee learning they commonnly share the core principle of ileralive sell-inprovement and au- tonomous adaptation to enhance coding capabilities. These advancements highlight how self- eellvevolving agents can dramatically enhance coding efficioncy and code quality by contimuously learning and optimizing.
Graphical User Interfaces (GUI). Self-evolving GUI agents extend LLM capabilities from pure text reasoning to direct manipulation of desktop, web, and mobile interfaces, where they must cope with large discrete action spaces, heterogeneous layouts, and partial visual observability. Yuan et al, couple pixel-level vision with self-reinforcement, enabling the agent to iteratively refine click-type grounding accuracy with- out additional human labels (Yuan et al. 2025c On real desktop software, the Navi agent from Win- dowsAgentArena replays and critiques its own failure trajectories, ultimately doubling its task-completion rate across 150 Windows challenges (Bonatti et al. 2024). For open-web automation, Web Voyager fuses screenshot features with chain-of-thought reflection; successive self-finettuing raiscs its end-to-end success on unseen sites from 30% % to 59 % (He et al.] 2024), while ReAP adds episodic memories of past outcomes, recovering a further 29- peppeeeentge-poiin margin on previously failed queries (Azam et al. 2025). Beyond RL and memory, AutoGUT continuously mincs functionality annotations from live interfaces to expand a reusable skill library each training cycle (Li et al.) (2025)), and Mobile deploys a hicrarchical reflection stack that monitors, verifies, and revises smartphone actions in situ [Li et al. 2025ch. Collectively, these
33

---

Published in Transactions on Machine Learning Research (01/2026)
systems epitomize the full triad of self-evolutin- what evolves (grounding modules, skill memories), when it evolves (offline consolidation vs. online reflection), and how it evolves (reinforcement learning, synthetic data, hierarchical monitoring) -charting a path toward universally competent interface agents.
The primary bottleneck in customizing agents for specialized domains like financial tasks lies in efficiently constructing and integrating domain-specifie knowledge base into the agent's lcarning process challenge that can be effectively mitigated by incorporating self-evolving mechanisms. Quant.Agent (Wang et al. 2024d) proposed a two-layer framework that iteratively refines the agent's 's responses and automatically enhances its domain-specific knowledge base using foedback from simulated and real-world environments. This iterative process helps the agent progressively approximate optimal behavior, reduces reliance on costly human-curated datasets, and demonstrably improves its predictive accuracy and signal quality in trading tasks. TradingAgents (Xiao et al. 2024) incorporates dynamic processes such as reflection, reinforcement learning, and feedback loop from real-world trading results, alongside collaborative debates, to continuously refine its strategies and enhance trading performance. These developments underscore the potential of self- evolving agents to revolutionize the financial domain by autonomously building domain expertise, adapting to dynamic market conditions, and continuously improving decision-making and trading performance.
Financial.
Medical. Self-cvolving agents have become a powerful paradigm in medical AI, where adaptability and the ability to evolve are essential for managing the complexity and ever-changing nature of real-world clinical practice. One of the most prominent applications is hospital-scale simulation. For example, Agent Hospi- tal (Li et al.) 2024a) creates closed environments with LLM-driven doctors, patients, and murses, allowing the doctor agent to treat thousands of virtual cases. This process helps these agents autonomously refine and evolve their diagnostic strategies without manual labeling, ultimately achieving strong performance on USMLE-style exams. Similarly, MedAgentSim (Almansoori et al. 2025a) integrates an LLM doctor, patient, and tool agent. It records successful consultations as reusable trajectories and employs chain-of-thoughl re- flection and consensus to drive evolutio,,,,,, improving success rates over successive interactions. Another example is EvoPatient (Du et al., (024) places a doctor agent and patient agent in continuous dialogue. With each generation, they update their memory with high-quality exchanges: the patient develops more realistic symptom narratives, while the doctor learns to ask sharper questions. Notably, this happens with- out explicit gradient updates or hand-crafted rewards. Reinforcement learning is also central to building adaptive medical agents. For instance, AectorAgent-RL. (Feng et al.] [202cc) models consultations as a Markov decision process, using a reward function that scores diagnostic accuracy, coverage, and efficiency. This guides policy-gradiens updates that help the agent ask more relevant questions and reach correct di- agnoses faster than imitation- based approaches, thus achieving self-improvement. In addition, automated architecture-searchh approaches like Learning Be Doctor treat the workflow itself as an evolvable object, iteratively inserting specialist sub-agents or new reasoning hops to cover observed failure modes and improve multimodal diagnostic accuracy (Zhuang et al. 2025). Finally, beyond clinical decision-making, self-evolving agents have also been extended to biomedical discovery. OriGene (Zhang et al.) 2025th functions as a virtual disease biologist that evolves by iteratively refining its analytical process. It leverages human and experimen- tal feedback to update core reasoning templates, adjust tool usage strategics, and refine analytical protocols. Similarly, STELLA (Jin et al., 2022)) is a self-evolving biomedical research agent that improves over time by distilling successful reasoning workflows into reusable templates through its Template Library and expanding its Tool Ocean with external or newly assembled tools to meet emerging analytical needs.
Education. Self-evolving LLM agents have also found strong applications the education domain. At the learner level, self-evolving agents like the personalized tutor PACE (Liu et al., 2025cp adjust their prompts based detailed student profiles and continually refine their questioning during conversations. Meanwhile, LLM self-play framework generates diverse tutor student dialogues that further fine-tune the agent, allowing its teaching strategics to evolve both during and after interactions. Another example is MathVC (Yue et al.) 2025), which employs symbolic persona profiles for virtual students and a meta-planner that orchestrates realistic problem-solving stages. This setup enables the agent's conversational process to evolve step by step toward correct solutions, closcly mirroring how collaborative lcarning naturally unfolds. platform i-vip (Yang) lt al. 2025bb deploy a team of cooperating L.M.M agents a coach, assessor, and feedback generator that
On the instructor side, self-evolving agent systems like the professionalldevelopeeee
34

---

Published in Transactions on Machine Learning Research (01/2026)
critique and enhance each other's outputs in real time. These agents adapt their explanations based on teacher-learners' responses and continue to evolve by incorporating expert feedback after deployment, thereby refining their prompt strategies over time Similarly, EduPlanner (Zhang et al. 2025p) frames lesson-plan creation as an adversarial loop where a planner's draft is repeatedly reviewed and refined by evaluator and optimizer agents until it meets diverse educational goals. Similarly, SEFL (Zhang el al.] 2025k) uses teacher-student self-play to generate large sets of nomework-foodbbcck examples, which then fine-tune a lightweight feedback model. This self-evolving process significantly improves the clarity and usefulness of the comments. Collectively, these examples illustrate how self-evolving LLM agents can dynamically adapt to both learners and instructors, driving more personalized, effective, and scalable educational experiences.
Others. Beyond the four major verticals discussed above, self-evolving agents demonstrate broader appli- cability, delivering superior adaptability and performance in specialized domains where conventional agents often fall short. For instance, Arxiv Copilot (Lin et al. 2024) learns and adapts by incorporating histor- ical user interactions, including generated answers, research trends, and ideas, into its thought database, enhancing its ability provide personalized and augmented academic assistance. In a very different con- text, Voyager (Wang et al., 2023a), an agent in the game Minccraft, excels at solving novel tasks from scratch in new worlds through a process of self-evolution. It continually refines its task goals via auto- matic curriculum, expands its skill library, and enhances its actions using an iterative prompting mechanism without human intervention. Transitioning to domains that require explicit strategic planning, Agents-of- Change (Belle et al., 2025) autonomously refines prompts and rewrites code based on iterative performance analysis and strategic research, thereby helping agents overcome inherent limitations in long-term strategic planning and achieve consistently superior and more coherent gameplay in complex environments like Set- tlers of Catan. Lastly, in the realm of diplomacy, Richclieu (Zhao et al., 2024d) introduces AI diplomacy agents that can self-evolve through their self-play mechanism, which allows the agent lo augment its memory by acquiring diverse experiences without human data, thereby enhancing its strategic planning, reflection, and overall performance in diplomacy activities. While these diverse examples operate in distinct environ- ments from academic research and virtual game worlds to strategic board games and complex diplomatic negotiations- they all share the fundamental characteristic of leveraging continuous learning, self-refinemnnt, and autonomous adaptation to achieve increasingly sophisticated and effective performance within their re- spective domains. These diverse examples reinforce the versatility of self-volvvng agents, showcasing their growing potential to excel in a wide range of complex, dynamic, and human-like like tasks beyond traditional domains.
7 Evaluation of Self-evolving Agents
Evaluating self-evolving agents presents a unique set of challenges that extend beyond the traditional assess- ment of static AI systems. Unlike conventional agents typically evaluated on a fixed set of tasks at a single point in time, self-evolving agents are designed to contimuuusy learn, adapt, and improve through ongoing interaction with dynamic environments. Cousequently, their evaluation must capture not only immediate task success but also crucial aspects such as adaptation over time, knowledge accumulation and retention, long-term generalization, and the ability to transfer learned skills across sequential or novel tasks, all while mitigating catastrophic forgetting. This requires fundamental shift from conventional "single-shot" scoring to longitudinal, cost-aware trajectory view.
7.1 Evaluation Goals, Metrics, and Benchmark Coverage
To effectively evaluate self-evolving agents, must move beyond traditional metrics and establish com- prehensive framework that captures their dynamic, adaptive, and long-term learning capabilities. A truly capable and desirable self-cvolving agent must not only learn and improve but also remember past knowledge, transfer it to new situations, operate sustainably, and behave responsibly. Grounded in these critical requirements for continuous and robust AI, we categorize the key evaluation goals into five core dimensions: Adaptivity, Retention, Generalization, Efficiency, and Safety, as illustrated in Table D Each dimension addresses vital aspect of an agent's self-evolutionary process and assessed through its corresponding metrics and benchmark coverage analysis. While Table 7 provides comprehensive catalog of
35

---

Published in Transactions on Machine Learning Research (01/2026)
Evaluation Goals and Metrics
Evaluation Paradigm
Long-horizon Lifelong Learning Ability Assessment
Static Assessment
Short-horizon Adaptive Assessment
Adaplivity
Goal
Retention
Safety
External Task-Solving
Augmented Traditional Benchmarkss
Lifelong Benchmarks
Evaluation
Dynamic/ Evolving Benchmarks
Internal Agent
Built-in Dynemic
Companents Evaluation
Benchmarks
Generalization
Efficiency
An overview of evaluation goals and paradigms for self-cvolving agents. Left: Evaluation Goals and Metrics. We summarize five core objectives guiding agent evaluation: adaplivily, relention, generalization, efficiency, and safety, which reflect how agents should evolve and perform over time. Right: Evaluation Paradigm. Evaluation methods are organized along an incrcasing temporal scalc: from static assessment of fixed capabilities (e.g., external task-solving and component-level evaluation), to short-horizon adaptive assessment capturing within-task adaptation (e.g., augmented or dynamic benchmarks), and finally to long-horizon lifelong learning assessment that examines continual improvement under evolving lifelong benchmarks. Together, these axes link the goals of self-evolution with the evaluation settings used to measure them, forming unified framework for assessing agent adaptivity and sustainability. See Table i for metric definitions, Table 10 for protocol summary.
Figure 9:
evaluation resources, synthesize these limitations to map coverage gaps across these five-goal framework and identify directions for trajectory-oentric, cost-aware assessment at Table S
7.1.1 Adaptivity
Goals & Metrics Adaptivity serves as a foundational evaluation criterion for any self-evolving agent, measuring its ability to improve performance on in-domain tasks through experience. This dimension focuses on quantifying the learning curve and the extent of performance enhancement as an agent iterates and evolves within specific domain. Rather than a static success rate, adaptivity is gauged over time, steps, iterations. Typical metrics include the Success Rate by Iteration Steps (Hu et al. 2024c Wang et al. 2024j Zheng et al. 2025b), which tracks performance in downstream tasks function of the agent's 's interaction history.
Coverage Gaps Although adaptivity has the richest benchmark ecosystem -spaninn code generation (SWE-bende (Jimenez et al.] 2023). MLE-Bench (Chan et al. 2024) ), web navigation (WebArena Zhouu ett al. 2023), WebShop (Yao et al. 2022 ) and general reasoning (GAIA (Mialon et al. 2023 Agent- Bench [Lu et al. 2023b) evaluations remain constrained by practical design choices. ScienceAAent.Bencc limits tasks single programming language and imposes execution-time bounds, excluding computation- ally intensive scientific workflows (Chen et al. 2024e). MLE-Bench offers well-structured tasks but diverges from authentic research scenarios where problem formulation is itself part of the challenge (Chan et al. 2024). These simplifications permit controlled in-domain assessment but confine evaluation to narrow, static task distributions. Moreover, most benchmarks measure improvement. under pre-specified learning proto- cols (e.g., fixed iteration budgets or predetermined replay schedules) rather than assessing whether agents autonomously discover effective adaptation strategies. AgentBench further reports weaknesses sustained reasoning and strategic decision-making (Liu et al. 2023b), yet evaluation horizons remain limited short- term improvement curves.
36

---

Published in Transactions on Machine Learning Research (01/2026)
7.1.2 Retention
Goals & Metrics Retention is a crucial criterion for evaluating the stability of self-evolving agent's knowledge base This dimension specifically focuses on the challenge of catastrophic forgetting, a common issuc in lifelong lcarning where new knowlodge acquisition erodes previously learned information, and knowl- edge retention within extended interactions. Two key metrics can be used to quantily this stability from different perspectives: Forgetting (FGT) and Backward Transfer (BWT) (Zheng al. 2025c). Specifically, let Ji, denote the performance on task after completing tasks:
FGT, Σ max JE (E......)
Jog Jec] BWT, t-1
1
(Ji,- Jta).
A positive BWT indicates that new learning positively benefits old tasks, signifying successful knowledge transfer and a more robust, stable learning process.
Coverage Gaps Retention remains the most underserved dimension. Current memory mechanisms strug- gle significantly with dynamic state updates and maintaining consistency across extended interactions (Hu) et al. 2025). Experience replay exhibits an inherent tension: while replay buffers can improve learning, scal- ing them beyond optimal thresholds triggers performance degradation through context overflow and resource exhaustion (Zheng et al. 2025b). Economic and computational constraints limit evaluation robustness, with some benchmarks conducting minimal repetitions that may not capture stochastic variation (Castillo-Bolle Bolado jtt al. 2024). Most critically, the overwhelming majority of existing benchmarks adopt episodic evaluation where agent state resets between tasks, fundamentally precluding measurement of knowledge accumulation or degradation precisely the phenomena that distinguish self-evolving agents from static systems.
7.1.3 Generalization
Goals & Metrics While Adaptivity and Retention focus on in-domain performance, Generalization is a pivotal measure of a self-evolving agent's ability to apply its accumulated knowledge to new, unseen domains or tasks. A truly intelligent agent should not only perform well within its familiar territory but also demonstrate a capacity for cross-domain generalization. This capability can be evaluated by assessing an agent's 's performance approaches include computing aggregate performance metrics (e.g., mean success rates) across multi-domain test suites [Liu et al. 2023b Sun et al. 2023), and conducting out-of-domain evaluations using held-out task distributions that simulate real-world novelty scenarios (Hu et al. 2024b Peng et al. 2025).
diverse set of tasks that span multiple task distributions and domains. Common
Coverage Gaps Generalization is lypically evaluated through multi-domain test suites (Agent Bench [Liu et al. 2023b), TheAgent Companyy (Xu et al.) (024b) and held-out task distributions, measuring an agent 's ability to transfer knowledge across domains. Current evaluations, however, rely on static snapshots: agents are tested once diverse tasks without tracking whether cross-domain transfer degrades as they evolve over extended learning trajectories. Technological progress further reduces the discrininative power of older tasks, as modern agents benefit from algorithmic advances unavailable to earlier systems (Chan et al. 2024). while training data contamination complicates cross-domain assessment (Chan et al.] 2024). No existing benchmark examines whether agents preserve generalization breadth as they specialize within domains, or whether knowledge acquired in one domain continues transfer after hundreds of learning episodes.
AAgentBench
7.1.4 Efficiency
Goals & Metrics Efficiency quantifies the resourcefulness of a self-evolving agent during its learning and operation. As agents operate continuously and make decisions autonomously, is essential to evaluate the cost and speed of their evolutionary process.
These metrics in Table are particularly important for practical, real-world applications where resources like computation, memory, time, and human effort are finite:
37

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Refined cost taxonomy and units for self-evolving agents, with Real-world example (aan
et al. 2025): On SWE-bench (Jimenez al. 2024), SWE-Agent (Yang al. 2024 + Qwen3-32BB (440K
tokens, 35 calls, 28% success) vs. GPT-4o-mini (8.1M tokens, 181 calls, 10% success) model-scaffold synergy critical for efficiency.
Typical Unit
Dimension
Symbol
Example Measurement
Worked Example
Prompt + completion tokens
Token usage
Cooken
#tokens
440K vs. 8.1M tokens per task; 18x difference
per episode/step
Step count latency
#turus
Ostup
Ressoning/interactionn turns
va. 181 API calls per task; Quality vs. quantity trade-off
rounds
Wall-clock time
End-to-end incl. 1/O
elapsed
runtime
Came
#seconds
12 200+ turns per task; Simple vs. complex bugs
Tool/ API calls
External functionn/eviroomment calls
Crool
#calls
15 vs. 38 calls (AutoCodeRover); High-quality reasoning reduces iterations
Memory growth
#tokens MB
Persistent memory;
Cmeen
Linear growth per call; Token snowball effect amplifies invalid context
context window expansion
Human over- sight
# hours /
Review, labeling, red-team; guidance tokens
Oruman
Failed attempts: cost vs. successful ones; Expensive failures pattern
tokens
Token consumption Hu et al. operations
2024a:: Total Lokens used in reasoning, generation, and memory
Time consumption (Lu al. formance thresholds
2024b): Wall-clock time required to complete tasks or reach per-
Step count (Wang et al.
2023a
Number of interaction rounds needed for task completion
Tool calls: Number of external API/environment invocations, can be combined with performance
gains to assess Tool Productivity (TP) (Wang al. 2025h), formulated as:
Ascore
TPs 8
Σ, cost(tool.)
Memory growth: Expansion of persistent memory and context window over the agent's operational duration
Time and effort required for human intervention, including review, labeling, and corrective guidance
Human oversight:
To relate efficiency to performance gains, we report Cost-per-Gain (CPG) as:
Total Costz
CPG, Performance Gain; +
where cost can be mcasured in tokens, time, memory, human effort, or normalized composite (c.g., monetary cost), and performance gain is the improvement over baseline at horizon t. Lower CPG indicates more efficient learning.
38

---

Published in Transactions on Machine Learning Research (01/2026)
Coverage Gaps Efficiency suffers from sparse and inconsistent reporting of evolution costs. While bench- mark papers occasionally document aggregate resource consumption during evaluation (Chan et al. 2024 they rarely decompose costs into evolution-specific components: tokens consumed during self-refiection or experience replay, wall-clock time spent on architecture search or memory updates, tool invocations trig- gered by autonomous exploration. Cost constraints human baseline collection (Xu et al.; 2024b) reflect broader accessibility challenges but do not illuminate the efficiency of the evolution process itself. More fundamentally, standard evaluation protocols permit unconstrained optimization agents maximize task success without facing the hard token budgets, iteration limits, or latency constraints that govern real-world deployment.
7.1.5 Safety
Goals & Metrics From the perspective of self-evolving, the Safety domain critically examines whether these agents develop unsafe or undesirable behavioral patterns throughout their continuous evolution. This dimension assesses an agent's adherence to predefined rules and its propensity for harmful actions. Key metrics in evaluating safety of self-evolving agents may include: (1) Safety Score (Zhang et al. 2024i
which measures the proportion of test cases where the agent's behavior is labeled "safe"; (2) Harm Score AAdriishcheenko et al. 2024), computed via a detailed manually written grading rubric where outputs earn partial crodit whenever some but not all harmful critcria are triggerod; (3) Completion Under Policy (CuP) (Levy al.] 2024), which assesses whether an agent. successfully completes task while strietly adhering to given set of rules policies; (4) Risk Ratio (Levy et al. 2024), which calculates the frequency of an agent's 's rule violations along specific dimension, providing quantitative mcasure of non-compliance; (5) Refusal Rate (Zhang et al. 2024b Andriushchenko et al. 2024). which evaluates the proportion of tasks an agent refuses to perform due to their aggressive, malicious, or otherwise unsafe nature; (6) Leakage Rate (hha et al. 2024a), which tracks how often an agent unintentionally leaks sensitive or private information.
Coverage Gaps Safety evaluation predominantly captures risks in isolated episodes. Saftyy identifics two core deficiencics: inadequate robustness when deploying tools across varied contexts, and limited recognition of potential hazards in specific operational scenarios (Zhang et al. 2024i). In multi-agent settings, SwarmBench reveals fundamental coordination challenges where local interactions fail to produce coherent collective strategies, with agents unable to maintain shared situational understanding necessary for safe collaborative opcration (Ruan et al., 2025). Yet no benchmark tracks safety trajectories over extended evolution whether risks accumulate through repeated exposure to edge cases, or whether unsafe behaviors emerge through autonomous exploration and self-directed learning.
7.1.6 Self-Directedness and Evaluation Trade-offs
A defining characteristic of self-evolving agents is their capacity for aulonomous exploration and self-directed evolution, distinguishing them from passive learning paradigms. The degree of self-directddnss, specifically whether the agent autonomously generates tasks and evolution strategies or follows externally provided curricula, creates fundamental trade-offs across evaluation dimensions.
Highly self-directed systems demonstrate substantial performance gains through autonomous curriculum generation. WebRL improved from 4.8% to 42.4% by self-generating tasks from exploration failures 'Qi et al. 2024 while SEAgent achieved 23.2 percentage point improvements, advancing from 11.3% to 34.5% via autonomous software exploration and curriculum evolution (Sun et al., 2025c)- However, autonomous evolution incurs mcasurable risks. Alignment faking rates escalated from 12% to 78% when agents au- tonomously evolved under conflicting objectives (Greenblatt et al., 2024), illustrating safely challenges when evolution proceeds with minimal external oversight.
Despite these documented impacts, the field lacks standardized metrics for quantifying self-directedness in evolution. To enable fair comparison, we recommend transparently reporting three aspects of the evolu- tion process. (1) specify whether evolution strategies and task sequences are predetermined, procedurally sampled, or autonomously generated by the agent; (2) Document the source of feedback signals, indicat- ing whether they come from external human labels, rule-based evaluators, or self-generated reflection; (3) Report the frequency of external interventions in the evolution process. Clearly documenting the degree of
39

---

Published in Transactions on Machine Learning Research (01/2026)
Table Overview of Agent Evaluation Metrics Across Core Dimensions
Goal
Metric
Description
Success Rate by Iteration Steps (Hu et al.
Performance downstream tasks function agent's interaction history
2024c
Wang al. 2024 Zheng et al.
Adaptivity
2025b
How quickly an agent reaches a certain performance threshold or converges to an optimal strategy within a given adaptation period
Adaptation Speed Wang et al.
2023a
The average accuracy drop on old tasks after an agent learns a new one, measuring whether useful experience is successfully maintained
Forgetting (FGT)
Zheng et al.
2025c
Retention
Backward Transfer (BWT)
Zheng al.
The average accuracy improvement old tasks due the experience gained from new tasks
2025c
Aggregate Performance
Liu et al.
2023b
Mean success rates or other performance indicators across multi-domain suites gauge overall proficiency
Generalization
Sun et 2023
Out-of- D Domain (OOD) Performance
The agent's performance in held-out task distributions
et al. 2024b Peng et al.
2025
Token Consumption
Hu al.
Computational overhead reasoning and generation steps 2024a
Time Expenditure
et
2024b
Total duration required for task completion
Efficiency
Number of Steps
Waag et al.
2023a
Minimal actions needod to accomplish objectives
The ratio between task benefit (e-g., answer accuracy) and tool usage cost. (e-g,, number of tool calls)
Tool Productivity
Wang et
2025h
Proportion of test cases where agent behavior meets predefined safety criteria
Safety Score
Zhang
2024i
Graded assessment of harmful outputs based violation severity
Harm Score
Andriushchenko et al.
2024
Safety
Completion
Under Policy (CuP)
Levy
Task success while complying with specified constraints
et al. 2024
Risk Ratio
Levy al.
2024
Frequency of policy violations per interaction opportunity
Refusal Rate [Zhang et al.
2024b
An-
Percentage of tasks declined due to safety concerns
driushchenko et al. 2024
Leakage Rate Shao ct al.
2024a
Incidence of unintended sensitive information disclosure
autonomous control is essential for interpreting performance claims, as gains may reflect true self-evolution capability or extensive external guidance.
Such trajectory-centric evaluation aligns with the core objective of assessing self-evolving agents: measuring not just what they achieve, but how autonomously they learn to evolve.
7.2 Evaluation Paradigm
The evaluation of self-evolving agents, given their continuous learning paradigm, necessitates a multi-faceted approach that extends beyond traditional static assessments. Current evaluation paradigm be broadly categorized based on the temporal scope of the assessment: Static Assessment, Short-horizon Adaptive Asscssment, and Long-horizon Lifelong Learning Ability Assessment. Each category addresses different aspects of an agent's 's evolving capabilities, from its instantaneous performance to its long-term learning trajectory. Terminology note. Here, "continuous" refers to system-level self-evolution (changes in policics, memorics, skills, tools, or procodures across episodes), not necessarily parameer continual learning. Our evaluation therefore traces trajectories across interaction episodes and evolving environments, independent of her weights updated.
40

---

Published in Transactions on Machine Learning Research (01/2026)
Table 7: Representative Benchmarks for Evaluating
Agents
Beuchmaakk Name
Tusk Domain
Goal
Core Metries
Task Quantity
Temsporal Scope
202
Somnedgentleechh Chen al
Scientife Data Anabsts
Adaxtivity Etiriisy
Coat
Vidid Eascsion Hate, Surson
Rate. CodrHERRSonne API
102
Statie
MI.E-Bondh Clasn
Mi-Engiawring
Adaysivily
75
Stalic
Taik Scocess Cost,
D8 Beads Jing
2125
Data Science
Adaptivity
Srence Tame, Coupetition kend.
Statie
Accuracy
SWE bauch Finaene ei
2022
Sulwane Engianrring
Adautivily
Frees Rate
2.254
Stadie
OSWerid Xir
2024
Compyttes Use / GUI
Adiativity
Success Rate
369
Statie
Mishlle
Exal- Wang
20050
Competee-UUe GUT
Adgeetvity, Effictency
curnty, Temiitation Eroor
Action Ancunayy Rellection Ac-
Statie, Short-horiton
WekShop You
Brosss
Adageivity
Statto
Webskzean Auon
Web Sourci
Browse
Adaotivity
Success
Statie
WebWalla
Wa L K4AD
Web Soarch
Drowse
Adipeivity, Efficteccy
Accuracy Action Count
580
Statte
2022
ST-WebAAguibbang Tary
Web Soarch
Drowse
Safety
Complrtion under Paley
215
Statie
4ivers Clum
Webh Swanch
Brouse
Adleyeivity
L.M.lllge Store
100
Stadic
Adagsivviy
Accurncy
1,206
Stalie
2012
Generol
Safety
Safety Soore
20,000
Statte
20266
A.Eomazggestteee Dench Zhrng et al.
Geseml
Adapeivity, Gunerdiizaionn
Retcuticon
Success Raic
1306
Loug laorioca
Ages Beach Lia al.
20236
Guarn
tion
Adiativity,
Genenulizo-
Success Rate, F1, Reward, Game
Progres
1300
Stalae
GAIA Mashon al.
2025
Guarnd
Adaptivity
Actursey
Stalie
2024b
Generol
Adaptivity, Etfictnney
pet lustance
Compleston Scome, Steps, Coss
Statte
Accuracy Slope, Posttion of lst
EodLom Don vl al.
21n26
Guamd
Adaptivily, Ellicisnyy
solution. Num of consescutive so-
618
Lecug-hurino
Ialions
PlanBanch
Valmekkm it al
Plaming
Adaeeivvity
Accurny
-26,25
Stadie,
Short-hurines
Natural
Pauniug
Adacsiviiy
Mascc
3,600
Stalie
ACPDench
Kolal
2121
Panning
Adagetiitty
Accuracy
3,730
Statte
AppDench
Wang es al
H124hb
Paaning
Adaptivity
Stecess Rote, F1
500
Statte
Thi mi M14M
Toul Usag
Adaasiviily
Prses Rale, Wis Rade
126.486
Stalie
TootSundle
Tuol Usage
Adautivity
Similarity Sooce
1,032
Statie
Seal-Tinie
Wh 1i
Thal Usegi
Aduepiivity
Accurary, P/R,F1 Acturnyy, ROUGE
1,,70
Stadic
API-Bauk
Usage
Adactivity
4.125
Stadle
Usage
Adageivityy
Dpectt-Spetticc
21,33,
Statte
r-Beuch
5344
Usage
Adkotivity
Pus''s 2
Statie
AeeDend Cin.l o
Tool Usage
Adaptivity
Azcuracy
2.000
Statte
JTSBBonditerr (Ceniiiii-oooo
20124
Agent Memoty
Receation, Dibolency
Socen. Speed, Cost, Virbunity Accurayy First-Try Aurmmry.
Secest Accuracy, GoudAl LTM
Long horzaoa
Story Beerhh Wen Ma. 2005
Agent Memory
Rotention, Rificimey
Lingst Curs. Consump- Cost,
Comn..
311
modes
Shuri-borizum,
Lang-
tho
Mennor Bench (El
et
al.
Ayaul Monsar
Adaptivily
SubEM, Recall, ROUGE F1,
cursty. Ressdlif,, Moshi-llasel
2200
Stalie, Slect--aurieun
2022
Asc/W/
KPI, Tect-Raaed Sorm, Commu-
MultiAgon Bencch Zen
Multi-Agen Collibonntio
Adaxtivity
nicaiim Saore, Plazaing 8ooes
Stadie
Cocelimation Score
SwarmBench Fram et
2025
Agltt-Agest Collabonntion
Adautivity
Pespoclieeppelll:: Mitries
Shart- Sltrrt-ottian
7.2.1
Static Assessment
Static asscssment evaluates the instantancous performance of self-evolving agents at specific point in time. Although these agents are designed for continuous improvement, static methods remain crucial for establishing baseline performance, comparing different agent architectures on fixed task sets, or evaluating capabilities after discrete training phases. This approach aligns with conventional Al evaluation, focusing on immediate performance in fixed environments. While useful for assessing generalization in an "in-domain evolving, out-of-domain evaluation" paradigm, static assessment inherently does not capture the dynamic, continuous lcarning, or long-term evolutionary aspocts central to sclf-evolving agents.
For evaluating an agent's general capabilities given moment, standard benchmarks designed for static Al systems are often employed. These benchmarks offer diverse task domains and test various core agent competencies, providing a snapshot of an agent's proficiency before or at specific stages of its evolution. These assessments can be systematically categorized into External Task-Solving Evaluation and In- ternal Agent Components Evaluation, where External Task-Solving Evaluation measures end-to-end performance in completing domain-specific or cross-domain tasks, and Internal Capability Evaluation fo- cuses on fundamental components in the agent, including planning, tool utilization, memory management, multi-agent coordination, etc.
41

---

Published in Transactions on Machine Learning Research (01/2026)
Table & Goal-to-benchmark mapping with limitations and coverage gaps.
Goal
Benchmarks
Limitations
Coverage Gaps
Restricted programming languages and exe-
Open-ended exploration
SWE-bench; bArena, BencBench, ScienceAgenBeench GAIA
We
cution time limits (Chen al. 2024e]: Sim-
without predetermined ob- jectives; Long-horizon adap- tation under non-stationary distributions
plified problem specifications
vs. aut hentic
Adaptivity
R&D ambiguity (Chan al.
2024 Weak
sustained reasoning and (ocision-making Liu
0t al. 2023b
Persistent challenges in dynamic memory and long-range consistency (Hu et al. 2025 Re-
LgfellongAgenBBeee Bench, LTMBenccmark, Memo- ry Benct
Episodic designs reset state between lasks; No retention with safety constraints
Retention
play bulfer trade-ofTs with context overflow
[Zhnng et al. 2025b]; Limited robustness lest-
ing due to costs (Castillo- Bolado et al. 2024
AgentBench, Compgen,, ALLE-Bencl
GAIA, Progressive task obsolescence from algorith- mic advances (Chan et al. 2024 5 data contamination risks [Chan et al. 2024
Temporal robustness under distribution drifi; Adversarial co-evolutionary assessment
Generalization
Training
High resource demands limit accessibility
No enforced budgets dur- ing evolution; Multi-objective constraints absent
MLE-Benchh Company
TheAgent-
Efficiency
Chan et. al. 2024 Cost constraints prevent
baseline collection
Xu al. 2024b
Static evaluation only; No long- long-horizon safety drift tracking: Co-evolutionary safety unexplored
Inadequate tool robustness and risk awareness
Agent-Safety Bench, SwarmBench
Safety
[hang et al. 2024i : Local-global coordins-
tion disconnect [Ruan et al. 2025
External Task-Solving Evaluation This category assesses an agent's end-to-end proficiency in complet- ing tasks across various real-world simulated environments. In scientific data analysis and machine learning engineering, benchmarks like Aentt (Chen et al. 2024e] and MLE-Bench (Chan
ct al.
2024) test agents' ability to generate and execute code for data analysis and solve Kaggle-style prob-
lems.
For web Search/browsiing environments such as WebShop (Yao et al.
2022), WebbArena ZZouu
ctt al.
2023 Beeb (Wang et al., 2025i), Mind2Web (Deng at al., (Wei et al. 2025b) simulate realistic web interactions, complex browsing scenarios, and lask completion
2023), and BrowseComp
under security constraints. In software engineering, the SWE-bench series (Jimenez et al. 2023 Ope-
nai 2024 Aleithan et al. 2024 Yang et al.) 2024) uses real GitHub issues assess agents' code repair capabilities. For computer-use interactions, OSWorld (Xie et al. 2024 offers unified environment for open-ended tasks involving various desktop and web applications. Specialized domains like marketing also feature benchmarks such as xbench (Chen et al. [222ff). Beyond specific domains, generalist agent benchmarks like AgentBench (Liu et al. 2023b), GAIA (Mialon et al. 2023). and TheAgent (Xu) et al. 2024b) evaluate broad problem-solvingg abilities across multiple knowledge domains and professional tasks, simulating real-world demands on general AT assistants.
Internal Agent Components Evaluation Beyond end-to-end task completion, assessing an agent's 's underlying core competencies is crucial. These benchmarks evaluate fundamental capabilities that contribute to an agent's overall intelligence and self-evolutionary potential. As for Planning, benchmarks such as PlanBench /Valmeekam et al.] 2023), Natural Plan (Zheng al., (2244)), AutoPlanBench (Stein et al. 2025), and ACPBench (Kokel et al.] 2025 comprehensively evaluate an agent's 's ability understand dynamic enviromments, devise strategies, decompose complex problems, and execute reasoning in various planning domains. For Tool Usage, simple benchmarks like ToolAlpaca. (Tang et al. 2023) and ToolBench (Qin
ct al.
2023) test basic sclection and paramcter mapping, while more complex ones like ToolSandbox (Lu
el al.
2024a), Seal-Tools (Wu el. al. 2024). API-Bank (Li et al. 2002),, T-Eval (Chen el. al.) 2023) T-
Bench (Yao et al.] 2024), AceBench (Chen et al. 2025a) simulate real-world scenarios involving multi-ture turn
interactions, implicit state dependencies, and nested calls. Memory Management benchmarks such as
LTMBenchmark (Castillo-Bolant Bolado al. 2024),
(Hu et al., 2025), and StoryBench (Wan & Ma 2025) evaluate the agent's capacity to retain and utilize information across multi-turn interactions,
42

---

The primary metrics and meth- Published in Transactions on Machine Learning Research (01/2026)
Table 9: Differences between Short-horizon Adaptive Assessment and Long-horizon Lifelong Learning Ability
Assessment
Long-horizon Lifelong Learning Ability As- sessment
Dimension
Short-horizon Adaptation Assessment
knowledge accumulation and sustained performance across diverse, evolving tasks and envi- romments.
Coutimuous
Primary
Fo-
Immediate learning and incremental improvement within consistent slightly varying tasks
Miligating catastrophic forgehing: Robust know]- edge transfer; Maintaining afficiecy/ssafety over time; handling true novelty and significant distribu- tion shifts
Core
Chal- Rapid adaptation to minor changes; Improving on similar, repeated tasks
lenges
Small mumber of sequential tasks or iterations over a short period; Improvement on the same or similar task types.
Large, potentially unbounded sequence diverse, cross-domain tasks; Very long interaction periods re- quiring integration of new skills with old
Temporal Scope
dynamic scenarios, and long-rand rang-rage dependencies. For evaluating Multi-Agent Collaboration, benchmarks such as MultcAgenttBench (Zhu et al.) 2025) and SwarmBench (Ruan al. 2025 assess coordination, communication, and emergent swarm intelligence in both collaborative and competitive settings.
Typical metrics for static assessment inchude accuracy, success rate, progress rate, completion rate, and various domain-specific performance indicators (e.g., CodeBertScore, Valid Execution Rate, Pass Rate, F1 score). These motrics provide a singular performance score for an isolated invocation or a fixed set of tasks.
7.2.2 Short-Horizon Adaptive Assessment
Short-horizon adaptations extend beyond static evaluations by assessing an agent's ability to adapt and improve over a relatively short period or a limited number of interactions. The agent might improve per- formance the same task instance with more attempts, or adapt to new instances of the same task type. This category focuses capturing the capacity of the self-evolving agent for immediate adaptability and incremental learning within a relatively consistent or slightly varying task distribution. These evaluation schemes can be broadly categorized into two ways: (1) augment traditional benchmarks with a temporal dimension, and (2) specially design benchmarks and metrics that can inherently support Short-Horizon dynamic learning.
Augmented Traditional Benchmarks Many studies leverage existing benchmarks but introduce a new dimension to track performance over time. This typically involves analyzing performance as a function the number of iterations, steps, or examples. For example, ADAS (Hu et al. (024c) evaluated the held-out test accuracy with the number of agent system itcrations on the ARC benchmark (Chollet, 2019); AWM (Wang et al. 2024j) studied the cumulative success rale over the process of online evaluation under WebArena map test split (Zhou et al., 2023), using a number of examples to mark the evolution progress; WebEvolver (Fang
ct al. 2025b) studied the success rate with self-improving iterations under Mind2web-Live (Pan ct al. 2024
This approach allows for tracking the Adaptivity of the agent within confined scope.
Benchmarks with Built-in Dynamic Evaluation Some benchmarks are designed with short-horizon dynamic learning in mind. AgentteegeeeBeencc (Hu et al. 2025), for example, includes a "Test-Time Learn- ing" (TTL) dimension that evaluates an agent's ability to learn new tasks directly from conversation within a single interaction scssion. In practice, TTL is ovaluated through two types of tasks: Multi-Class Classifica- tion and Recommendation. In these settings, the agent must utilize previously provided information such as labeled examples in context a long movie-related dialogue history- to perform new tasks like map- ping sentences to class labels or recommending relevant movies. This assesses immediate adaptation and knowledge acquisition during ongoing interaction.
Metrics and Methods for Evaluating Short-Horizon Adaptations ods for short-horizon adaptations designed to quantify Adaptivity. These include: (1) Success Rate
43

---

Published in Transactions on Machine Learning Research (01/2026)
by Iteration Steps (Hu et al. 2024c; Wang al. 2024j Zheng et all 2025b which tracks performance improvements as the agent interacts more with the environment or attempts task multiple times; (2) Learning Curve Analysis, which visualizes how performance (e.g., success rate, accuracy) changes over a lim-
ited number of training steps, episodes, or interactions (Hu et al. 2024c; Wang et al., 2024j (3) Adaptation Speed (Wang et al. 2023a), which measures how quickly an agent reaches a certain performance threshold or converges to an optimal strategy within the short horizon.
Short-horizon adaptations are well-suited for evaluating the initial learning capabilities and immediate adapt- ability of self-evolving agents. They can effectively demonstrate whether an agent can learn from recent experiences and improve its performance on in-domain tasks. This category is widely used for current self- evolving agents. However, the limited temporal window makes it challenging to assess long-term knowledge retention (mitigating catastrophic forgetting) and true lifelong learning capabilities across vastly different or soquentially presented tasks.
7.2.3 Long-Horizon Lifelong Learning Ability Assessment
Long-horizon lifelong learning ability assessment is crucial for truly assessing self-evolving agents, as they focus the agent's s ability to continuously acquire, retain, and reuse knowledge across diverse environments and over extended periods. As shown in Table g it mainly focuses on continuous learning, knowledge accumulation, and sustained performance across a diverse and polentially ever-changing stream of lasks or environments over an extended period. This is a nascent but critical area, where unique challenges include catastrophic forgetting, robust knowledge transfer across disparate tasks, efficient resource management over extended durations, and mitigating data leakage when continuously evaluating on evolving data distributions. Specialized benchmarks are emerging to tackle these complexities.
Currently, there are few benchmarks of this type. LTMBBnchmark Castillo-Boladee et al. 2024 is a special- ized benchmark focusing on long-term memory (LTM) evaluation. It assesses LLM agents memory retention and continual learning through dynamic conversational tests, using interleaved dialogues with controlled dis- tractions to simulate real-world recall challenges. Key metrics include task accuracy, memory-span weighttdd LTM Score, and efficiency measures (tests/hour cost) for cross- cros-arrcite comparison. Lifelong Bench (Zheng et al., 2025b) is another pioneering benchmark specifically designed evaluate agent lifelong learning. It constructs sequences of interdependent tasks across domains like Database (DB), Operating System (OS), and Knowledge Graph (KG), requiring agents to progressively build upon previously acquired skills. This allows for systematic tracking of performance improvement and knowledge retention across a prolonged learning trajectory. To address the lack of diverse environments for testing generalization, Au- toEnv (hang et al., 2025i) introduces a framework for automatically gencrating heterogeneons worlds from lactorizable rule distributions. This work also contributes the AUTOENV-36 datasel to systematically mea- sure an agent's cross-environment learning and adaptation capabilities. In addition, there is a solution that constructs dynamic benchmark through continuously updating benchmark datascts @White et al., 2024 Yang et al. 2025c) evolving the benchmark itself by reconstructing original benchmarks to evalu- ate self-evolving agents, which can alleviate data leakage to some extent (Chen et al., 2025g). Benchmark Self-Evolving (Wang et al., (202e),, for example, proposes a solution to continuously update the existing benchmark through iteration. Similarly, the TRACE framework (Guo et al.] 2025a) addresses benchmark saturation by enabling agents evolve tasks to higher difficulty through test-time exploration. The validity of these new, more complex tasks is ensured by a hnlldatteby-rrprooueingg paradigm, which confirms the agent's 's recorded trajectory is reprodueible. Preliminary findings from such dynamic benchmark scenarios have shown that model performance can degrade as the benchmark evolves, highlighting the difficulty of continuous adaptation.
Metrics for long-horizon lifelong learning go beyond simple success rates to quantify the agent's 's evolving ability, such as Forgetting (FGT), Backward Transfer (BWT) (Zheng et al., 2025c), and Cost-per-Gatt Gain. Long-term Generalization metrics could involve assessing performance on a continuously evolving set of out- of- of-distribution tasks or measuring the breadth of tasks an agent can still perform effectively after prolonged learning across many domains.
44

---

Published in Transactions on Machine Learning Research (01/2026)
Table 10: Standardized evaluation protocols
Example for Long-horisoon (EvoAgent Yuaa et al. 2025a
Aspect
Short-horizon
Long-horizon
Optimizes long horison. Success Rate (SR) and Ex- ploration Elficiency (EE) an 67 Minscraft tasks (five tiens] plus Atari; c.g,, Overall SR improves from 21. 90% to 30.21% (velative pain ss 105.9%).
Goal aligned
Adaptivity, efficiency
Retention, generalization, efficiencyy long-term safety
Maintains a persistent Multimodal Experience Pool Full ponsistence model prompt / and continual World Model, updating parameters and experieaace afler each sublask and reusing Ihem across subeequeus tasks without reset.
persistence across taslsy all updates resel afler each epasode
State persistence
meenory lools acroes laoks
Uses a fixedd onrgioorizon Minceraft benchmark (67 tasks split into (Ston//ttoe//ive Gaadnn/GGi//aaaaaaaaaa tiems) plus Atari as a coes-eaviroaueeu lest sel: tacks pre defined rather than streamed ver- stoned.
Streamed sequemees with non- stationnry distribbt fonn:: versioned laske: explicit OOD clusters for trans-
Fixed benelmmark or episodic sampling: IDD aear-IID tack variants
Dataset structure
fer
Eadorces per-subtask stop Taux and matches Dresmeer33 V3's enviromment-etep budgel, reporting wall-clock of 2.7 days ~ days one A1on. but without explicir formulas for Katggo Kueaa or a formal memorry /tool growth polley.
Stage cup Kanago + cunulative cap Ruasi: explicit memoory /tool growth policy
Per-task cap Kahort (iteratione, tool calls, tolarns, call-clook)
Evolution budget
trajectory (states, rewards, completion ratios) into the experi- ence pool CL-based sampling and world-model updittss, but does not publish full replyyble per- iterationn logs, checkoonns, or scheduled retention probes.
Logs each terminated sublack
Per Per-iteration: model /tool versions, full reason- Ingiaction traee,, cost broa knlown
seeds, prouupts,
persistentt state checkpoints, replayable trajectories, scheduled reten- tion probes, evolution decision loys
Above
Required logging
Retention: BWT/FGT, forgetting
Uses SR and EE per plus Overall aggregaae; e.g, on Gold and Diaroon tiers EroAgen roughly doubks SR over baselines, but no BWT/PGT explicit forgetting curves are reported.
Adopticity: success- pneeess-b---teratiion curves, AUT.C: Gonernlinatio::: disttibation irausfer
carws; GeneriMaatinn: temporal & cluster-OOD; Efficienayy Cost-per- Gain (CPG)
Primary metries
withii-
Elficiency is captuzed by EE and wall-clokk (e.g. > 6x fewer ineflective steps on average and 2.7 days training), while CPG, token-drift, and memary-growrll staristies are nos explicitly re- ported.
Cumulative CPG, slage-wise efficienacy token-drift (tokens /task time)) memorr growth rate
Cost-per- Gain (CPG), tokeus per- mucoess, calls, lateney per iteration
Efficiency metrics
Loug horison safety drift tracking; pe- riodic probes /Halleyy//am/eeeee /Cul /Cul/Risk Ratio); persistent Leukage across stapes
Relies on an internal self-verification module with goad similarity and Lous to terminate unproductive subtasks, does marks or track long-horizon safety drift.
Per-episude; Salety Score, Harm Score, Refusal Hate; windom-level Leakage Rate
Sufety auditing
external sufety bench-
Cumulative luman--time + guidance- tolonns; stage wise intervention fre- queney; interverdioon-t-arreess ratio
After initial task specifccation training and evalua- tion are fully autonomoes with no human feed labeling, and luman-time, timennttim///ee aaaman--iine//iinneeeeeee /ar///giddnnnooooo tokees effeetively 0 (not numerscally lly reported).
Humaan time + guidance-tokern per episode; intervention count
Human-in-loop
Provides SR/EE tables over all tiers and ablations over planner ccotroll redlection world model mod- ules, but no learning learning/Sargettingg matrices or detailed oss--awomomy [tokens/tine] (oommorr))))) break- downs.
Leaming ffeegetting matrix: stage ta- bles long-horison eurves; detailed cret taxonomy breakdown
Required outputs
Learning curve AULC; per-task sum- mary table; coet breakdoww
Long-horizon horizon lifelong learning ability assessment is essential for comprehensively evaluating the core promise of self-cvolving agents: their ability to learn contimuously, retain knowledge, and generalize effectively over ex- tended periods. They are critical for assessing Retention, Generalization to truly novel scenarios, and the Efli- ciency of long-term operation. This area remains a key frontier for research in evaluating evolvigg agents. Beyond fixed Long-horizon streams, an open-ended variant continuously evolves tasks/tools, /oool//eviiroomensss
7.2.4 Standardized Evaluation Protocols
To address the heterogeneity of existing setups and to make trajectory-centric evaluation reproducible, we further distill the above considcrations into standardized protocols for short-horizon and long-horizon as- sessment, summarized in Table 10 For short-horizon settings, we assume no cross-lask state persistence and impose a per-task evolution budget Kshort (in iterations, tool calls, tokens, or wall-clock time), require per-iteration logging of prompts, model versions, full reasoning and action traces, and cost breakdowns, and report adaptivity via success-by-iteratin curves and area-underleaaming curve together with basic effi- ciency metries (e.g., perssces,,,, latency). In contrast, long-horizon protocols assume full persistence
45

---

Published in Transactions on Machine Learning Research (01/2026)
of model parameters, prompts, memories, and toolsets across tasks, specify both stage-wise and cumulative evolution budgets (Kstage; (atal) alongside explicit memory/tod growth policies, and mandate richer log- ging: replayable trajectories, persistent checkpoints, scheduled retention probes, evolution decision logs, and human-in-the-loop statistics to expose the degree of self-directedness. Primary long-horizon metrics then expand beyond instantaneous success to include retention (FGT/BWT and forgetting curves), temporal and cluster out-of-distribution generalization, cost-per-gain and efficiency drift over time, as well as long-run safety indicators such as safety incident rates and policy-complianee under evolving behavior. Our worked example, EvoAgent (Yuan et al. 2025a), illustrates both the feasibility and current limitations of prac- tice: it already satisfies key elements of the long-horizon updates, explicit per-subtask step caps, and reporting of SR/EE and wall-clock efficiency but leaves many recommended axes (e.g., standardized relention metrics, explicit CPG/ CiG///oke-drif, and long-term safely drift tracking) unreported, highlighting concrete gaps for future self-evolving agent evaluations to close.
peliocoo-eerssstennt world-model and experience
7.3 Limitations of Current Evaluation Practices
While previous sections outlines the core evaluation dimensions and their coverage, a broader view reveals that current practices still leave substantial blind spots and hinder fair comparisons across methods. To contextualize these limitations, we examine both the capability dimensions that remain under-evaluated and the factors that complicate apples-to-apples comparison under shared settings.
7.3.1 Underserved Capability Intersections
Long-horizon retention with privacy constraints: No benchmark combines extended memory assess- ment (as in LTMBenchmark (Castillo-oolad) et al., 2024) with rigorous safety auditing (as in Agent- SafetyBench (Zhang et al. (024i)), leaving open whether agents can maintain personalization across thou- sands of interactions while guaranteeing zero sensitive information leakage.
Architecture adaptation under operational constraints: Current architecture search methods (e.g.. AFlow (Zhang et al. 2024c), ADAS (Hu et al., 2024ch) operate offline over extended periods to discover optimal workflows. No evaluation examines whether agents can autonomously evolve their architecture selection strategies, learning which lopologies work best for different query lypes, while respecting real-time operational constraints (millisecond response latencies, per-query token budgets). This requires assessing not just final architecture quality, but whether the agent's 's stratogy for choosing or generating architectures improves persistently through experience under hard resource limits.
Tool ecosystem evolution: Existing tool benchmarks provide fixed APls; no evaluation captures the self- directed lifecycle of tool discovery, integration testing, and productivity mcasurement caapulleent-capabliiee demon- strated by systems like Alita but absent from standard assessment.
Multi-agent safety under collaborative evolution: SwarmBench (Ruan et al. 2025 identifies coordi-
nation failures at isolated time points; whether these failures amplify or attenuate over extended multi-agent co-evolution, and whether unsafe behaviors exhibit social contagion when agents learn from each other, remains unexplored.
7.3.2 Challenges for Fair Comparison
To supplement our discussion of evaluation benchmarks, we provide Table 11 which aligns a represen- tative subset of self-evolving agents evaluated under partially matched conditions, i.e., similar domains, benchmarks, and backbone models. This table aims to make explicit how different methods instantiate the hhaatwhhen/how dimensions when the surrounding experimental context is held constant as the litera- ture allows. However, as the table also makes clear, true apples-to-apples comparison remains infeasible at present. Existing works differ substantially in (1) reporting practices: key metrics such as latency, cost, and safety are often omitted or defined inconsistently; (2) evaluation pipelines: prompt formats, rollout budgets, tool access, and environment configurations vary widely; (3) backbone model choices: even nomi- nally similar models differ in size, training data, or inference settings; and (4) architectural design: agents implement distinct control loops, credit-assignment mechanisms, and memory systems that not directly
46

---

Published in Transactions on Machine Learning Research (01/2026)
comparable. Because these factors interact with one another, normalizing results across methods would risk over-interpreting uncontrolled differences. As result, Table 11 is presented as an illustrative snapshot rather than a definitive comparison.
Despite these limitations, the table highlights several qualitative trends: methods using richer what struc- tures (e.g,, architecture-level search) and inter-test when mechanisms often achieve stronger performance in domains where multi-step optimization is feasible, while lightweight intra-test reflection methods tend to be more cost-efficient but yield smaller gains. The absence of consistent aatency/cost//eee
reporting across nearly all methods underscores key gap that limits broader synthesis. We hope this table serves as a concrete reference for how current approaches operationalize the what /how dimensions under comparable settings, while simultaneously motivating the need for standardized reporting to support future apples-to-apples evaluations.
8
Future Direction
8.1 Personalize Al Agents
With the increasing interest in self-evolving agents, deploying personalized agents has become a crucial and increasingly significant objective for the research community (Zhang et al.) 2024h). ) For instance, in applica- tions such as chatbots, digital twins, and emotional support dialogues, a key challenge is enabling AT agents to accurately capture and adapt to users' unique behavioral patterns or preferences over extended interac- tions. Existing personalized agents typically depend heavily on labeled data and post-training methodologies [Cheng et al. 2024]. Recent work by Zhang et al. (2025g) proposes self-generated preference data approach aimed at rapidly personalizing LLMs TWIN-GPT Wang et al. (2024h) leverages electronic health records to create digital twins of patients, enhancing the accuracy of clinical trial outcome predictions. However, these existing strategies hinge on the critical assumption that LLMs can consistently obtain high-quality, large-scale user data.
In practical deployment scenarios, the primary challenge remains the cold-start problem: agents need progressively refine their personalized understanding, accurately interpret user intentions, and effectively construct user profiles, even when initial data is limited. Additionally, significant challenges persist in per- sonalized planning and exccution, such as effective long-term memory management, external tool integration, and personalized generation (ensuring outputs consistently align with individual user facts and preferences) et al. 2025d). Moreover, it is essential to ensure that self-evolving agents do not inadvertently reinforce or exacerbate existing biases and stereotypes, highlighting another critical direction for future research. These governance principles are particularly important as personalized agents continuuuly evolve and adapt their memory or decision-making processes. Building upon these governance principles, evaluation frameworks should also evolve to ensure fairness, accountability, and alignment in personalized settings.
Data governance. Responsible personalization should balance adaptivity with privacy protection. First, agents ought to adopt data minimization: collect only task-relevant data and surface transparent, re- vocable controls. Empirically, web-agent benchmarks show that state-of-(he-arl agents frequently process sensitive data unnecessarily, motivating minimization-by -eeffuul designs (Zharmagambetor et al. 2025) Complementarily, effective data governance for personalized agents entails deploying on-device person- alization frameworks that enable local learning from user interactions, together with user-led privacy mechanisms such as Rescriber that perform on-device redaction and approval prior to any remote data exchange (Zhou et al. 2025c). To prevent indefinite retention, memory decay and forgetting poli- cies should support selective deletion and
unlearning for personalized traces [Staufer 2025). Finally, because self-evolution can drift safety or fairness over time, systems should include bias monitoring and fairness auditing loops that adapt criteria and interventions as the user and context evolve [Basu& & Das 2025), and guard against misevolution (safety/ /alignment degradation during evolution) via continuous checks on memory/ /tto///ooo////o wwoowwowwwwwww updates (Shao al. 2025).
Evaluation.
With the integration of personalized data, evaluation metrics for personalizing self-evolving agents should extend beyond intrinsic evaluations (e.g., directly assessing personalized generated text quality
47

---

Published in Transactions on Machine Learning Research (01/2026)
Table 11: Comparative synthesis of some representative self-evolving agents under shared set- tings. Methods grouped by domain; repeated entries use a_" We summarize each method using the rchat/when /how taxonomy and report performance. Latency, cost, and safety metrics are not consistently reported, limiting full apples-to-apples comparisons.
Area Benchmark
Base model
Method
What
When
How
Perf. (%)
code SWE
Gemini-L1.-prr
Reflexion
Shinn
Context lesson Intra-tost (ICL)
Reward-based (textual)
14.3
2023
reflection
Leart- ww-
Interact
Context / experi- Intra-test (ICL)
Reward-based
18.7
Su al
2025
ence: / reflection
(textual)
Claude-3.5-
Reflexion
Shinn
Context lesson Intra-test (ICL)
Reward-based
54.4
sonnet
al. 2023
reflection
Learn by-
Interact
Context / experi- Intra-test (ICL)
Reward-bassd
60.0
Su al.
2025
ence
WebArena
Gemini-L.5-pro
Reflexion
Shinn
Context
lesson Intra-test (ICL)
Reward- based (textual)
20.2
al.
2023
reflection
Leam- by
Interact
Context / experi- Intra-test (TCL)
Reward-hased
25.6
Su al
2025
emce
web WebArena
Claude-3.5- sonnet
Reflexion
Shinn
Context
lesson Intra-test (ICL)
Reward- based
40.4
al. 2023
reflection Context / experi- Intra-test (ICL)
Learn- by- Interac
Reward-baa sesard-based
48.0
Su et al
2025
ence
WebAreea-Litee GLM-4-9B
Di
Bai et al.
Model policy
Inter-test (RL)
Reward-based
31.5
2024
(external env.)
WebRLL
Qi al.
Model policy
Inter-test (RL)
Reward-based
43.0
2024
(external env.)
Llama3.1-8B
DigiRL
Bai et al.
Model policy
Inter-test (RL)
Reward- based
30.3
2024
WebRL
Qi et al.
Model policy
Inter-test (RL)
neward-based
42.4
2024
math GSMSK
CPT--o-mini
ADAS
Hu ct al.
Architecture multi-agent
Inter-test (RL / Population- 90.5
2024c
sys- SFT hybrid)
workflow search
tem
AFlow
Zhang
Architecture
Inter-test
Population-basee (MCTS)
90.8
al.
2024c
multi-agent topol-
ogy
ScoreFlow
Wang
Architecture
Inter-test.
Imitation preference mization
94.6
al.
2025m
multi-agent
opti-
(query-specific workflow)
MATH
Gemini-1.5- pro-002
ADAS
Hu et al.
Architecture multi-agent
Inter-test
Population-based
80.0
2024c
sys-
tem
AFlow
Zhang
Architecture
Inter-test
Population- based
76.0
al.
2024c
multi-agent topol-
ogy
Mass
Zhang
Architecture sin- Inter-test.
Population- evolutionary
84.7
2025d
gle / multi-agent
design search
using metrics such ROUGE Lin 2004] and BLEU (Papineni et al.) 2002 extrinsic evaluations (e.g., indirect assessments of personalization effects through recommendation systems, classification tasks, and other specific applications). Traditional personalization evaluation metrics often fail to adequately capture the evolving dynamics inherent in self-evolving agents. Consequently, future research calls for more lightweight and adaptive evaluation metrics (Zhang et al. 2024h). Additionally, to better assess self-evolving personalized agents, there clear need for flexible, dynamic benchmarks capable of accurately evaluating agents' performance, particularly in managing long-tailed personalization data throughout their self-evolving processes. We advocate reporting:
48

---

Published in Transactions on Machine Learning Research (01/2026)
Personal Adaptation Gain (PAG): improvement per user over lt sessions vs. non-personalized baseline.
Retention Forgetting Balance: metrics such as forward/ baardd/bbwwrrddd transfer and selective-
forgetting efficacy (e.g., reduction in exposure to user facts following deletion requests).
Privacy Privacy-Utility Trade-off: ratio of utility gain to bits of retained personal data; complemented by a Data Minimization Score (proportion of shared sensitive information only when necessary) in web-agent scenarios (Zharmagambetov al. 2025).
On-device Learning Ratio: proportion of personalization updates or memory writes executed locally on the user device, potentially with user consent/redactee mechanisms (e.g., as studied in Rescriber) (Zhou et al. 2025c).
Bias/Drif Monitors:
longitudinal measurement. of disparity or safety degradation across user groups; one might define indices such as Fairness-Drift or Jafety-Drift though standardized versions remain to be developed Basu & Das
2025
Shao et al. 2025
Longitudinal User Outcomes: tracking session-level satisfaction, goal-completion trends multi-phase development trajectories (e.g,, virtual "campus-life" agents in benchmarks like StuLife) (Cai et al. 2025).
8.2 Generalization
Self-evolving agents also face considerable challenges in achieving robust generalization across diverse task domains and environments. The fundamental tension between specialization and broad adaptability remains one of the most pressing challenges in the field, with significant implications for scalability, knowledge transfer, and collaborative intelligence.
Scalable Architecture Design: central challenge in developing generalizable self-evolving agents lies designing scalable architectures capable of maintaining performance complexity and scope increase. Cur- rent agent systems frequently encounter a trade-off between specialization and generalization, where agents optimized for specific tasks struggle to transfer their learned behaviors novel environments (Chen et al, 2024d). Additionally, the computational cost. associated with dynamic reasoning in L..M-based agents grows non-linearly with the complexity of adaptation mechanisms, imposing practical constraints on achievable generalization within realistic resource limitations (Kim et al.) 2025). Recent studies indicate that self- capabilities show substantial promise for evolving agents equipped with reflective and resource-constrainee models (Liang al. 2024g. Nonethe- enhancing generalization, particularly in smaller, less, these approaches continue to encounter limitations when addressing complex real-world scenarios that require sustained adaptation over prolonged periods.
Cross-Domain Adaptation: Achieving generalization across domains represents a critical frontier for self-evolving agents. Current methods frequently rely on Nomain--speitie fine-tuning, restricting agents' adaptability to new environments without retraining (Belle et al. 2025). Recent advancements in test-time scaling and inference-time adaptation provide promising pathways for enhancing cross-domain generalization (Snell et al.) 2024; Zhang al., 20251). These techniques allow agents to dynamically allocate addit ional reasoning capacity to unfamiliar scenarios by scaling computational resources during inference, avoiding the need for increasing model parameters. Additionally, meta-learning stratogics have demonstrated consider- able polential in facilitating rapid few-shot adapiation to new domains (Bilal et al. 2025). However, their effectiveness critically depends on an agent's capability to accurately determine when supplementary com- putational resources are necessary and efficiently distribute these resources across diverse reasoning tasks.
Continual Learning and Catastrophic Forgetting: Self-evolving agents must continuously adapt to new tasks while retaining previously acquired knowledge, challenge exacerbated by the catastrophic for- getting phenomenon (Ghosal et al. 2024) of continual memorization (Chen et al. (024b) inherent in LLMs [Bell et al. 2025). The stability- dilemma becomes particularly acute in foundation model-based
49

---

Published in Transactions on Machine Learning Research (01/2026)
agents, where the computational costs of retraining for every new task are prohibitive (Zheng et al. 2025c) Recent research has explored parameter-efficienl fine-tuning methods, selective memory mechanisms, and in- cremental learning strategies to mitigate catastrophic forgetting while preserving adaptability (Wang et al, 2024ch. Nonetheless, achieving an optimal balance between efficiency and preventing model drift remains a significant open challenge, especially when agents operate under resource constraints or manage streaming data with stringent privacy considerations.
Knowledge Transferability:: Recent studies have identified critical limitations in knowledge transfer among Al agents. Shi et al. (2025a) emphasized that knowledge integration and transfer capabilities in current agents still require significant optimization. In particular, Geng al. (2025b) found that LLM- based agents often fail to effectively propagate newly acquired knowledge from interactions other agents, rostricting their collaborative potential. Furthermore, |afaa et al. (2025) revealed that foundation models might depend heavily shallow pattern matching, rather than developing robust and transferable inter- nal world models. These findings indicate several important future research directions: 1) it is essential to better understand the conditions under which knowledge acquired by one agent can be reliably generalized and communicated to others; 2) developing methods to quantify the limitations in agents' knowledge trans- ferability could lead to clearer insights into agent collaboration bottlenecks; 3) we need to have an explicit mechanism that encourage the formation of robust, generalizable world models could significantly improve the collaborative effectiveness of self-cvolving agents.
8.3 Safe and Controllable Self-Evolving Agents
As autonomous AT agents become increasingly capable of learning, evolving, and performing complex tasks independently, ensuring their safety and controllability has become paramount concern. Unlike static systems, the very nature of self-evolution introduces unique and amplified risks that emerge dynamically over the agent's 's lifecycle. These risks are not merely extensions of traditional AI safety issues, such as those arising from vague user instructions or environmental threats like malicious phishing links (Zhou et al. 2025d), but are fundamentally tied to the agent's capacity for autonomous self-modification and adaptation (Shao al., 2025 Han et al.] 2025). This section delineates the emergent risks unique to self-evolving agents and then discusses a set of prescriptive guardrails and mitigation strategies for building safer systems.
8.3.1 Emergent Risks in Self-Evolving Systems
Recent research has identified new risks that arise from the autonomous self-improvement process itself. These risks are not necessarily present in the initial agent but can manifest over time as it evolves. We categorize these risks along the primary evolutionary pathways of an agent: the backbone model, memory, and tools.
Uncontrolled behavior drift in model evolution: A core risk is that an agent's 's goals and values may drift away from original human intent as it evolves. This is exacerbated by the learning uncertainty inherent self-evolution, especially when operating in ambiguous contexts (Anwar
ct al. 2024 Bagdasarian et al., 2024p. A phenomenon termed "misovolution" (Shao et al. (2025) can occur during model evolution. For instance, self-training on agent-generated data lead to 'catastrophic forgetting' of safety alignment, causing agents to execute harmful instructions they previously refused, such as interacting with malicious content they were trained to avoid (Sho et al.
2025 Halm et al. 2025
Deployment-time reward hacking in memory evolution: Open-ended evolution is susceptible to reward hacking. Agents may find and exploit loopholes in self-defined reward signals or internal leedback. This is particularly evident in memory evolution, where the accunulation of experience can inadvertently induce unsafe behaviors. For example, an agent might learn to issue unnecessary refunds because its memory correlates them with high satisfaction ratings (Shao et al. 2025). and
such risks can be further exacerbated by poorly designed memory modules (Wang et al. 2025b). A
related concept is the Alignment Tipping Process (ATP)," where an initially aligned agent discovers
50

---

Published in Transactions on Machine Learning Research (01/2026)
that misaligned behaviors are more rewarding, causing its policy to 'tip" and abandon its initial constraints (Han et al. 2025).
Safety of self-created and ingested external tools: The ability of agents to autonomously generate and use tools introduces significant safety issues. Agents may spontaneously create tools with security vulnerabílities or fail to identify malicious code when ingesting external tools (Shao et al. 2025p. This turns the agent into potential vector for security threats. major challenge here that agents still struggle differentiate between necessary and irrelevant sensitive informa- tion (Zharmagambetor et al., 2025). potentially leading them to create tools that leak private data, Furthermore, agents improve, they may become more adept at creating and executing offensive cyber-operations, risk that requires dynamic assessment (Wei et al. 2025a
8.3.2 Prescriptive Guardrails and Mitigation Strategies
Addressing the emergent risks of self-evolution requires moving beyond descriptive warnings to implement- ing prescriptive, actionable guardrails. While early frameworks like TrustAgent have explored multi-stage strategies (i.e., pre-, in-, and post-planning) to foster safer behavior (Hua et al. 2024), the unique dynamics of self-evolution call for a more comprehensive 'safety lifecycle' approach. Future research and development are expected to focus on integrating safeguards at every stage of the agent's operation.
Sandboxing and verification for tool and code exccution: To mitigate risks from tool use, all agent-generated or externally-soureed tools must be executed in a strictly sandboxed environment. Furthermore, automated safety verification, such as static analysis and vulnerability scanning, should be default step before new tool is integrated (Labs] 2025 For securing tool interaction protocols, runtime defense pipelines (Xing et al. 2025 Wang et al. 2025a provide layered detection against threats like tool poisoning and prompt injection.
Audit trails and failsafes for self-modiffcatio: Any self-modification must be accompanied by a comprehensive audit trail. This ensures that changes are traceable and reversible. Implementing rollback and failsafe patterns is critical, allowing the system to revert to a previously known safe state if undesirable bchavior is detected. For memory, proactive defenses like A-MemGuard propose dual-memory structures and consensus- validation to identify and isolate "poisoned" memories before they corrupt behavior (Wang et al., 2025b Wei et al. 2025d].
Continuous monitoring and red-teaming for long-horizon drift: Static, pro-deployment safety evaluations are insufficient. Continuous monitoring of agent. behavior is necessary to detect long-horizon value drift. This can be achieved through red-tcaming scenarios designed to test for emergent misalignment. For GUI agents, hybrid validation frameworks like OS-Sentinel combine formal verifiers with contextual judges to provide robust, in-workflow safety monitoring (Sun et al. 2025b).
Approval gates and requiring human--n-the--lop confirmation should be implemented. Furthermore, given that agents struggle with handling sensitive data (Zharmagambetor et al., 2025), robust privacy-protectin measures necessary to prevent leakage and ensure balanced and secure deployment.
measures: For high-stakes actions, approval gates
To aid practitioners, we synthesize these strategies into compliance checklist for deploying self-cvolving agents (See Table 12p. In conclusion, deploying reliable, controllable, and safe self-evolving systems is a critical and active area research. Future work must move towards building a comprehensive safety- aware evolutionary lifecycle, integrating robust verification, continuous monitoring, and adaptive guardrails to ensure that the agents remain aligned with human values and safety constraints as they become more autonomous.
8.4 Ecosystems of Multi-Agents
Multi-agent self-evolving systems face several unique challenges that require further exploration.
51

---

Published in Transactions on Machine Learning Research (01/2026)
Table 12: Compliance checklist for deploying self-evolving agents
Category
Compliance Checklist for Deployment
Tool & Code
Strict Sandboxing: All tools and agent-generated code execute in an isolated envi- ronment with no default access to host files, network, or sensitive processes.
Safety
D Resource Limiting: The sandbox imposes strict limits on CPU, memory, and execu- tion time prevent denial-of-service or runaway processes.
D Automated Security Verification: A mandatory pipeline performs static analysis (e.g.. SAST) and vulnerability scamning on all new or modified tools before use. D Dependency Scanning: The verification pipeline checks all third-party libraries and dependencies for known vulnerabilities.
Risk-Based Access Control: Tools classified by risk level, and high-risk capa- bilities (e.g., file system writes, API calls) require explicit approval via an approval gate.
Self-Modificatin
Immutable Audit Trail: All seff-modificatoons (to model weights, memory, toolset, or core logic) are logged with details on the trigger, changes made, and outcome. Version Control for Safe States; The agent's state (model, memory, tools) is sioned, with known 'safe' versions clearly tagged.
Control
Tested Rollback Mechanism: A reliable, one-click rollback mechanism exists revert the agent to previously known safe version. This mechanism is regularly tested. Pre- Pre-ee Safety Validation: Before a self-modified model is deployed, it is au- tomatically evaluated against a 'golden dataset' of safety-critiall prompts to prevent catastrophic forgetting of alignment.
Behavioral & Alignment Safety
Continuous Runtime Monitoring: An active monitoring system tracks agent ac- tions, flagging deviations from expected behavior, anomalous resource usage, signs of unsafe actions.
Reward Hacking Detection: Key metrics are monitored for signs of reward hacking (e.g., exploiting loopholes in reward functions). Alerts are configured for sharp, unex- plained metric changes.
I Automated Red- Red-Teaming: A continuous red-teaming framework is active, program- matically generating and running test scenarios to probe for emergent misalignment, value drift,, and new failure modes.
Goal Guardrails: Strict constraints are placed on the agent's ability to modify its fundamental goals safety constraints. Any such change requires human review.
Data Privacy & Memory Integrity
Mechanisms like dual-memory structures or consensus validation are in place to detect. isolate, and neutralize potentially "poisoned" harmful memories before they influence behavior.
Proactive Memory Defense:
D PII Detection and Sanitization: Automated tools are used to detect and redact/ /aatt///nommzeee Personally Identifiable Information (PII) before it is stored in long- term memory used in training.
Data Minimization Principle: The agent is configured to only collect and retain data that strictly necessary for its tasks, and data retention policies are enforced. Privacy Regulation Compliance: The system is designed to comply with relevant data privacy regulations (e.g., GDPR, CCPA).
Operational Controls & Governance
D Human-in-the- Loop for Critical Actions: High-stakes actions (e-g., large financial transactions, data deletion, communication with external users) are gated by mandatory human approval step-
Clear Incident Response Plan: documented plan in place for responding to safety failures, including steps for immediate shutdown, rollback, and analysis.
Centralized Dashboard for Oversight: dashboard provides human operators with real-time visibility into the agent's behavior, state, and active safety alerts. Explainability & Traceability: The system provides clear explanations for why a particular action was taken, linking it back to specific memories, goals, model inferences in the audit trail.
Balancing Individual and Collective Reasoning: Recent studies highlight the difficulty of balancing independent reasoning with effective group decision-making in multi-agent environments (Chen et al. 2025d
52

---

Published in Transactions on Machine Learning Research (01/2026)
Sun al. 2025a]. While collective discussions significantly enhance diagnostic reasoning, agents often risk becoming overly reliant on group consensus, thereby diminishing their independent reasoning capabil- ities. To mitigate this issue, future research should explore dynamic mechanisms that adjust the relative weight of individual versus collective input. Such an approach would help prevent decision-making from being dominated by a single or a small subset of agents, ultimately promoting robust, balanced consensus-builling and innovation. Additionally, developing explicit knowledge bases and standardized updating methodolo- gies leveraging agents' successes and failures could further improve the agents' self-evolution abilities and strengthen their individual reasoning contributions within collaborative contexts.
Efficient Frameworks and Dynamic Evaluation: Another crucial challenge lies in developing efficient algorithms and adaptive frameworks that allow agents to collaborate effectively while preserving their in- dividual decision-making strengths. (Hu et al., 2024d) introduced adaptive reward models and optimized dynamic network structurcs, which can significantly enhance cooperative self-improvement among agents. However, a major gap identified by {Sun et al. 2025al the absence of clear mechanisms for agents to dynam- ically manage and update their knowledge. Addressing this issue will require new frameworks that explicitly integrate continuous learning and adaptive collaboration mechanisms. Furthermore, existing benchmarks for multi-agent evaluation are predominantly static (Zhu et al. 2025) and therefore fail to capture the long- term adaptability and continuous evolution of agent roles. Future benchmarks should incorporate dynamic assessment methods, refeecting ongoing adaptation, evolving interactions, and diverse contributions within multi-agent systems, thus providing more comprehensive evaluation metrics for self-evolving agents.
9 Conclusion
The emergence of self-evolving agents marks paradigm shift in artificial intelligence, moving beyond static, monolithic models toward dynamic agentic systems capable of continual learning and adaptation. As lan- guage agents are increasingly deployed in open-ended, interactive environments, the ability to evolve, adapt- ing reasoning processes, tools, and behaviors in response to new tasks, knowledge, and feedback, has become essential for building the next generation of agentic systems. In this survey, we provide the first compre- hensive and systematic review of self-evolving agents, organized around three foundational questions: what aspects of an agent should evolve, when evolution should occur, and how to implement evolutionary processes effectively. Moreover, discuss several methods for evaluating the progress of self-evolving agents in terms of metrics and benchmarks, followed by corresponding applications and future dircctions. The evolution of these agents will require significant advancements in models, data, algorithms, and evaluation practices, and so on. Addressing issues such as catastrophic forgetting, human preference alignment during autonomous evolution, and the co-cvolution of agents and environments will be key to unlocking agents that are not only adaptive but also trustworthy and aligned with human values. We hope this survey provides foundational framework for researchers and practitioners to design, analyze, and advance the development and progress of self-evolving agents.
References
Emre Can Acikgoz, Cheng Qian, Heng Ji, Dilck Hakkani- Türr, and Gokhan Tur. Self-improving Ilm agents al Lest-time. arXiv preprinl arXiv:2510.070 0,,044,, 2025.
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent S2: A composi- tional generalist-speciillit framework for computer use agents. arXie preprint arxi::2504.000 009006, 2025.
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 7(2):195-204, 2025.
Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. Swe-bench+: Enhanced coding benchmark for 1lss. arXiv preprint arXie:2410.0690 2024.
Mohammad Almansoori, Komal Kumar, and Hisham Cholakkal. Self-evolving multi-agent simulations for realistic clinical interactions. Xix preprint ar 2an2553322677 2025a.
53

---

Published in Transactions on Machine Learning Research (01/2026)
Mohammad Almansoori, Komal Kumar, and Hisham Cholakkal. Self-cvolving multi-agent simulations for realistic clinical interactions. arXiv preprint 2a:25033226776 2025b.
Shun-ichi Amari. Backpropagation and stochastic gradient descent method. Neurocomputing, 5(4-5):185- 196, 1993.
Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, al. Agentharm: A benchmark for measuring harmfulness of Ilm agents. arXiv preprint arx:::2 10. 10.09024, 2024.
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jemner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safely of large language models. arXiv preprinl Xrxi::2404.0 09932, 2024.
Leonhard Applis, Yuntong Zhang, Shanchao Liang, Nan Jiang, Lin Tan, and Abhik Roychoudhury. Unified software engineering agent as software engineer. arXiv preprint arXiv:2006. wrX::2506.14683, 2025.
Henrique Assumçção, Diego Ferreira, Leandro Campos, and Fabricio Murai. Codeevolve: An open source evolutionary coding agent for algorithm discovery and optimization. arXiv preprint arXiv:2510.14 14150, 2025.
Ruhana Azam, Aditya Vempaty, and Ashish Jagmohan. Reflection-angmented planning (ReAP): Memory for web navigation agents. arXiv preprint WxXx:2506.00255 2025.
Yaniv Azaria and Moshe Sipper. Gp-gammon: Using genetic programming to evolve backgammon players. Maarten Keijzer, Andrea Tettamanzi, Pierre Collet, Jano van Hemert, and Marco Tomassini (eds.), Genetic Programming, pp. 132 142, Berlin, Heidelberg, 2005, Springer Berlin Heidelberg. ISBN 978-3- 540-31989-4.
Eugene Bagdasarian, Ren Yi, Sahra Ghalebikesabi, Peter Kairouz, Marco Gruteser, Sewoong Oh, Borja Balle, and Daniel Ramage. Airgapagent: Protecting privacy-conscious conversational agents. In Proceedings of the 2024 on ACM SIGSAC' Conference on Computer and Communications Security, pp. 3868 1188-388,,40 2024.
Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems, 37:12461-12495, 2024.
Debabrota Basu and Udvas Das. The fair game: Auditing & debiasing ai algorithms over time. In Cambridge Forum AI: Law and Governance, volume 1, pp. e27, Cambridge University Press, 2025.
James Bell, Luca Quarantiello, Ethan N Coleman, Ling Li, Meng Li, et al. The future of continual learning in the era of foundation models: Three key directions. arXiv preprint arXiv:2506.0333 2025.
Nikolas Belle, Dakota Barnes, Alfonso Amayuelas, Ivan Bercovich, Xin Eric Wang, and William Wang. Agents of change: Self-evolving 1lm agents for strategic planning- arXiv preprint Xrx::25000004 04651, 2025.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. Forest-of-thought: Scaling test-time compute for enhancing 1lm reasoning. ar preprint arXin:2414 10.00077,, 2024.
Areeb Bilal, Muhammad Abdul Mohsin, Muhammad Umer, Muhammad Arslan Khan Bangash, et al. Meta- thinking in 1mm via multi-agent reinforcement learning: A survey. arXiv preprint (rXi:22504.1552) 2025.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, et al. Windows Agent Arena: Evaluating multi-modal OS agents scale, arXiv preprint or Xix: 08264, 2024.
54

---

Published in Transactions on Machine Learning Research (01/2026)
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMP. 2TAT'2010: 19th International Conference Compulational Stalistics France, August 22-27, 2010 Springer, 2010.
Keynote, Invited and Contributed Papers, pp.
Stefan Braun, Daniel Neil, and Shih-Chii Liu. A curriculum learning method for improved noise robustness in automatic speech recognition. In 2017 25th European Signal Processing Conference (EUSIPCO), pp. 548 552. TEEE, 2017.
Yuxuan Cai, Yipeng Hao, Jie Zhou, Hang Yan, Zhikai Lei, Rui Zhen, Zhenhua Han, Yutao Yang, Junsong Li, Qianjun Pan, et al. Building self-evolving agents via experience-driven lifelong learning: A framework and benchmark. arXiv preprint Xiv:2500. 19005, 2025.
David Castillo- Bolado, Joseph Davidson, Finlay Gray, and Marck Rosa. Beyond prompts: Dynamic conver- sational benchmarking of large language models. In The Thirly-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://opaarevie...etttt otr
twF1D3C992
Jun Shern Chan, Neil Chowdhury, Oliver Jafse, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on 2024.
machine learning engineering. XrXiv preprint arXiv:2410.00
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, et al. Acebench: Who wins the match point in tool usage? arXiv preprint arXiv:2501.1287 2025a.
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process. orXiv preprint arXiv:2405.0955 2024a.
Hardy Chen, Haoqin Tu, Fali Wang, Hui Lin, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft rl? early investigation into training rl-like reasoning large vision-language models. arXiv preprint arXiv:2504.1 1145504.11448 11468, 2025b.
Howard Chen, Jiayi Geng, Adithya Bhaskar, Dan Friedman, and Danqi Chen. Continual memorization of factoids in language models. arXiv preprint 0rXiv:2/11.07115, 2024b.
Jiaqi Chen, Bang Zhang, Ruotian Ma, Peisong Wang, Xiaodan Liang, Zhaopeng Tu, Xiaolong Li, and Kwan-Yat Yee K Wong. Spe: Evolving self-play critic via adversarial games for 1lm reasoning. arXie preprint arXiv:2504.1911 2025c.
Kai Chen, Xinfeng Li, Tianpei Yang, Hewei Wang, Wei Dong, and Yang Gao. Mdteamgpt: A self- evolving lased multi-agent framework for munti-disciplinary team medical consultation. arXiv preprint arXiv:2503. 13856, 2025d.
Kai Chen, Ji Qi, Jing Huo, Pinzhuo Tian, Fanyu Meng, Xi Yang, and Yang Gao. A self-cvolving frame- work for multi-agent medical consultation based on large language models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025e.
Kaiyuan Chen, Yixin Ren, Yang Liu, Xiaobo Hu, Haotong Tian, Tianbao Xie, Fangfu Liu, Haoye Zhang, Hongzhang Liu, Yuan Gong, et al. xbench: Tracking agents productivity scaling with profession-aligned real-world evaluations. arXiu preprint orXiv:2506. 13651, 2025f.
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. When machine unlearning jeopardizes privacy. In Proceedings of the 2021 ACM SIGSAC conference on computer and communications security, pp. 896 911, 2021.
Qizhou Chen, Taolin Zhang, Xiaofeng He, Dongyang Li, Chengyu Wang, Longtao Huang, and Hui Xuc. Lifelong knowledge editing for 11ms with retrieval-augmeeneed continuous prompt learning. arXiv preprint 0r500:2405.03279, 2024c.
55

---

Published in Transactions on Machine Learning Research (01/2026)
Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Tá, Tao Xie, et al. Recent advances in large langauge model benchmarks against data contamination: From static to dynamic evaluation. arXiv preprint arXiv:2502. 17521, 2025g.
Weize Chen, Jiarui Yuan, Chen Qian, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Optima: Optimizing effectiveness and efficiency for llm-based multi-agent system. arXiv preprint 11X0012100008155, 2024d.
Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. T-eval: Evaluating the tool utilization capability of large language models step by step. arXwv preprint arXiv:2812.14033 2023.
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, al. Scienceagentbendt Toward rigorous assessment of language agents for data-driven scientific discovery. arXiv preprint or XrXiv:2 10. 05.05080 2024c.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXi:2201.01 2024f.
Yi Cheng, Wenge Liu, Kaishuai Xu, Wenjun Hou, Yi Ouyang, Chak Tou Leong, Xian Wu, and Yefeng Zheng. Autopal: Autonomous adaptation to users for personal ai companionship. arXiv preprint arXiv:2406. 13960, 2024.
Prateck Chhikara, Dev Khant, Saket Aryan, Taranject Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory. ar preprint arXiv:2504.190 19418, 2025.
François Chollet. On the measure of intelligence. ar preprint arXiv:1911.0154/ 2019.
Sanjiban Choudhury. Process reward models for Ilm agents: Practical framework and directions. arXiv preprint arXie:2002.100 10325, 2025.
Yufan Dang, Chen Qian, Xueheng Luo, Jingru Fan, Zihao Xie, Ruijie Shi, Weize Chen, Cheng Yang, Xiaoyin Che, Ye Tian, et al. Multi-agent collaboration via evolving orchestration. ar preprint orXiv:2505. 19591, 2025.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091 28114, 2023.
Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, Quanquan Gu, James Y Zou, Kai-Wei Chang, and Wei Wang. Enhancing large vision language models with self-training on image comprehension. Advances in Newral Information Processing Systems, 37:131369 111119999-31399,, 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Tee, and Kristina Toutanova. Bert: Bidirectional encoder represen- tations from transformers. arXie preprint 1rXi::1810.04805, 15, 2018.
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint arXiv:2310.05/2 2023.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey in-context learning. arXw preprint arXiv:2301.00 002044, 2022.
Shihan Dou, Ming Zhang, Chenhao Huang, Jiayi Chen, Feng Chen, Shichun Liu, Yan Liu, Chenxiao Liu, Cheng Zhong, Zongzhang Zhang, Tao Gui, Chao Xin, Wei Chengzhi, Lin Yan, Qi Zhang, Yonghui Wu, and Xuanjing Huang. Evalearn: Quantifying the learning capability and efficiency of lims via sequential problem solving, 2025. URL https://arxiv.orr httpgaabb/250..2.7777
Yaxin Du, Yuzhu Cai, Yifan Zhou, Cheng Wang, Yu Qian, Xianghe Pang, Qian Liu, Yue Hu, and Siheng Chen. Swe-dev: Evaluating and training autonomous feature-driven software development, 2025. URL
https://arxiv.. wwt.://wwwwwbbb.......ss
56

---

Published in Transactions on Machine Learning Research (01/2026)
Zhuoyun Du, Lujie Zheng, Renjun Hu, Yuyang Xu, Xiawei Li, Ying Sun, Wei Chen, Jian Wu, Haolei Cai, and Haohao Ying. Lims can simulate standardized patients via agent coevolution. ar Xio preprinl arX1v:2412.11116, 2024.
Zhiyu Fan, Kirill Vasilevski, al. Swe-effi: Re-evaluating software ai agent system effectiveness under resource constraints. arXiv preprint arXi::209.00855, 2025.
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat- Song Chua. Alphaedit: Null-space constrained model editing for language models. In The Thirteenth International Conference on Learning Represenlations, 2025a.
Tianqing Fang, Hongming Zhang, Zhisong Zhang, Kaixin Ma, Wenhao Yu, Haitao Mi, and Dong Yu. Webevolver: Enhancing web agent self-improvement with coevolving world model. arXiv preprint arXiv:2504.2102. 2025b.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for Ilm agent training. Xixiv preprint 2rXiv:2505.109 2025a.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for Ilm agent training, 2025b. URL hhttps://arxii org/
Yichun Feng, Jiawei Wang, Lu Zhou, and Yixue Li. Doctoragent-rl: A multi-agent collaborative reinforce- ment learning system for multi-ture clinical dialogue. arXiv preprint arXiv:2505.19630, 2025c.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Prompt- breeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309. 16797, 2023.
Yao Fu, Dong-Kt Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selection of state-aware guidelines for large lan- guage model agents. arXiv e-prints, pp. arXiv-2403, 2024.
Hang Gao and Yongfeng Zhang. PTR: preeision-driven tool recommendation for large language models.
CoRR, abs/2411.09613, 2024.
Hongehong Gao, Yue Liu, Yufei He, Longxu Dou, Chao Du, Zhijic Deng, Bryan Hooi, Min Lin, and Tianyu Pang. Flowreasoner: Reinforcing query-level meta-agents. ar preprinl aiv::22004. 15257, 2025.
Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wn. On designing effective rl reward at training time for Ilm reasoning, arXiv preprint arXiv;2400. 15115, 2024.
Anthony Garuecio. A genetic programming approach to solving optimization problems on agent-based models. Master's thesis, Duquesne University, 2016.
Yingqiang Ge, Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang, et al. Openagi: When Im moets domain experts. Advances in Neural Information Processing Systems, 36:5530-5568, 2023.
Jiahui Geng, Qing Li, Herbert Woisotschlacgor, Zongxiong Chen, Fengyu Cai, Yuxia Wang. Preslav Nakov, Hans-Arno Jacobsen, and Fakhri Karray. comprehensive survey of machine unlearning techniques for large language models. arXi preprint S5Ni:2553300865 2025a.
Jiayi Geng, Howard Chen, Dilip Arumugam, and Thomas L Griffiths. Are large language models reliable ai scientists? assessing reverso-engincering of black-box systems. arXiv preprint arXiv:2505.1 1968,, 2025b.
Gaurav Ghosal, Tatsunori Hashimoto, and Aditi Raghunathan. Understanding finetuning for factual knowl-
edge extraction. arXiv preprint 4rXiv:2/06.14755, 2024.
Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabicn Roger Sato. Alignment faking in large lan- guage models. Anthoopic Research, December 2024. URL httpp::///wut hwtp:/awwwpppc..ccc/reeeeeeeeeee
alignment-fakin
57

---

Published in Transactions on Machine Learning Research (01/2026)
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. or preprinl arXiv:2501.042 045119 2025.
Zhenyu Guan, Xiangyu Kong, Fangwei Zhong, and Yizhou Wang. Richelieu: Self-evolving llm-bascd agents for ai diplomacy. Advances in Neural Information Processing Systems, 37:123471 223471-123497, 2024.
Dadi Guo, Tianyi Zhou, Dongrui Liu, Chen Qian, Qihan Ren, Shuai Shao, Zhiyuan Fan, Yi R. Fung, Kun Wang, Linfeng Zhang, and Jing Shao. Towards self-evolving benchmarks: Synthesizing agent trajectories via test-time exploration under baiidate-b--reproduce paradigın ar preprial or orXiv:2510.004 00,4115 2025a.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. ar preprint arXiv:2501. 12948, 2025b.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. Tn Proceedings of the European conference computer vision (ECCV), pp. 135 150, 2018.
Dongyoon Hahm, Taywon Min, Woogyeol Jin, and Kimin Lee. Unintended misalignment from agentic fine-tuning: Risks and mitigation. Xiv preprinl or arXiv:2508. 14031, 2025.
Siwei Han, Jiaqi Liu, Yaofeng Su, Wenbo Duan, Xinyuan Liu, Cihang Xie, Mohit Bansal, Mingyu Ding, Linjun Zhang, and Huaxiu Yao. Alignment tipping process: How self-evolution pushes Ilm agents off the rails. Xiv preprint ar arXic:2510.0480 2025.
Mohd Ariful Haque, Justin Williams, Sunzida Siddique, Md. Hujaisa Islam, Hasmot Ali, Kishor Datta Gupta, and Roy George. Advanced tool learning and selection system (ATLASS); A closed-loop framework using LLM. CoRR, abs/22033.107// 2025.
Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. urXiv preprinl arXiv:2305.18444 2023.
Ami Hauptman and Moshe Sipper. Evolution of an efficient search algorithm for the matc-in-n problem chess. In European Conference on Genetic Programmingg pp. 78 89, 2007.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Web Voyager: Building end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.1300 2024.
Yufei He, Ruoyu Li, Alex Chen, Yue Liu, Yulin Chen, Yuan Sui, Cheng Chen, Yi Zhu, Luca Luo, Frank Yang, and Bryan Hooi. Enabling sell-improving agents to learn at test time with human-in-the-loop guidance. In Saloni Potdar, Lina Rojas-Barahona, and Sebastien Montella (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 1625-1653, Suzhou (China), November 2025a. Association for Computational Linguistics. ISBN 979-8-89176-333-3 URL httpps://cclatthollse wwwpggg......rreeeeeee org 202/ emnlp- 1157
Yufci Hc, Juncheng Liu, Yue Liu, Yibo Li, Tri Cao, Zhiyuan Hu, Xinxing Xu, and Bryan Hooi. Evotest: Evo- lutionary test-time learning for self-improving agentic systems. arXiv preprint Xrv::22110.122: 13220, 2025b.
Erik Hemberg, Stephen Moskal, and Una-May O'Reilly. Evolving code with a large language model. Genetic Programming and Evolvable Machines, 25(2): 2024.
John H. Holland. Adaptation in natural and artificial systems, 1976.
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint or arXiv:2402. 02457777.77... 2024.
58

---

Published in Transactions on Machine Learning Research (01/2026)
Mengkang Hu, Yao Mu, Xinmiao Yu, Mingyu Ding, Shiguang Wu, Wenqi Shao, Qiguang Chen, Bin Wang, Yu Qiao, and Ping Luo. Tree-planner: Efficient close-loop task planning with large language models, 2024a. URL
Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation. arXiv preprint Xiu:2408.000644 2024b.
Shengran Hu, Cong Lu, and Jeff Clune. arXiv:2408.0844 08435, 2024c.
Automated design of agentic systems. ar preprint
Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory 1l agents via incremental multi-turn interactions. In ICML 2025 Workshop on Long- Contert Foundation Models, 2025.
Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Si- heng Chen. Self-evolving multi-agent collaboration networks for software development. arXiv preprint arXiv:24 10.1 16946, 2024d.
Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang, Trustagent: Towards safe and trustworthy Ilm-based agents throngh agent constitution. In Trustworthy Mulli-modal Foundation Models and AT Agents (TiFA), 2024.
Baixiang Huang, Zhen Tan, Haoran Wang, Zijie Liu, Dawei Li, Ali Payani, Huan Liu, Tianlong Chen, and Kai Shu. Model editing as a double-edged sword: Steering agent ethical behavior toward beneficence or harm. arXiv preprint Xiv:2506.226000 2025a.
Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning Ilm from zero data. arXiv preprint arXiv:2508. 05004, 2025b.
D Huang, JM Zhang, M Luck, Q Bu, Y Qing, and H Cui. Agentcoder: Multi-agent code generation with effective testing and sell-optimization. Universily of Hong Kong, King's College London, Universily of Susser, Shanghai Jiao Tong University, 2024.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXi::2210.11011 116100 2022.
Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning test-time: Active fine-tuning of lims. arXiv preprint arXiv:2410.0e0 08000,, 2024.
Dongwei Jiang, Alvin Zhang, Andrew Wang, Nicholas Andrews, and Daniel Khashabi. Feedback friction: Lims struggle to fully incorporate external feedback. arXiv preprint arXiv:2506. 11930, 2025a.
Lu Jiang, Deyu Meng, Teruko Mitamura, and Alexander G Hauptmann Easy samples first: Self-paced reranking for zero-example multimedia search. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 547-556, 2014.
Yuhua Jiang, Yuwen Xiong. Yufeng Yuan, Chao Xin, Wenyuan Xu, Yu Yue, Qianchuan Zhao, and Lin Yan. PAG: Multi-turn reinforced Ilm self-correction with policy as generative verifier. arXiv preprint arXiv:2006. 251::2506.0446, 2025b.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.067700 2023.
Carlos E Jimenez, John Yang, et al. Swe-bench: Can language models resolve real-world github issucs? In ICLR, 2024.
Ruofan Jin, Zaixi Zhang, Mengdi Wang, and Le Cong. Stella: Self-ovolving LLM agent for biomedical
research. aixio preprint (an:2507.02004 2025. URL https://arxiv.. www.wog.////o.o.ooooooo...o....a 2507 ar502 02004
59

---

Published in Transactions on Machine Learning Research (01/2026)
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents from becoming data science experts?, 2025. URL https://arxiv. http:///aaxiivooraaa//////////// 07703
Leslie Pack Kaelbling, Michael I. Littman, and Andrew W Moore. Reinforcement learning: survey. Journal of artificcial intelligence research, 4:237 285, 1996.
Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainly, 2025. CRL http://aaxi...orrraa///...2....
Zixuan Ke, Austin Xu, Yifei Ming, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. Mas-zero: Designing multi-agent systems with supervision, 2025. URL httpps://axxi....... org www./rggg/////e//////..o/..eeeee 14996
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint or wwXiv:2310.03711 2023.
Jiin Kim, Byeongjun Shin, Jinha Chung, and Minsoo Rhu. The cost of dynamic reasoning: Demystifying ai agents and lest-time scaling from an ai infrastructure perspective. ar preprinl arXim:2506.04 04301, 2025.
Louis Kirsch, James Harrison, Daniel Freeman, Jascha Sohl-Dickstein, and Jürgen Schmidhuber. Towards general-purpent in-context learning agents. Workshop on Distribution Shifts, 37th Conference on Neural Information .... 2023.
Woosung Koh, Wonbeen Oh, Jacin Jang, MinHyung Lec, Hyeongjin Kim, Ah Yeon Kim, Joonkee Kim, Junghyun Lee, Taehyeon Kim, and Se- Young Yun. Adastar: Adaptive data sampling for training self- taught reasoners. arXiv preprint ar Xrx::2500. 16322, 2025.
Harsha Kokel, Michael Katz, Kavitha Srinivas, and Shirin Sohrabi. Acphench: Reasoning about action, change, and planning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 26559 26568, 2025.
John R Koza. Human-competitive results produced by genetic programming. Genetic programming and evolvable machines, 11(3)2251-22 284, 2010.
Aviral Kumar, Rishabh Agarwal, Disha Shrivastava, Feryal Behbahani, Aleksandra Faust, et al. Training language models to self-correct via reinforcement learning. arXie preprint N00::2409..111,, 2024.
Invariant Labs. mep-scan, 2025. URL
Hanyu Lai, Xiao Liu, Tat Long Tong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2024.
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, et al. In-context reinforcement learning with algorithm distillation. arXiv preprint ar Xrx:::2200. 14215, 2022.
Jonathan Lee, Annic Xic, Aldo Pacchiano, Yash Chandak, Cholsca Finn, Ofir Nachum, and Emma Brun- skill. Supervised pretraining can learn in-context reinforcement learning. Advances Neural Information Processing Systems, 36:43057-43083, 2023.
Ido Tevy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov. St-webagentbench: A benchmark for evaluating safcty and trustworthiness in web agents. arXiv preprint arXiv:2410.0670 2024.
Hongxin Li, Jingfan Chen, Jingran Su, Yuntao Chen, Qing Li, and Zhaoxiang Zhang. AutoGUI: Scaling gui grounding with automatic functionality annotations from LLMs. Xix preprint arXio:2502.01977 2025a.
60

---

Published in Transactions on Machine Learning Research (01/2026)
Junkai Li, Yunghwei Lai, Weitao Li, Jingyi Ren, Meng Zhang, Xinhui Kang, Siyu Wang, Peng Li, Ya-Qin Zhang, Weizhi Ma, and Yang Liu. Agent hospital: A simulacrum of hospital with evolvable medical agents. arxiv preprint arxiee:405.. 029577, 2024a.
Loka Li, Zhenhao Chen, Guangyi Chen, Yixuan Zhang, Yusheng Su, Eric Xing, and Kun Zhang, Confi- dence matters: Revisiting intrinsic self-correction capabilities of large language models. arXiv preprint arXiv:2402. 12563, 2024b.
Mengdi Li, Jiaye Lin, Xufeng Zhao, Wenhao Lu, Peilin Zhao, Stefan Wermter, and Di Wang. Curriculum-rlail: Curriculum aligument with reinforcement learning from ai feedback. arXiv preprint 2.Xi:22555.20055, 2025b.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented 1lms. ar preprinl arXiv:2304.0880 2023.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann- Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403 03218, 2024c.
Ning Li, Xiangmou Qu, Jiamu Zhou, Jun Wang, Muning Wen, Kounianhua Du, Xingyu Lou, Qiuying Peng, Jun Wang, and Weinan Zhang. MobileUse: A gui agent with hierarchical reflection for autonomous mobile operation. Xixee preprint or XXiv22507.1688) 2025c.
Xisopeng Li, Pengyue Jia, Derong Xu, Yi Wen, Yingyi Zhang, Wenlin Zhang, Wanyu Wang, Yichao Wang, Zhaocheng Du, Xiangyang Li, et al. A survey of personalization: From rag to agent. orXiv preprint arXiv:2504.111 10147, 2025d.
Yi-Chen Li, Tian Xu, Yang Yu, Xuqin Zhang, Xiong-Hui Chen, Zhongxiang Ling, Ningjing Chao, Lei Yuan, and Hua Zhou. Generalist reward models: Found inside large language models. arXiv preprint arXiv:2506. 222025 2025e.
Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. Autoflow: Automated workflow generation for large language model agents. arXiv preprint 2x7.v:2407.12882, 2024d.
Xuechen Liang, Yangfan He, Yinghui Xia, Xinyuan Song, Jianhui Wang, Meiling Tao, Li Sun, Xinhang Yuan, Jiayi Su, Keqin Li, Siyuan Chen, and Tianyu Shi. Self-evolving agents with reflective and memory- augmented abilities. arXiv preprint arXiv:2409. 008887,,, 2024.
Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang. Marft: Multi-agent reinforcement fine-tuning, 2025. URL https: //axxii//aaxxv.oooo abs a25g//4../5500...... 16129
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Toddy Lec, Jan Leike, John Schulman, lya Sutskever, and Karl Cobbe. Let's verily step by step. In The Twelfth International Conference on Learning Representations, 2023.
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. Swiftsago: A generative agent with fast and slow thinking for complex interactive tasks. Advances in Newral Information Processing Systems, 36:23813 23825, 2023.
Chin-Yer Yew Lin. Rouge: A package for automatic evaluation of summaries. In Tert summarization branches out, pp. 74-81, 2004.
Guanyu Lin, Tao Feng, Pengrui Han, Ge Liu, and Jiaxuan You. Arxiv copilot: self-evolving and efficient LLM system for personalized academic assistance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 122 122-130, 2024. URL Intpp:// arxiv. hwt.:/orggbb/aaaeee 04593 arXiv:2409.04593.
61

---

Published in Transactions on Machine Learning Research (01/2026)
Sam Lin, Wenyue Hua, Lingyao Li, Zhenting Wang, and Yongfeng Zhang. Ado: Automatic data optimization for inputs in Ilm prompts. arXiv preprint arxi::502. 1 11436, 2025a.
Zi Lin, Sheng Shen, Jingbo Shang, Jason Weston, and Yixin Nie. Learning to solve and verify: A self-plag framework for code and test generation. arXiv preprint Xrx:22502. 14948, 2025b.
Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, Yuheng Cheng, Suyuchen Wang, Xiaoqiang Wang, Yuyu Luo, Haibo Jin, Peiyan Zhang, Ollie Liu, Jiaqi Chen, Huan Zhang, Zhaoyang Yu, Haochen Shi, Boyan Li, Dekun Wu, Fengwei Teng, Xiaojun Jia, Jiawei Xu, Jinyu Xiang, Yizhang Lin, Tianming Liu, Tongliang Liu, Yu Su, Huan Sun, Glen Berseth, Jianyun Nie, Ian Foster, Logan Ward, Qingyun Wu, Yu Gu, Mingchen Zhuge, Xiangru Tang, Haohan Wang, Jiaxuan You, Chi Wang, Jian Pei, Qiang Yang, Xiaoliang Qi, and Chenglin Wu. Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems, 2025a. URL httts: www. 2abbb//504009999
Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, et al. Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. arXiv preprint arXiv:2504.0199 01999, 2025b.
Ben Liu, Jihan Zhang, Fangquan Lin, Xu Jia, and Min Peng. One size doesn't fit all: A personalizod conversational (utoring agent. for mathematics instruction. arXiv preprint ar arXiv:2502. 12633, 2025c. URL
https://arxiv.. www.oorg///aao/eeeeoeee 2502.
Bo Liu, Leon Guertler, Simon Yu, Zichen Luu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, et al. Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. arviv preprint or Xiw::2506.24111 2419,, 2025d.
Dongrui Liu, Huiqi Deng, Xu Cheng, Qihan Ren, Kangrui Wang, and Quanshi Zhang. Towards the difficulty for a deep neural network to learn concepts of different complexities. Advances in Newral Information Processing Systems, 36:41284 08:41283-41300 41304, 2023a.
Shaoteng Liu, Haoqi Yuan, Minda Hu, Yanwei Li, Yukang Chen, Shu Liu, Zongqing Lu, and Jiaya Jia. Rl-gpt: Integrating reinforcement learning and code-as-policy Advances in Neural Information Processing Systems, 17:28430-284551 2024.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Iims as agents. arXiv preprint XrXio:2308. 03688, 2023b.
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-rl: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprint ar Xiv:22004 14239, 2025e.
Yuxuan Läu, Hongda Sun, Wei Liu, Jian Luan, Bo Du, and Rui Yan. Mobilesteward: Integrating multiple app-oriented agents with self-cvolution to automate cross-app instructions. arxiv preprint 1:Xix:2502.16796 2025f.
David Lopez-Paz and Marc' Aurclio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, volume 30, pp. 6467-6476, 2017.
Reza Lotfian and Carlos Busso. Curriculum learning for speech emotion recognition from crowdsourced labels. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(4):8155 81688666.6668888 2019.
Jianqia Lu, Wanjun Zhong, Wenyong Huang, Yufei Wang, Qi Zhu, Fei Mi, Baojun Wang, Weichao Wang, Xingshan Zeng, Lifeng Shang, Xin Jiang, and Qun Liu. SELF: Self-evolution with language feedback. arXiv preprint ar XrXi::3310000:: 005333, 2023.
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for Il tool use capabilities. arXie preprint arXiv:2408.0440 2024a.
62

---

Published in Transactions on Machine Learning Research (01/2026)
Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Axis: Efficient human-agent-computer interaction with api-first Ilm-based agents. arxiv preprint 0rXiu2449.171440 2024b.
Xiaoya Lu, Dongrui Liu, Yi Yu, Luxin Xu, and Jing Shao. X-boundary: Establishing exact safety boundary to shield Ilms from multi-turn jailbrakk without compromising usability. arXiv preprint orXiv:2502.0999) 2025.
Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, Rongcheng Tu, Xiao Luo, Wei Ju, Zhiping Xiao, Yifan Wang, Meng Xiao, Chenwu Liu, Jingyang Yuan, Shichang Zhang, Yiqiao Jin, Fan Zhang, Xian Wu, Hanqing Zhao, Dacheng Tao, Philip S. Yu, and Ming Zhang, Large language model agent: A survey methodology, applications and challenges, 2025a. URL https://arxiv hotps://aarrvv.rr//ab//555 250.//2200322600 21460.
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: A generalist rl-style vision-language action model for gui agents. arXiv preprint arXic:2504410445, 2025b.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 2023a. URL abs/2303. 17651
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self- feedback. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Newral Information Processing Systems, volume 36, pp.
Curran Associates, Inc., 2023b.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109 165. Elsevier, 1989.
Ethan Mendes and Alan Ritter. Language models can self-improve at state- estimation for better search. arXiv preprinl Xrm: 028778, 2025.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context.
arXiv preprinl Xic:22110. 15943, 2021.
Amir Moeini, Jiuqi Wang, Jacob Beck, Ethan Blaser, Shimon Whiteson, Rohan Chandra, and Shangtong Zhang. A survey of in-context reinforcement learning. arXie preprint arXiv:2502.07978, 2025.
Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee- ChenCh Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint arXim:2209.00 002::220900299, 2022.
Kolby Nottingham, Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Sameer Singh, Peter Clark, and Roy Fox. Skill set optimization: Reinforcing language model behavior via transferable skills. In JCML. OpenReview.net, 2024.
Alexander Novikov, Ngån Vü, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mchrabian, al. Alphaevolve: coding agent for scientific and algorithmic discovery. arXiv preprint (eXic:2506.1119 2025.
Openai.
Introducing swe-bench
verifidd,
2024.
https://openai..oom htm.////ppaii.com////o/o
introducing-swe mntrrddcchhn--s---bnnnveevvrrr berchbberhh--iiedd
63

---

Published in Transactions on Machine Learning Research (01/2026)
Siru Ouyang, Jun Yan, 1-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. Reasoningbank: Scaling agent self-evolving with reasoning memory. arXiv preprint arXi::2509.25. 25140, 2025.
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifci Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents in online environments. arXiv preprinl or Xiv:2400 12373, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automalic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Shubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling, Sushil Vemuri, Blake Olson, Eric Li, Yu Zhang, James Caverlee, Dileep Kalathil, et al. Curriculun reinforcement learning from easy to hard tasks improves Ilm reasoning. arXiv preprint arXiv:2006. 26666622 2025.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: review. Neural networks, 113:54 71, 2019.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacca of human behavior. In Proceedingss of the 36th annual acm symposium on user interface software and technology, pp. 1 1-22, 2023.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. connected with massive apis. In NewrIPS, 2024.
Gorilla: Large language model
Joachim Winther Pedersen, Erwan Plantec, Eleni Nisioti, Marcello Barylli, Milton Montero, Kathrin Korte, and Sebastian Risi. Hypernetworks that evolve themselves. In Artificial Life Conference Proceedings 37, volume 2025, pp. 17. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info 2025.
Dengyun Peng, Yuhang Zhou, Qiguang Chen, Jinhao Liu, Jingjing Chen, and Libo Qin. Dipo: Towards a robust, efficient, and generalizable prompt optimization framework from a deep-learning perspective. arXiv preprint Xix::22503. 13413, 2025.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom M Mitchell. Competence-basedd curriculum learning for neural machine translation. arXiv preprint arXix:190s 09848, 2019.
Julien Pourcel, Cédric Colas, and Pierre- Yves Oudeyer. Self-improving language models for evolutionary program synthesis: A case study on arc-agi. arXiv preprint arXiv:2507.1417, 2025.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with' gradient descent" and beam search. arXiv preprint arXim:2305.023: 03495,, 2023.
Yingming Pu, Tao Lin, and Hongyu Chen. Piflow: Principlo-aware scientific discovery with multi-agent collaboration. ar preprinl arXiv:2505.15447 2025.
Pranav Putta, Edmund Mills, Naman Garg, Suneet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. orXiv preprint arXiv:2408.071 07199, 2024.
Zehan Qi. Xiao Liu, lat Long Tong, Hanyu Lai, Xueqiao Sun, Jiadai Sun, Xinyue Yang, Yu Yang, Shuntian Yao, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training IIm web agents via self-evolving online curriculum reinforcement learning. arXio preprint or orXi:2441100233 2024.
Chen Qian, Yufan Dang, Jiahao Li, Wei Liu, Zihao Xic, Yifei Wang, Weize Chen, Cheng Yang, Xin Cong, Xi- aoyin Che, et al. Experiential co-learning of software-developing agents. arXie preprint arXiv:2312. 17025, 2023a.
64

---

Published in Transactions on Machine Learning Research (01/2026)
Cheng Qian, Chi Han, Yi Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023b. URL https::/openreviiet httppppewweiiii.eeeoorrrrrrrra.a
aCHqI0rQiH
Cheng Qian, Shihao Liang, Yujia Qin, Yining Ye, Xin Cong, Yankai Lin, Yesai Wu, Zhiyuan Liu, and Maosong Sun. awraaiigate-onsolllaeeeeppoooloo A general stralegy for inter-task agent self-evolution. arXiv preprinl Xiv:24001 13996, 2024a.
Lang Qian, Peng Sun, Yilei Wang, Jiayue Jin, Awediine Boukerche, and Liang Song. A new online evolu- tive optimization method for driving agents. In GLOBECOM 2024-2024 IEEE Global Communications Conference, pp. 22499. IEEE, 2024b.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolll:: Facilitating large language models to master 16000+ real-world apis. arXiv preprint 2rv: 16789, 2023.
Jiahao Qiu, Xuan Qi, Hongru Wang, Xinzhe Juan, Yimin Wang, Zelin Zhao, Jiayi Geng, Jiacheng Guo, Peihang Li, Jingzhe Shi, Shilong Liu, and Mengdi Wang. Alita-g: Self-evolving generative agent for agent generation. arXiv preprint Xrv:225100.22 236011, 2025a.
Jiahao Qiu, Xuan Qi, Tongcheng Zhang, Xinzhe Juan, Jiacheng Guo, Yifu Lu, Yimin Wang, Zixin Yao, Qihan Ren, Xun Jiang, Xing Zhou, Dongrui Liu, Ling Yang, Yue Wu, Kaixuan Huang, Shilong Liu, Hongru Wang, and Mengdi Wang. Alita: Generalist agent enabling scalable agentic reasoning with minimal predefinition and maximal self-evolution. CoRR, abs/2505. als/2505200286, 2025b.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Towards completeness-oriented tool retrieval for large language models. In CIKM, pp. 1930-1940.
ACM, 2024a.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Roon Ji-Rong Wen. From exploration to mastery: Enabling 1lms to master tools via self-driven interactions. In ICLR. OpenReview.net, 2025.
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. Advances in Neural Information Processing Systems, 37:55249-5588 55285, 2024b.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Newral Information Processing Systems, 36:53728 53741, 2023.
Kiran Ramnath, Kang Zhou, Sheng Guan, Soumya Smruti Mishra, Xuan Qi, Zhengyuan Shen, Shuai Wang, Sangmin Woo, Sullam Jcoung, Yawci Wang, et al. A systematic survey of automatic prompt optimization techniques. arXiv preprint 2ix: 16923, 2025.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, (7)2))285, 1990.
Maxime Robcyns, Martin Szummer, and Laurence Aitchison. A self-improving coding agent. arXie preprint MxXiv:2544.12222 2025a.
Maxime Robeyns, Martin Szummer, and Laurence Aitchison. A self-improving coding agent, 2025b. URL http://aaxxvvv..//bbaaaaa 2504. 15228
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Kai Ruan, Mowen Huang, Ji-Rong Wen, and Hao Sun. Benchmarking 1lms' swarm intelligence. arXiv preprint ar Xiv:2200..00 04364, 2025.
65

---

Published in Transactions on Machine Learning Research (01/2026)
Rana Salama, Jason Cai, Michelle Yuan, Anna Currey, Monica Sunkara, Yi Zhang, and Yassine Benajiba. Meminsight: Autonomous memory augmentation for Ilm agents. arXiv preprint Xrx:::5503.2 21760, 2025.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancodda, and Thomas Scialom. Toolformer: Language models can teach themselves use tools. In NewrIPS, 2023.
Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schncider, and Andrea Zanctte. Can large reasoning models self-train?, 2025. URL
Lianlei Shan, Shixian Luo, Zezhou Zhu, Yu Yuan, and Yong Wu. Cognitive memory in large language models. arXiv preprint Xix:22500 02441, 2025.
Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. Agentsquare: Automatic LLM agent search in modular design space. In ICLR. OpenRevicw.nes nett, 2025.
Shuai Shao, Qihan Ren, Chen Qian, Boyi Wei, Dadi Guo, Jingyi Yang, Xinhao Song, Linfeng Zhang, Weinan Zhang, Dongrui Liu, and Jing Shao. Your agent may misevolve: Emergent risks in self-evolving Ilm agents. arXiv preprinl ar Xrx:22009..200 26354, 2025.
Yijia Shao, Tianshi Li, Weiyan Shi, Yanchen Liu, and Diyi Yang. Privacylens: Evaluating privacy norm awareness of language models action. Advances in Newral Information Processing Systems, 37:89373-
89407, 2024a.
Zhihong Shao, Poiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint Xrx::22402. 00000000,, 2024b.
Ming Shen. Rethinking data selection for supervised fine-tuning, arXiv preprint arXiv:2402.0609, 2024.
Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna L'brahimi, and Hao Wang. Continual learning of large language models: A comprehensive survey, ACM Computing Surveys, 2024.
Quan Shi, Carlos E Jimenez, Shunyu Yao, Nick Haber, Diyi Yang, and Karthik Narasimhan. When models know more than they can explain: Quantifying knowledge transfer in human-ai collaboration. anXiv preprint ar Xiv:25506. 05579, 2025a.
Yucheng Shi, Wenhao Yu, Zaitang Li, Yonglin Wang, Hongming Zhang, Ninghao Liu, Haitao Mi, and Dong Yu. Mobilegui-rl: Advancing mobile gui agent through reinforcement learning in online environment, 2025b. URL http://arii...o//aaa///////
Zhengliang Shi, Yuhan Wang, Lingyong Yan, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, and Zhaochun Ren. Retrieval models aren't tool-savvy: Benchmarking tool retrieval for large language models. CoRR, abs/2503.01760 017663 2025c.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Pro- cessing Systems, 2023. URL
11366
David Silver, Thomas Hubert, Julian Schrittwieser, Joannis Antonoglou, Malthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXie preprint orXiv: 2017.
Toby Simonds and Akira Yoshiyama. Ladder: Self-improving 1mms through recursive problem decomposition. arXiv preprinl arXiv:2503.0077 2025.
Toby Simonds, Kevin Lopez, Akira Yoshiyama, and Dominique Garmier. Self rewarding self improving, 2025. URL btsttp://axxiv.orr//bb//////////
66

---

Published in Transactions on Machine Learning Research (01/2026)
Karl Sims. Evolving virtual creatures.
In Proceedings of the 2ist Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '94, pp. 15-22, New York, NY, USA, 1994. As- ISBN 0897916670. doi: 10.1145/12111.11211777 URL https:
sociation for Computing Machinery.
odgoi.or//0.. 11....11//99911..1111 192116
Charlic Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint (tte::200.083144, 2024.
Kefan Song, Amir Moeini, Peng Wang, Lei Gong, Rohan Chandra, Yanjun Qi, and Shangtong Zhang, Reward is enough: Lims are in-context reinforcement learners. arXiv preprint 0rx:25006. fe:2506606633,,, 2025.
Kenneth O Stanley and Risto Mikkullannnn Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99 1000999-127111111 2002.
Dimitri Staufer. What should Ilms forget? quantifying personal data in Ilm for tottt-----frgoteeee re- quests. arXiv preprint arXio:2507. 11108 2025.
Katharina Stein, Daniel Fišer, Jörg Hoffmann, and Alexander Koller. Automating the generation of prompts for llm-based action choice in pddl planning. In Proceedings of the Fourteenth International Conference on Automated Planning and Schedullngg (ICAPS 2025). AAAT Press, 2025.
Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö. Ari.. Learn-by-interacl. A data-centric framework for self-adaptive agents in realistic environments, 2025. URL https://arxiv ptg/abs/2501.10
Chuanmeng Sun, Songjun Huang, and Dario Pompili. Llm-based multi-agent reinforcement learning: Current and future directions. arXiv preprint arXtv:2405.111000 11106, 2024a.
Haochen Sun, Shuwen Zhang, Lujie Niu, Lei Ren, Hao Xu, Hao Fu, Fangkun Zhao, Caixia Yuan, and Xiaojic Wang. Collab-overcooked: Benchmarking and evaluating large language models as collaborative agents. arXiv preprint arXim:2502.0000 20073, 2025a.
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. AdaPlanner: Adaptive planning from feedback with language models. In Advances in Neural Information Processing Systems, 2023. URL
https://openreviie net/ffoummmiiiie iidkunigkddddnnkeeeeee
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXie preprint arXiv:2) 12. 19723, 2024b.
Qiushi Sun, Mukai Li, Zhoumianze Liu, Zhihui Xie, Fangzhi Xu, Zhangyue Yin, Kanzhi Cheng, Zehao Li, Zichen Ding, Qi Liu, al. Os-sentinel: Towards sabeny-e mobile gui agents via hybrid validation in realistic workflows. arXio preprint Xiv:22000 2an:2510.2444 214444,,,,,, 2025b.
Zeyi Sun, Ziyu Liu, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Tong Wu, Dahua Lin, and Jiaqi Wang. Seagent: Self-evolving computer use agent with autonomous learning from experience. arXiv preprint arXiv:2508.00 04 700, 2025c.
Heng Tang, Feng Liu, Xinbo Chen, Jiawei Chen, Bohao Wang, Changwang Zhang, Jun Wang, Yuegang Sun, Bingde Hu, and Can Wang. Bridging the gap: Self-optimized fine-tuning for llm-based recommender systems. Xiv preprint 3rXiv:2505.2077 20771, 2025.
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. are preprint ar Stw:2306600301 2023.
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou, A survey on self-evolution of large language models. orXiv preprint TxXiv:240414338 2024.
67

---

Published in Transactions on Machine Learning Research (01/2026)
Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Foder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence improves sell-consistency in lms.. aixiv preprint or 2rXx::2502.06233, 2025.
Yi Tay, Shuohang Wang, Luu Anh Tuan, Jie Fu, Minh C Phan, Xingdi Yuan, Jinfeng Rao, Siu Cheung Hui, and Aston Zhang. Simple and effective curriculum Dointer-geneaate networks for reading comprehension over long narratives. arXw preprint or Arxi::1905. 10 10847, 2019.
Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. Agentic predictor: Performance prediction for agentic workflows via multi-view encoding. arXiv preprint ar vrXiv:2505.19764 2025.
Keyon Vafa, Peter G Chang, Ashesh Rambachan, and Sendhil Mullainathan. What has foundation model found? using inductive bias to probe for world models. arXiv preprint arXim:207700952, 2025.
Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Newral Information Processing Systems, 36:38975-38987, 2023.
Luanbo Wan and Weizhi Ma. Storybench: A dynamic benchmark for evaluating long-term memory with multi turns. arXiv preprint arXiv:2506. 13356, 2025.
Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, and Ying Wen. Rema: Learning to meta-think for 1mms with multi-agent otg/abs/2503.09500
reinforcement learning, 2025. URL https://arxi:: httpp////axxiv.or/ab///////
Bin Wang, Zexin Liu, Hao Yu, Ao Yang, Yenan Huang, Jing Guo, Huangsheng Cheng, Hui Li, and Huiyu Wu. Mepguard: Automatically detecting vulnerabilities in mep servers. arXiv preprint or 0rXiv:2510.230% 2025a.
Bo Wang, Weiyi Hc, Pengfei Hc, Shenglai Zeng, Zhen Xiang, Yue Xing, and Jiliang Tang. Unveiling privacy risks in Ilm agent memory. arxie preprinl arvi:22002. 13172, 2025b.
Borui Wang, Kathleen McKeown, and Rex Ying. Dystil: Dynamic strategy induction with large language models for reinforcement learning, 2025c. URL httpp:///axxiv.or/abb///// 03209
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arxio preprinl arXiv:2305.1620 2023a.
Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. Spa-rl: Reinforcing Iim agents via stepwise progress attribution. arXiv preprint arXim:2505.2073 2025d.
Haoyu Wang, Dong Fang, et al. Fela: A multi-agent agent evolutionary system for feature engineering of industrial event log data. Xiviv preprint ar arXiv:2510 10. 0.22223, 2025c.
Hongru Wang, Yujia Qin, Yankai Lin, Jeff 7. Pan, and Kam-Fai Wong. Empowering large language models: Tool learning for real-world interaction. In Proceedings of the 4 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, pp.
New York, NY, USA, URL
2024a. Association for Computing Machinery. ISBN 9798400704314. 4. doi: (4.115/3626772.366 https://doi. oww.ooog//// 10.
Hongru Wang, Rui Wang, Boyang Xue, Heming Xia, Jingtao Cao, Zeming Liu, Jeff Z. Pan, and Kam- Fai Wong, AppBench: Planning of multiple APIs from various APPs for complex user instruction. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (ods.), Proceedings of the 2024 Conference on Empirical Methods Natural Tanguage Processing, pp. 15322- Miami, Florida, USA, November mainllpmmain.866.. URL https:
2024b. Association for Computational Linguistics. doi: 10. Tacllanthology.org//ao0 www.:o///22...e1mmmmaaaaa.....aa emnlp-main. .866/
Hongru Wang, Deng Cai, Wanjun Zhong, Shijue Huang, Jeff Z. Pan, Zeming Liu, and Kam-Fai Wong. Self-reasoning language models: Unfold hidden reasoning chains with few reasoning catalyst, 2025f. URL http:://arxii..rrraa////.0....
68

---

Published in Transactions on Machine Learning Research (01/2026)
Hongru Wang. Cheng Qian, Manling Li, Jiahao Qiu, Boyang Xue, Mengdi Wang, Heng Ji, and Kam- Fai Wong. Toward a theory of agents as tool-use decision-makers, 2025g. URL. httpss///axxtii org/ abs/
00886
Hongru Wang, Cheng Qian, Jiahao Qiu, et al. OTC-PO: Optimal tool calls via reinforcement learning. arXiv preprint arx:::200 14870, 2025h.
Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, et al. A comprehensive survey in Ilm (-agent) full stack safety: Data, training and deployment. arXiv preprint orXiv:2504.1535 2025i
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pottern analysis and machine intelligence, 46(8): 3362-5388, 2024c.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce 11ms step-by-step without human annotations. arXiv preprint arXiv:2312.0880 08935, 2023b.
Pong Wang, Ruihan Tao, Qiguang Chen, Mengkang Hu, and Libo Qin. X-webagentbench: A multilingual interactive web benchmark for evaluating global agentic system. arXiv preprinl ar 2xx::2255.1554 2025j.
Renxi Wang, Xudong Han, Lei Ji, Shu Wang, Timothy Baldwin, and Haonan Li. Toolgen: Unified tool retrieval and calling via generation. In ICLR. OpenReview.net, 2025k.
Rui Wang, Ce Zhang, Jun-Yu Yu Ma, Jianshu Zhang, Hongru Wang, Yi Chen, Boyang Xue, Tianqing Fang, Zhisong Zhang, Hongming Zhang, Haitao Mi, Dong Yu, and Kam-Fai Wong. Explore to evolve: Scal- ing evolved aggregation logic via proactive online exploration for deep research agents. arXiv preprint arXiv:2510. 14438, 20251.
Saizhuo Wang, Hang Yuan, Lionel M Ni, and Jian Guo. Quantagent: Seeking holy grail in trading by self-improving large language model. arXiv preprinl ar arXiv:2402.0377 2024d.
Shaobo Wang, Zhengbo Jiao, Zifan Zhang, Yilang Peng, Xu Ze, Boyu Yang, Wei Wang, Hu Wei, and Linfeng Zhang. Socratic-zero: Bootstrapping reasoning via data-free agent co-evolution. arXie preprint arXiv:2509. 0035099.27775 24726, 2025m.
Siyuan Wang, Zhuohan Long, Zhihao Fan, Zhongyu Wei, and Xuanjing Huang. Benchmark evolving A multi-agent framework for dynamic Ilm evaluation. arXiv preprint arXiv:2402.11444 11443, 2024e.
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge editing for large language models: survey, ACM Computing Surveys, 57(3):1 37, 2024f.
Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Distrl: An asynchronous dis- tributed reinforcement learning framework for on-device control agents. arXiv preprint arXiv:2410.144 14803, 2024g.
Tevin Wang and Chenyan Xiong. Autorule: Reasoning prove preference learning. arXie preprint arXiv:2506.1565. 2025.
extracted rule-based rewards im-
Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, et al. Trace: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXim:2310006702 2023c.
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. 1EEE transactions on pattern analysis and machine intelligence, 455:455554476 2021.
Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P Xing, and Zhiting Hu. Promptagent: Strategic planning with language models enables expert-level prompt optimization. arXiv preprint Xin:2331 16427, 2023d.
69

---

Published in Transactions on Machine Learning Research (01/2026)
Yinjie Wang, Ling Yang, Guohao Li, Mengdi Wang, and Bryon Aragam. Scoreflow: Mastering LLM agent workflows via score-based preference optimization. ar preprint 2Xix:2502.04300 2025n.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint 211 10560, 2022.
Yue Wang, Tianfan Fu, Yinlong Xu, Zihan Ma, Hongxia Xu, Bang Du, Yingzhou Lu, Honghao Gao, Jian Wu, and Jintai Chen. Twin-gpt: digital twins for clinical trials via large language model. ACM Transactions on Multimedia Computing, Communicaatons and Applications, 2024h.
Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. Mobile-agent-e: Self-evolving mobile assistant for complex tasks. arXiv preprint WxXi:2501.11733, 2025o.
Zhenting Wang, Guofeng Cui, Kun Wan, and Wentian Zhao. Dump: Automated distribution-level curriculum learning for rl-based Ilm post-training. arXiv preprint arXiv:2004.000 09710, 2025p.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang. Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning. arXiv preprinl or Xiv:22504 20073, 2025q.
Ziqi Wang, Le Hou, Tianjian Lu, Yuexin Wu, Yunxuan Li, Hongkun Yu, and Heng Ji. Enabling lanuguage models to implicitly learn self-improvement. In The Twelfth International Conference on Learning Repre- senlations, 2024i. URL https://openreview..r wttp://www...etooommm/eee//eeiee
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprinl MXXi::2409.072200 2024j.
Boyi Wei, Benedikt Strocbl, Jiacon Xu, Joie Zhang, Zhou Li, and Peter Honderson. Dynamic risk assessments for offensive cy cybersecurity agents. Xiv preprinl arXiv:2505.1898 2025a.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaeso. Browsocomp: A simple yet challenging bench- mark for browsing agents. arXiv preprint ar KKt:2254.122114 2025b.
Lai Wei, Yuting Li, Chen Wang, Yue Wang, Linghe Kong, Weiran Huang, and Lichao Sun. Unsupervised post-training for multi-modal 1lm reasoning via grpo, 2025c. URL https://arxiv.. orwiii...rr/aaaa abgg/bbs/5055...
Qianshan Wei, Tengchao Yang, Yaochen Wang, Xinfeng Li, Lijun Li, Zhenfei Yin, Yi Zhan, Thorsten Holz, Zhiqiang Lin, and XiaoFeng Wang. A-memguard: A proactive defense framework for llmebased agent memory. Xiv preprint ar arXiv:2510.02371 2025d.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Zii Neel Jain, Khalid Saifullah, Sreemanti Dey, et al. Livebench: A challenging, contamination-limita. I1m benchmark. Xrxiv preprinl arXiv:2406.190 19314, 2024.
Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. Advances in Neural Information Processing Systems, 36:36637-36651. 2023.
Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, et al. Webwalker: Benchmarking lims in web traversal. orXiv preprint w:Xiv:2501.07572 2025.
Mengsong WU, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang, and Wenliang Chen. Scal-tools: Self- instruct tool learning dataset for agent tuning and detailed benchmark. In CCF International Conference on Natural Language Processing and Chinese Computing, pp. 372 -84.. Springer, 2024.
70

---

Published in Transactions on Machine Learning Research (01/2026)
Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, et al. Enhancing 11m reasoning via critique models with test-time and training- time supervision. arXiv preprint arX1v:2411.1657, 2024.
Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, and Lingming Zhang. Live-swe-agent: Can software engincering agents self-evolve on the fly? arXiv preprint XrXi:22511. 13666, 2025a.
Peng Xia, Kaide Zeng, Jiaqi Liu, Can Qin, Fang Wu, Yiyang Zhou, Caiming Xiong, and Huaxiu Yao. AgentO: Unleashing self-evolving agents from zero data via tool-integrated reasoning. ar preprint arXiv:2511.16443 2025b.
Jinyu Xiang, Jiayi Zhang, Zhaoyang Yu, Fengwei Teng, Jinhao Tu, Xinbing Liang, Sirui Hong, Chenglin Wu, and Yuyu Luo. Self-supervised prompt optimization. orXiv preprint wxXi::250.06655, 2025.
Han Xiao, Guozhi Wang, Yuxiang Chai, Zimu Lu, Weifeng Lin, Hao He, Lue Fan, Liuyang Bian, Rui Hu, Liang Liu, et al. Ui-genie: A sell-improving approach for iteratively boosting millmbbseed mobile gui agents. arXiv preprint ar Kiv:2255.21496 2025a.
Yijia Xiao, Edward Sun, Di Luo, and Wei Wang. Tradingagents: Multi-agents Ilm financial trading frame- work. arxiv preprint 2rXiv:2412.000
2024.
Yunzhong Xiao, Yangmin Li, Hewei Wang, Yunlong Tang, and Zora Zhiruo Wang. Toolmem: Enhancing multimodal agents with learnable tool capability memory. aiviv preprint 1xXi::251000660, 2025b.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongehan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040- 52094, 2024.
Wenpeng Xing, Zhoughao Qi, Yupeng Qin, Yilin Li, Caini Chang, Jiahui Yu, Changting Lin, Zhenzhen Xie, and Meng Han. Mcp-guard: A defense framework for model context protocol integrity in large language model applications. arXiv preprint arXiv:2508. 10991, 2025.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Represenlations, 2024a.
Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking Ilm agents on consequential real world tasks. arXiv preprint arXin:2412. 161, 2024b.
Wujiang Xu, Kai Mei, Hang Gao, Juntao Tan, Zujie Liang, and Yongleng Zhang. A-mem: Agentic memory for Ilm agents. arXiv preprint arXiv:2502.111 12110, 2025a.
Zicheng Xu, Guanchu Wang, Guangyao Zheng, Yu-Neng Chuang, Alexander Szalay, Xia Hu, and Vladimir Braverman. Self-ensemble: Mitigating confidence distortion for large language models, 2025b. URL
attps://arxtv.orr/ab////66 21950
Cilin Yan, Jingyun Wang, Lin Zhang, Ruihui Zhao, Xiaopu Wu, Kai Xiong, Qingsong Liu, Guoliang Kang, and Yangyang Kang. Efficient and accurate prompt optimization: the benefit of memory in exemplar- guided reflection. arXiv preprint arXiv:2411.07446, 2024.
Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nic, Zifeng Ding, Zonggen Li, Xiaowen Ma, Kristian Ker- sting, Jeff 7. Pan, Hinrich Schütze, Volker Tresp, and Yunpu Ma. Memory-rl: Fahancing large language model agents manage and utilize memories via reinforcement learning, arxie preprint arXiv:2508. 19828, 2025.
Cheng Yang, Xuemeng Yang, Licheng Wen, Daocheng Fu, Jianbiao Mei, Rong Wu, Pinlong Cai, Yufan Shen, Nianchen Deng, Botian Shi, Yu Qiao, and Haifeng Li. Learning on the job: An experience-driven self-evolving agent for long-horizon tasks. arxi preprint arXiv:2510.08000 2025a.
71

---

Published in Transactions on Machine Learning Research (01/2026)
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXim:2309.0844 2023.
John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaove, Karthik R Narasimhan, ct al. Swe-bench multimodal: Do ai systems 2024.
generalize to visual software domains? arXiv preprint
Kaiqi Yang, Hang Li, Yucheng Chu, Ahreum Han, Yasemin Copur-Genctin Jiliang Tang, and Hui Liu. A llm-driven multi-agent systems for professional development. of mathematics teachers. arXio preprint arXiv:2507.0522 2025b. URL https://arxiv. org/ arg/absl. aba/ 05292
Yue Yang, MingKang Chen, Qihua Liu, Mengkang Hu, Qiguang Chen, Gengrui Zhang, Shuyue Hu, Guangtao Zhai, Yu Qiao, Yu Wang, et al. Truly assessing fluid intelligence of large language models through dynamic reasoning evaluation. arXiv preprint trvi22206. 02648, 2025c.
Yutao Yang, Jie Zhou, Xuanwen Ding, Tianyu Huai, Shunyu Liu, Qin Chen, Yuan Xie, and Liang He. Recent advances of foundation language models- basod contimual learning: A survey, ACM Computing Surveys, 57(5):1-33 2025d.
Ziyi Yang, Weizhou Shen, Chenliang Li, Ruijun Chen, Fanqi Wan, Ming Yan, Xiaojun Quan, and Fei Huang. Spell: Self-play reinforcement learning for evolving long-context language models. arXie preprint arXiv:2509.2385 2025e.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances Neural Information Processing Systems, 35: 20744-20757, 2022.
Shunyu Yao, Noah Shimn, Pedram Razavi, and Karthik Narasimhan. tan-bench: A benchmark for tool- agent-user interaction in real-world domains. arXiv preprint arXiv:2406. 100006...0444 12045, 2024.
Rui Ye, Xiangru Liu, Qingyun Wu, Xinyuan Pang, Zhaopeng Yin, Lu Bai, et al. X-mas: Towards building multi-agent systems with heterogeneous llms. arXiv preprint arXiv:2505. 16997, 2025a.
Rui Ye et al. Mas-gpt: Training 1lms to build Ilm-based multi-agent systems. arXiv preprint, 2025b.
S. Yellamraju, et al. TextGrad: Automatic "differentiation' via text. arXie preprint vxXiv:2406.07990 2024.
Li Yin and Zhangyang Wang. Llm-aulodiff: Auto-differentiale any Ilm workflow. ai e-prints, pp. arXiv 2501, 2025.
Xunjian Yin, Xinyi Wang, Liangming Pan, Li Lin, Xiaojun Wan, and William Yang Wang. Gödel agent: A self-referential agent framework for recursive self-improvement, 2025. URL https::// hoapsi///axiv.... org/ aba/ 2410. 04444
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. anXiv preprint ar Xiv:2550.. 144 76, 2025a.
Wenhao Yu, Zhenwen Liang, Chengsong Huang, Kishan Panaganti, Tianqing Fang, Haitao Mi, and Dong Yu. Guided self-evolving llss with minimal human supervision. arXiv preprinl arxi:22512.02/77 2025b.
Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi Fung, Hao Peng, and Heng Ji. CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets. In 12th International Conference on Learning Representations, 2024a. URL
Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang. Evoagent: Towards automatic multi-agent generation via evolutionary algorithms. arXiv preprint arXiv:2406. 14228, 2024b.
72

---

Published in Transactions on Machine Learning Research (01/2026)
Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang. Evoagent: Towards
automatic multi-agent generation via evolutionary algorithms, 2025a. URL https: bttps://arrxvv.rrs org/ abs/ #406.14223
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Sell-rewarding language models. ar preprint or urXie:2401.1 1000000 2024c.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason F. Weston. Self-rewarding language models. Tn Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 57905 57923. PMLR, 2024d.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Selfrrearrding language models, 2025b. URL httpp:://arxt..orrg//////// www.o//////...///////// 10020
Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng- Tao Tao Jiang, and Bo Li. Enhancing visual grounding for gui agents via self-cvolutionary reinforcement learning. arviv preprinl or Xiv:2205. 12370, 2025c.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang, Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint or arXix:2304.00 05302, 2023.
Murong Yue, Wenhan Lyu, Wijdane Mifdal, Jennifer Suh, Yixuan Zhang, and Ziyu Yao. Mathve: An Ilm- simulated multi-character virtual classroom for mathematics education. arXiv preprint 0rX.v:2444.06711, 2025. URL
abs 22tp///44440777 06711
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Aduances in Newral Information Processing Systems, 35:15476-15488, 2022.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking, arXiv preprinl arXiv:2433.0000 2024.
Yumpeng Zhai, Shuchang Tao, Cheng Chen, Anni Zou, Ziqian Chen, Qingxu Fu, Shinji Mai, Li Yu, Jiaji Deng, Zouying Cao, Zhaoyang Liu, Bolin Ding, and Jingren Zhou. Agentevolver: Towards efficient self-evolving agent system. arXio preprint arXiv:2511.1039, 2025.
Danyang Zhang, Lu Chen, Situo Zhang, Hongshen Xu, Zihan Zhao, and Kai Yu. Large language models are semi-parametric reinforcement learning agents. Advances in Neural Information Processing Systems, 36, 2024a.
Enci Zhang, Xingang Yan, Wei Lin, Tianxiang Zhang, and Qianchun Lu. Learning like humans: Advancing Ilm reasoning capabilities via adaptive difficulty curriculum learning and expert-guided self-reformulation. arXiv preprint Skfw:2505.08366 2025a.
Genghan Zhang, Weixin Liang, Olivia Hsu, and Kunle Olukotun. Adaptive self-improvement 11m agentic system for ml library development. arXiv preprint SEK3:2200002594, 2025b.
Guanghui Zhang, Kang Chen, Guohao Wan, Hao Chang, Hao Cheng, al. Evoflow: Evolving diverse agentic workflows the fly. arXie preprint orXiv:2502.073: 07377, 2025c.
Guanghui Zhang, Li Niu, Jianwei Fang, Kun Wang, Lu Bai, et al. Multi-agent architecture search via agentic supernet. arXiv preprint arXiv:2502.04118 2025d.
Guibin Zhang, Muxin Fu, Guancheng Wan, Miao Yu, Kun Wang, and Shuicheng Yan. G-memory: Tracing hierarchical memory for multi-agent systems. arXiv preprint Xrx: 07398, 2025e.
Guibin Zhang, Muxin Fu, and Shuicheng Yan. Mengen: Weaving generative latent memory for self-evolving agents. Xiu preprint ar Sto:2509.47000 2025f.
73

---

Published in Transactions on Machine Learning Research (01/2026)
Guibin Zhang, Haotian Ren, Chong Zhan, Zhenhong Zhou, Junhao Wang, He Zhu, Wangchunshu Zhou, and Shuicheng Yan. Memevolve: Meta-evolution of agent memory systems. Xiv preprint or arXiv:2512.188 18746, 2025g.
Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang. Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents. arXic preprinl ar 20 00.02644, 2024b.
Jenny Zhang. Shengran Hu, Cong Lu, Robert Lange, and Jeff Chune. Darwin Godel Machine: Open-Ended Evolution of Sell-Improving Agents. arXiv preprinl ar fti::555522954, 2025h.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. arXie preprint arXiv:24100 10762 2024c.
Jiayi Zhang, Yiran Peng, Fanqi Kong, Cheng Yang, Yifan Wu, Zhaoyang Yu, Jinyu Xiang, Jianhao Ruan, Jinlin Wang, Maojia Song, HongZhang Liu, Xiangru Tang, Bang Liu, Chenglin Wu, and Yuyu Luo. Autoenv: Automated environments for measuring cross-enviromment agent learning. arXiv preprint arXiv:2511.19304, 20251.
Kongcheng Zhang, Qi Yao, Shunyu Liu, Yingjie Wang, Baisheng Lai, Jieping Ye, Mingli Song, and Dacheng Tao. Consistent paths lead to truth: Self-rewarding reinforcement learning for Il reasoning, 2025j. URL
fttps://arxiv.org/abbss. wttporggg/bbbaeeeeeeeeeeeeeee 0
Mike Zhang, Amalic Pernille Dilling, Léon Gondelman, Niels Erik Ruan Lyngdorf, Euan D. Lindsay, and Johannes Bjerva. Sell: Harnessing large language model agents to improve educational feedback systems. arXiv preprint Xr.i::2502. 12927, 2025k. URL https: org/ hts//bbs/25022 12927
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. ar preprint ar arXiv:2401.0122 011 286, 2024d.
Peiyan Zhang, Haibo Jin, Leyang Hu, Xinnuo Li, Liying Kang, Man Luo, Yangqiu Song, and Haohan Wang. Revolve: Optimizing ai systems by tracking response evolution in textual optimization. arXiv preprint 12. 1.Ni::212.000000
2024e.
Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, et al. A survey on test-time scaling large language models: What, how, where, and how well? arXiv preprint arXio:2503. 24235, 20251.
Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, et al. Agentic context engineering: Evolving contexts for self-improving language models. ar preprint arxin:25 2ii:2510.004 10.04618, 2025m.
Shaokun Zhang, Jieyu Zhang, Jiale Liu, Linxin Song, Chi Wang, Ranjay Krishna, and Qingyun Wu. Of- fline training of language model agents with functions as learnable weights. In Forty-first International Conference on Machine Learning, 2024f.
Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. vernolron-ressearcc--oooooo Exploring lool-using language models with reinforced reasoning. arXiv preprint arXiv:2505. 200002 2025n.
Xuemiao Zhang, Liangyu Xu, Feiyu Duan, Yongwei Zhou, Sirui Wang, Rongxiang Weng, Jingang Wang, and Xunliang Cai. Preference curriculum: Llms should always be pretrained on their preferred data. arXiv preprint ar Xrc::5001 13126, 2025o.
Xueqiao Zhang, Chao Zhang, Jianwen Sun, Jun Xiao, Yi Yang, and Yawei Luo. Eduplanner: Llm-basod multi-agent systems for customized and intelligent instructional design. arXiv preprint 05x5:22504.05370, 2025p. URL htttss///arxiii wttpe/ivvvrgab/////eeeeeoo atg/abs/2504 05370
74

---

Published in Transactions on Machine Learning Research (01/2026)
Yijing Zhang, Dyah Adila, Changho Shin, and Frederic Sala. Personalize your Ilm: Fake it then align it. arXiv preprint or N5c:2503.00043 2025q.
Yiqun Zhang, Peng Ye, Xiaocui Yang, Shi Feng, Shufei Zhang, Lei Bai, Wanli Ouyang, and Shuyue Hu. insure-ds population- based evolution of large language models. arXiv preprint arXiv:2503.01 01155005.11 2025r.
Yuanshuo Zhang, Yuchen Hou, Bohan Tang, Shuo Chen, Muhan Zhang, Xiaowen Dong, and Siheng Chen. Gnns as predictors of agentic workflow performances. arXic preprint arXiv:2503. 11301, 2025s.
Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji- Rong Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint WxXi::244.115500 2024g.
Zhehao Zhang, Ryan A Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, al. Personalization of large language models: A survey, arXiv preprint xrXiv:2411.0002, 2024h.
Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlic Huang, Agent-saletybenchh Evaluating the safety of Il agents. arXic preprint arXi::2412 12.2212..12 10004/2..44444 144 2024i.
Zhexin Zhang, Junxiao Yang, Pei Ke, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlie Huang. Safe unlearning: A surprisingly effective and generalizable solution to defend against, jailbreak attacks. arXiv preprint vrXiv:2407.08855 2024j.
Zhongyue Zhang, Zijie Qiu, Yingcheng Wu, Shuya Li, Dingyan Wang, Zhuomin Zhou, Duo An, Yuhan Chen, Yu Yongbo Wang, et al. Origene: A sell-evolving virtual disease biologist automating therapeutic larget discovery. bioRriv, pp. 2025-0220000 06, 2025t.
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Lim agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19632-19642, 2024a.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint Xix:2550 08335, 2025a.
Haitong Zhao, Chang Ma, Guoyin Wang, Jing Su, Lingpeng Kong, Jingjing Xu, Zhi-Hong Deng, and Hongxia Yang. Empowering large language model agents through action learning. CoRR, als//2402.15809, 2024b.
Henry Hengyuan Zhao, Pan Zhou, and Mike Zheng Shou. Genixer: Empowering multimodal large language model as powerful data generator. In European Conference on Computer Vision, pp. 129-147. Springer, 2024c.
Qianchuan Zhao, Yuxiang Xie, Wentao Wang, Jie Li, Lizhou Sha, Jialong Zhang, and Minlie Huang, Riche- lieu: Self-evolving LLM-based agents for AT diplomacy. arXio preprint arxiv:2407.066 2024d. URL
06813
Wanjia Zhao, Mert Yuksekgomul, Shirley Wu, and James Zou. Sirius: Self-improving multi-agent systems via bootstrapped reasoning. arXie preprint arXiv:2502.00 04 780, 2025b.
Arman Zharmagambetor, Chuan Guo, Ivan Evlimov, Maya Pavlova, Ruslan Salakhutdinov, and Kama- lika Chaudhuri. Agentdam: Privacy leakage evaluation for autonomous web agents. arXiv preprint SrXiv:2503.09750 2025.
Boyuan Zheng, Michael Y. Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yucqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, and Yu Su. Skillweaver: Web agents can self-improve by discovering and honing skills. CoRR, abs/ abs/2704.07779, 2025a.
75

---

Published in Transactions on Machine Learning Research (01/2026)
Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoe V Te, Fd H Chi, et al. Natural plan: Benchmarking llms on natural language planning, arXiv preprint Xrxi::2406. X06:2406.045200 2024a.
Junhao Zheng, Xidi Cai, Qiuke Li, Duzhen Zhang, ZhongZhi Li, Yingying Zhang, Le Song, and Qianli Ma. Lifelongagentbench: Evaluating Ilm agents lifelong learners. arXiv preprint arXiv:2505.1194. 2025b.
Junhao Zheng, Chengming Shi, Xidi Cai, Qiuke Li, Duzhen Zhang, Chenxing Li, Dong Yu, and Qianli Ma. Lifelong learning of large language model based agents: A roadmap. arXiv preprint orXiv:2501.07270 2025c.
Yuanhang Zheng, Peng Li, Wei Liu, Yang Liu, Jian Luan, and Bin Wang. Toolrerank: Adaptive and hierarchy-aware reranking for tool retrieval. In LREC/COLING, pp. 16263 16273. ELRA and ICCL, 2024b.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 19731, 2024.
Da-Wei Zhou, Hai-Tong Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre- trained models: A survey, XrXiv preprint ar Xr::22401.11338 2024a.
Han Zhou, Xingchen Wan, Ruoxi Sun, Hamid Palangi, Shariq Iqbal, Ivan Vulié, Anna Korhonen, and Sercan Ö. Arı. Multi-agent design: Optimizing agents with better prompts and topologies, 2025a. URL
petts://arxivy http://aaxiv..ogaaa/////a//
wwwgg/aa/////oooo 25022/2533322223333 02533
Huichi Zhou, Yihang Chen, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang, et al. Memento: Fine-tuning 1lm agents without fine-tuning 1mss. orxiv preprint arXiv:2508. 161125508..613, 2025b.
Jijie Zhou, Eryue Xu, Yaoyao Wu, and Tianshi Li. Rescriber: Smaller-ll-poweeee user-led data minimiza- tion for Ilm-based chatbots. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp. 1-28, 2025c.
Shuyan Zhou, Frank F Xu, Hao Zhn, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:230711885, 2023.
Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, et al. Symbolic learning enables self-evolving agents. arXiv preprint orXiv:2406.18855 18532, 2024b.
Xueyang Zhou, Weidong Wang, Lin Lu, Jiawen Shi, Guiyao Tie, Yongtian Xu, Lixing Chen, Pan Zhou, Neil Zhenqiang Gong, and Lichao Sun. Automating safety enhancement for Ilm- agents with synthetic risk scenarios. arXiv preprint Xrxv:2500 1 1775, 2025d.
Yifei Zhou, Sergey Levine, Jason Weston, Xian Li, and Sainbayar Sukhbaatar. Self-challenging language model agents, 2025e. URL https://arxii... orw. wrtp://///////////////oo
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-leevl prompt engineers. Tn The Eleventh International Conference on Learning Representations, 2022.
Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, et al. Multiagentbench: Evaluating the collaboration and competition of Ilm agents. arXiv preprint arxiv:200..0190 2025.
Yangyang Zhuang, Wenjia Jiang, Jayy Zhang, Ze Yang, Joey Tianyi Zhou, and Chi Zhang. Learning to be doctor: Searching for effective medical agent architectures. arxio preprint arx:2:2004. 11301, 2025.
76

---

Published in Transactions on Machine Learning Research (01/2026)
Mingehen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents as optimizable graphs. In TCML. OpenReview.net, 2024.
Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Zico Kolter, Matt. Fredríkson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. In The Tiirty-eighth Annual Conference Neural Information Processing Systems, 2024.
Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.100 2025.
Adam Zweiger, Jyothish Pari, Han Guo, Ekin Akyürek, Yoon Kim, and Pulkit Agrawal. Self-adapting language models. arxiv preprinl Xix: 10943, 2025.
77


Reviews:
[{"review_id": "y49Er5hfr2", "reviewer": "Reviewer_orgC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey introduces self-evolving agents that enable LLMs to adapt continuously. It organizes the field by evolving components, adaptation stages, and learning mechanisms, and reviews evaluation, applications, and key challenges. It provides a roadmap toward more adaptive, intelligent, and autonomous AI systems.\n\nThe principal strength of this survey is its structured \"what, when, how\" framework, which brings much-needed organization to the rapidly growing domain of self-evolving agents and serves as a useful entry point for newcomers.", "claims_and_evidence_yn": "No", "claims_explanation": "1. Limited Conceptual Novelty and Contribution\n- The core concept of \"self-evolving agents\" is not sufficiently distinguished from well-established paradigms like Continual/Lifelong Learning, Self-Improving LLMs, and general Adaptive Agents. The distinctions drawn in Section 2.2 are weak and not formally defined.\n- The claim that lifelong learning only updates model parameters is inaccurate; the design and management of memory and other non-parametric components are long-standing and central challenges in continual learning.\n- The dichotomy between \"passive\" (lifelong learning) and \"active\" (self-evolving) knowledge acquisition is arbitrary and lacks a rigorous definition. Many methods cited as \"self-evolving\" (e.g., using SFT or RL on pre-collected data) fit a \"passive\" paradigm.\n- This lack of a clear, novel conceptual boundary makes the survey feel like a rebranding of general agent adaptation and training techniques, significantly limiting its original contribution. The overlap with surveys like \"A Survey on Self-Evolution of Large Language Models\" [1] and \"A survey on the memory mechanism of large language model-based agents\" [2] is substantial.\n\n2. Overly Broad and Unfocused Scope\n- The survey attempts to encompass nearly every aspect of LLM-based agents—including model parameters, memory, tools, architecture, and all major learning paradigms (SFT, RL, ICL)—under the \"self-evolving\" umbrella.\n- This results in a lack of focus, making it difficult to discern what truly constitutes self-evolution versus standard agent capabilities or training procedures. By including everything, the survey fails to carve out a unique and meaningful niche.\n- The \"What to Evolve\" section (Section 3) reads like a general component-wise survey of LLM agents, much of which is covered in [2], rather than a focused analysis of how these components autonomously and continuously improve.\n\n3. Lack of Rigor in the Proposed Taxonomy\n- While the \"What, When, How\" framework is a useful organizational tool, it is not critically justified. The dimensions often map directly onto existing, well-understood concepts (e.g., \"When to Evolve\" is essentially online vs. offline learning; \"How to Evolve\" includes standard RL and imitation learning).\n- The taxonomy risks being a superficial repackaging of existing ideas rather than a novel analytical framework that provides new insights into the field of self-evolution.\n\n4. Insufficient Critical Analysis and Depth\n- The survey is primarily descriptive, cataloging methods without providing a critical analysis of their limitations, scalability, computational costs, or real-world applicability.\n- Many cited methods are demonstrated in narrow, simulated environments (e.g., Minecraft, coding benchmarks). The survey does not adequately address the significant challenges of transferring these approaches to complex, open-ended, real-world domains.\n- There is a lack of discussion on the maturity and readiness of these techniques for deployment, including their robustness to failure and the assumptions they make about the environment.\n\n5. Inadequate Treatment of Safety and Alignment\n   While Section 8.3 briefly mentions safety, it fails to grapple with the unique and amplified risks of self-evolving systems. These include:\n- Uncontrolled Behavior Drift: How to ensure an agent's goals and values remain aligned with human intentions as it evolves autonomously.\n- Reward Hacking in Open-Ended Evolution: The risk of agents finding and exploiting loopholes in their self-defined reward signals or internal feedback mechanisms.\n- Safety of Self-Created Tools: The potential for agents to autonomously generate and use tools in unsafe or malicious ways.\n- This is a critical oversight given the paper's emphasis on autonomy and its connection to the ambitious goal of Artificial Superintelligence (ASI).\n\n6. Structural and Organizational Deficiencies\n- The paper's structure is encyclopedic and repetitive rather than narrative-driven. It often lists methods without building a compelling story about the field's progression or the trade-offs among different evolutionary approaches.\n- The inclusion of very standard techniques (e.g., basic SFT or RL) dilutes the core message and creates redundancy with more general agent surveys.\n- Some figures (e.g., Figure 3) are overly complex and not explained clearly in the text, reducing their effectiveness.\n\n7. Superficial Evaluation Discussion\n- Section 7 on evaluation correctly identifies the challenge but does not delve deeply into how to benchmark true, long-term, open-ended self-evolution. The metrics and benchmarks discussed are essentially adaptations of those used for static or short-horizon adaptive agents.\n- There is a lack of discussion on evaluating the autonomy of the evolutionary process itself—for instance, measuring the degree to which improvement is self-directed versus externally guided.\n\n\nOther Minor Issues:\n\n   (1) The typo in the Introduction: `prompting strategies (?)fernando2023promptbreeder,` should be fixed.\n\n   (2) The term \"self-evolving\" is used inconsistently throughout the manuscript, sometimes referring to parameter updates, sometimes to prompt optimization, and sometimes to architectural changes, further contributing to the conceptual fuzziness.\n\n\n[1] A Survey on Self-Evolution of Large Language Models\n\n[2] A survey on the memory mechanism of large language model-based agents", "audience_yn": "Yes", "audience_explanation": "The paper will interest some readers.", "requested_changes": "Please refer to the comments above.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dmU8LqkcoZ", "reviewer": "Reviewer_NCWP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey focuses on Self-Evolving Agents, which I interpret as being closely related to self-improvement and post-training techniques aimed at building more capable agentic systems using foundational models and LLMs. The paper explores several mechanisms for enabling the evolution process and also discusses long-term goals for developing fully autonomous AI agents.\n\nStengths\n1. The survey is well-written and thoughtfully structured, covering key topics relevant to agent post-training and self-improvement.\n2. Section 6 is particularly engaging, offering a compelling overview of potential application areas where self-evolving agents could have a significant impact.\n\nWeaknesses\n1. I’m concerned about the framing of the title and abstract. While the paper discusses self-evolving agents, many of the techniques—such as heuristics, reward functions, and training strategies—are still human-designed. The claim that this work \"paves the way toward Artificial Superintelligence (ASI)\" feels overly ambitious and should be more cautiously worded.\n2. The discussion of evolutionary approaches is quite limited. Although AlphaEvolve is briefly mentioned, evolutionary and genetic algorithms have a long history of improving agent behavior and should be covered more thoroughly, especially given the paper’s focus on \"evolving\" agents. Including this would broaden the scope and enrich the survey beyond learning-based methods.\n3. A crucial omission is the topic of self-play, which by definition involves agents improving through interactions with versions of themselves or other agents. This is a core form of self-improvement and should be included in the discussion, along with how it differs from other techniques.\n4. The description of imitation learning could be refined. Since imitation learning involves learning from expert demonstrations, this distinction should be clearly articulated to avoid conflating it with other forms of self-improvement.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I believe the paper offers a solid survey of methods to improve agents and highlights promising areas of application. The supporting evidence is clear and compelling.", "audience_yn": "Yes", "audience_explanation": "The paper will be useful to a diverse audience, including those in reinforcement learning, NLP, and AI for science. I believe it would be a valuable addition to TMLR.", "requested_changes": "I encourage the authors to address the weaknesses outlined above. Once these concerns are resolved, I believe the paper will be a strong candidate for acceptance.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8o6LkNzG2o", "reviewer": "Reviewer_nmCF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Contributions - \nThe paper proposes a structured framework for 'self-evolving agents' centered on three orthogonal questions: what to evolve (model, context/memory/prompts, tools, and agentic architecture), when to evolve (intra-test-time vs. inter-test-time via ICL, SFT, RL), and how to evolve (reward-/feedback-driven, imitation/demonstration, and population-based evolution), adding a complementary where dimension for domains and settings. It differentiates self-evolving agents from adjacent paradigms (curriculum learning, lifelong learning, model editing/unlearning) by emphasizing active exploration, non-parametric evolution (memory, tools, workflows), and test-time adaptation, providing formal definitions and an agent system formulation.\nIt catalogs mechanisms and exemplars across the four 'what' pillars (e.g., self-rewarding/self-challenging policy updates, memory evolution such as Mem0/Expel, prompt optimization, and tool discovery/mastery/management) and across single- and multi-agent architectural optimization. It surveys evaluation goals and paradigms tailored to self-evolution (adaptivity, retention, generalization, efficiency, safety) and argues for moving beyond static benchmarks toward short-horizon adaptive and long-horizon lifelong assessments, listing representative benchmarks and metrics. It synthesizes applications (coding, GUI/computer-use, healthcare, finance, education) and outlines open challenges and future directions (personalization, scalable/generalizable architectures, safety/governance, multi-agent ecosystems, dynamic evaluation co-evolution).\n\nKey strengths -\n* Clear, comprehensive taxonomy that unifies scattered literature into a practical design space of 'what/when/how/where' with ample exemplars and cross-references that make the space navigable for both researchers and practitioners.\n* Useful formalization and careful distinctions from related paradigms, highlighting the centrality of non-parametric evolution and active exploration at test time.\n* Strong coverage of evaluation dimensions and the need for longitudinal, dynamic assessments; the emphasis on co-evolving benchmarks is timely and actionable.\n* Breadth of application survey and architectural perspectives (single vs. multi-agent) provides a balanced view from mechanisms to systems.\n\nKey weaknesses - \n* Scope and recency breadth come at the expense of depth: several cited lines (e.g., safety verification for code/tool creation, robust reward design, credit assignment in workflow co-optimization) would benefit from deeper critical analysis and clearer best-practice recommendations.\n* Limited quantitative synthesis: no meta-analysis or standardized comparative tables across methods under shared evaluation settings, making it hard to infer which evolution strategies work best under which constraints.\n* Evaluation guidance, while comprehensive conceptually, could include more concrete protocols, canonical task suites, and reporting templates to accelerate reproducibility and comparability.\n* Safety and governance sections flag important risks but stop short of prescriptive guardrails (e.g., sandboxing standards for tool creation, audit trails for self-modifying workflows, red-teaming for long-horizon drift).", "claims_and_evidence_yn": "Yes", "claims_explanation": "* The descriptive claims are well supported. The taxonomy and definitions are grounded in formalism, with clear what/when/how/where dimensions, and extensive, well-chosen exemplars mapped to each dimension. Figures and tables make the evidence traceable and convincing within a survey’s scope.\n\n* The distinctions from adjacent paradigms (curriculum, lifelong learning, model editing/unlearning) are accurate and justified, with specific criteria (test‑time adaptation, non‑parametric evolution, active exploration) and cited examples that substantiate the differences.\n\n* Evaluation guidance is directionally sound (adaptivity, retention, generalization, efficiency, safety) and supported by benchmark discussions, but it is more programmatic than prescriptive, concrete, standardized protocols and head‑to‑head results are limited.\n\n* Applications are documented with representative systems and use cases, supporting claims of relevance, though assertions about readiness and best practices remain qualitative.\n\n* Net: evidence is accurate and clear for the survey’s organizational and scoping claims, but it is less conclusive for comparative performance or one‑size‑fits‑all evaluation prescriptions due to the absence of unified quantitative syntheses.", "audience_yn": "Yes", "audience_explanation": "* Research fit: The survey targets an active shift from static LLMs to adaptive, agentic systems, offering a unifying 'what/when/how/where' framework that organizes a rapidly fragmented literature, squarely relevant to TMLR readers working on learning paradigms, agents, and evaluation.\n\n* Practitioner utility: It catalogs concrete mechanisms (model/context/tools/architecture), ties them to temporal adaptation regimes, and contrasts method families with design trade‑offs, giving practitioners actionable scaffolding for system design under real constraints.\n\n* Evaluation emphasis: It synthesizes goals and metrics for adaptivity, retention, generalization, efficiency, and safety, and discusses static, short‑horizon, and lifelong evaluation paradigms, useful to TMLR’s evaluation‑focused community.\n\n* Application breadth: Coding, GUI/web agents, healthcare, finance, and education case studies illustrate transfer from principles to practice, which appeals to applied ML researchers seeking domain-grounded guidance.\n\n* Timeliness and gaps filled: Prior surveys touch agents broadly, this one centers self‑evolution as a first‑class topic, addressing open questions on what to evolve, when, and how, meeting a current need for synthesis", "requested_changes": "Critical - \n* Add standardized evaluation protocols: Provide concrete, reproducible protocols for short‑horizon and long‑horizon assessments, including dataset splits, iteration budgets, reporting templates, and recommended metrics per goal (adaptivity, retention, generalization, efficiency, safety); include at least one end‑to‑end example protocol to anchor the taxonomy.\n\n* Include comparative synthesis: Add summary tables that normalize representative methods under shared settings (where possible), clarifying which “what/when/how” configurations work best under specific constraints (latency, cost, safety) and domains; note gaps where apples‑to‑apples comparisons are not yet feasible.\n\n* Make safety/governance guidance prescriptive: Translate the risk discussion into actionable guardrails (e.g., code/tool sandboxing defaults, audit trails for self‑modifying workflows, approval gates for tool creation, red‑team scenarios, rollback/failsafe patterns), plus a minimal compliance checklist for deployment.\n\nStrengthening - \n* Benchmark curation and gaps: Map evaluation goals to specific benchmarks with known limitations, propose modifications to cover dynamic/co‑evolution scenarios, and highlight underserved areas (e.g., long‑horizon memory with safety constraints, query‑specific architecture generation under latency budgets).\n\n* Cost/efficiency accounting: Provide a cost taxonomy (tokens, steps, wall‑clock, tool calls) and simple formulas for cost‑per‑gain and tool productivity; add a small worked example to illustrate trade‑offs.\n\n* Personalization and data governance: Offer guidance for personalized agents (data minimization, consent, on‑device memory, decay/forgetting policies, bias checks), plus evaluation metrics tailored to evolving personal contexts.", "broader_impact_concerns": "* Autonomy and misalignment: Self‑evolving agents can change prompts, memory, tools, and workflows without human oversight, raising risks of goal drift, deceptive strategies, or unsafe tool synthesis; mandate guardrails for self‑modification, human approval gates, and rollback mechanisms.\n\n* Tool/code creation risks: Autonomous tool discovery and code generation can introduce exploitable vulnerabilities, data exfiltration paths, or misuse; require sandboxing by default, static/dynamic analysis, provenance tracking, and least‑privilege execution policies.\n\n* Privacy and data governance: Memory evolution and personalization can accumulate sensitive user data over long horizons, creating re‑identification and leakage risks; specify data minimization, retention limits, consent, on‑device or partitioned storage, and redaction policies; report leakage metrics.\n\n* Safety and evaluation drift: Long‑horizon self‑evolution may degrade safety compliance over time; propose continuous safety auditing, policy‑compliance metrics, and red‑teaming protocols tailored to evolving workflows and tools.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)