EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems
Authors: Yufei He
Juncheng Liu
Yue Liu
Yibo Li
Tri Cao
Zhiyuan Hu
Xinxing Xu
Bryan Hooi
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Agents
Test Time Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce the J-TTL benchmark and EvoTest, a system where an LLM agent learns at test time as a second agent evolves its entire configuration from gameplay experience, no fine-tuning needed.

Abstract: A fundamental limitation of current AI agents is their inability to learn complex skills on the fly at test time, often behaving like “clever but clueless interns” in novel environments. This severely limits their practical utility. To systematically measure and drive progress on this challenge, we first introduce the Jericho Test-Time Learning (J-TTL) benchmark. J-TTL is a new evaluation setup where an agent must play the same game for several consecutive episodes, attempting to improve its performance from one episode to the next. On J-TTL, we find that existing adaptation methods like reflection, memory, or reinforcement learning struggle. To address the challenges posed by our benchmark, we present EvoTest, an evolutionary test-time learning framework that improves an agent without any fine-tuning or gradients—by evolving the entire agentic system after every episode. EvoTest has two roles: the Actor Agent, which plays the game, and the Evolver Agent, which analyzes the episode transcript to propose a revised configuration for the next run. This configuration rewrites the prompt, updates memory by logging effective state–action choices, tunes hyperparameters, and learns the tool-use routines. On our J-TTL benchmark, EvoTest consistently increases performance, outperforming not only reflection and memory-only baselines but also more complex online fine-tuning methods. Notably, our method is the only one capable of winning two games (Detective and Library), while all baselines fail to win any.

Published as a conference paper at ICLR 2026
EVOTEST: EVOLUTIONARY TEST-TIME LEARNING FOR SELF-IMPROVING AGENTIC SYSTEMS
Yufei He Juncheng Liu", Yue Liu', Yibo Li", Tri Cao', Zhiyuan Hu Xinxing Xu² Bryan Hooi National University of Singapore
Microsoff Research
(yufei.her yliu, liyibo, zhiyuan hu @ u..uus.oom, tricao2001 @maail.mm.mmmmm
bhooi @comp.nus.edu.sg.
{juncheng liu, xinxingxu} miccrosoft.com
ABSTRACT
A fundamental limitation of current Al agents is their inability to learn complex skills the fly at test time, often behaving like "clever but clueless interns" in novel environments. This severely limits their practical utility. To systemati- cally measure and drive progress on this challenge, we first introduce the Jeri- cho Test-Time Learning (J-TTL) benchmark. J-TTL is a new evaluation setup where an agent must play the same game for several consecutive episodes, at- tempting to improve its performance from one episode to the next. On J-TTL, we find that existing adaptation methods like reflection, memory, or reinforce- ment learning struggle. To address the challenges posed by our benchmark, we present EvoTest an evolutionary test-time learning framework that improves an agent without any fine-tuning or br evolving the entire agentic system after every episode. EvoTest has two roles: the Actor Agent, which plays the game, and the Evolver Agent, which analyzes the episode transcript propose revised configuration for the next run. This configuration rewrites the prompt, up- dates memory by logging effective state-action choices, tunes hyperparameters, and learns the tool-use routines. On our J-TTL benchmark, EvoTest consistently increases performance, outperforming not only reflection and memory-only base- lines but also more complex online fine-tuning methods. Notably, our method is the only one capable of winning two games (Detective and Library), while all baselines fail to win any.
INTRODUCTION
The pursuit of truly autonomous agents hinges on a critical human capability: the ability to learn "on the fly" (Maes, 1993; Franklin & Graesser, 1996). When faced with new task, humans can attempt it, reflect on their successes and failures, formulate a better strategy, and try again. By contrast, most AI agents arrive at deployment with fixed policy, behaving like "clever but clueless interns" that can execute instructions but cannot reform their own process from experience (Huang et al., 2024; Talebirad & Nadiri, 2023; Wang et al., 2024; 2025; Hou et al., 2023). This gap severely limits their reliability in dynamic settings. While the field acknowledges this problem, progress has been hampered by lack of standardized testbeds designed specifically to measure an agent's capacity for rapid, in-session improvement (Zhou et al., 2023; Mialon et al., 2023; He et al., 2025a; 2024; 2026; Li al., 2026; Yang et al., 2026; Sui et al., 2024a).
To address this, we first introduce the Jericho Test-Time Learning (J-TTL) benchmark, new evaluation framework designed to systematically measure and drive progress in on-the-fly agent learning. The benchmark's 's core task is straightforward: an agent must play the same complex, text- based adventure game (Hausknecht et al., 2020) for series consecutive attempts ("episodes"). In
The work done when the author intern Microsoft Research Asia Singapore
1 Corresponding author.
'The code is available at ittpt::///// hhtppp:://///tbb..t/e/ee com/ com/yffl hffee//ooo//sttttttt.....
Rvotest

---

Published as a conference paper at ICLR 2026
each episode, the agent interacts with the environment through standard loop: it receives a textual observation of its surroundings (state), submits a natural-language command (action), and receives numerical score change (reward). These games are difficult for LLM agents because they feature complex puzzles, long-range planning, sparse rewards (many critical actions yield no points), and irreversible consequences (a single wrong move can make the game unwinnable). The agent's goal is structured at two levels: 1) The Episodic Goal: Maximize the final score within a single playthrough. 2) The Learning Goal: Play the same game repeatedly and progressively increase its final score from one episode to the next, using only the experience gathered within that single session.
The J-TTL benchmark starkly reveals the inadequacies of existing adaptation paradigms. Consider a simple but critical failure in the game Detective: an agent gets stuck in navigation loop by repeat- edly attempting an invalid action, such as GO WEST, which the game rejects with "You can' t go that way. This seemingly simple failure reveals deep flaws in current adaptation methods: A Static agent has no learning mechanism and will likely repeat this error in every episode, leading to flat, low-scoring performance. An SFT (online) agent will have no good data to learn from in this failed episode. It is trapped because cannot generate the very data it needs to improve. An Reinforcement Learning (RL)(nlinne agent receives reward=0 for the invalid move, which is a weak signal a sparse-reward environment. A single update based on this noisy signal is insufficient to correct the policy, demonstrating failure of credit assignment. Methods based on reflection, such as Reflexion (Shinn et al., 2023), modify the agent's prompt with summaries of past failures. While useful, it does not alter the agent's core decision-makee making logic or its use of tools. Similarly, advanced memory systems (Packer et al., 2023; Zhong et al., 2024) improve an agent's ability recall information but do not teach how act differently. On the other end of the spectrum, RL and online fine-tuning are fundamentally ill-suited for the test-time learning setting. These methods are too slow and ineffiiiffic for the rapid learning J-TTL demands. To meet the challenge posed by our benchmark, we introduce EvoTest, an evolutionary test-time learn- ing framework designed for rapid, holistic adaptation without fine-tuning. EvoTest decouples acting from adaptation using two distinct roles: an Actor Agent that plays full episode and an Evolver Agent that improves the system between independent episodes. After each episode, the Evolver Agent analyzes the full transcript and proposes revised configuration for the entire agentic system. This process of whole-system evolution involves:
1. Rewriting the guiding prompt to encode new strategies;
Updating a structured deployment-time memory with records of successful and failure actions; Tuning recision-making hyperparameters like temperature and exploration strength;
4. Refining the tool-use routines that govern how and when memory or python code is accessed.
By evolving the agent configuration, EvoTest transforms the narrative of one episode into multi- faceted improvements for the next attempt, enabling a deeper form of learning than prior methods. We summarize contributions follows:
A Benchmark for Test-Time Learning: We propose J-TTL, a benchmark using Jericho games measure agent's 's on-the-fly learning ability across series of playthroughs of the same game.
A Test-time Learning Algorithm: We propose EvoTest, an evolutionary agent learning frame- work that evolves the entire agentic system (policy, memory, tool-use routines, and hyperparame- ters) via transcript-level analysis without gradients or fine-tuning.
State-of-the-Ar Empirical Results: We demonstrate on the J-TTL benchmark that Evotest shows 38% improvement over the strongest prompt-evolution baseline and a 57% improvement nemory-basede and gradient-based
over online RL, outperforming all strong reflection- baselines on every game.
2 RELATED WORK
From Static Agents to Test-Time Learning. The majority of current AI agents, while capable, op- erate with static configurations that are manually designed and fixed after deployment (Wang et al., 2024: Xi et al., 2025; He et al., 2025b; Chen et al., 2025b;a; He et al., 2025c:d; Chen et al., 2025c). This limits their ability to adapt to novel situations, a key challenge motivating the development of "self-improving Al agents" (Gao et al., 2025b; Fang al., 2025; Gao et al., 2025a; Sui al., 2025;
2

---

Published as a conference paper at ICLR 2026
1 WIN
LOSE
LOSE
Test-Time Learning
Episade
Evolve
Episode 2 Act
Evolve
Episade
Act
Act Phase
Episade & Agent Configuration
Evolve Phase
humpl Memsote
Configurations
Tesh-uee use
Selection vaa Upper
R
+
STaLE raward action
New Agent Configurttion
Evoluer Agent
Actor Agent
B
System Prompr
Sptsodo Trajectory Transcript
agent contiguran lons Fonstitess
Agent Eouurdonn de
Jdcntall ussiul, actiions Lee zncroaacs noeded progresasng the gemo..
Stap verura
The EvoTest architecture, designed to enable test-time learning (TTL). The agent operates a continuous Act-Evolve loop across multiple attempts at the same task. After each episode, the Evolver Agent analyzes the full trajectory transcript- -rich narrative feedback to perform gradient-free, whole-system evolution on the agent's s entire configuration. This allows the agentic system self-improve on the fly, directly from its experience at test time.
Figure 1:
2024b; Liu et al., 2025b). A prominent line of work enables agents to learn from past mistakes with- out updating weights. Reflexion (Shinn et al., 2023), key baseline for work, allows agent to verbally reflect on trajectory failures and append these reflections to its prompt for subsequent episodes. Other approaches focus enhancing agent memory. For instance, MemGPT (Packer et al., 2023) provides agents with a structured memory system to manage long contexts. Beyond re- flection/memory, Uncertainty of Thoughts (Hu et al., 2024) adds test-time uncertainty-aware plan- ning, deciding when to ask, verify, or revise without weight updates. While MemoryBam Bank (Zhong al., 2024) hicrarchical summarization retain information over long interactions.
Self-Evolving Agentic Systems. Another active area of research is the automated optimization of prompts that guide agent behavior (Liu et al., 2025a; Zhu et al., 2026; Hu et al., 2026). Generative approaches like APE (Zhou et al., 2022) and OPRO (Yang et al., 2023) use a powerful LLM to propose and score new prompts, iteratively refining them based on performance. Gradient-inspired methods like TextGrad (Yuksekgonul et al., 2024) refine prompts using LLM-generated textual feed- back. Closely related to our work are evolutionary methods such as AlphaEvolve (Novikov et al., 2025), Promptbreeder (Fernando et al., 2023), and EvoPrompt (Guo et al., 2024), which maintain population of prompts and apply genetic operators like mutation and crossover to discover more effective instructions. EvoTest generalizes prompt evolution to whole-system evolution, optimizing the entire agentic tines. This allows for more holistic adaptations, such tuning exploration strength, that are beyond the scope of prompt-editing alone. This vision for unified optimization is shared by EvoAgent (Yuan al., 2024) and MASS (Zhou al., 2025); Beyond 'Aha!' (Hu et al., 2025) complements this by aligning meta-abilities rather than only task prompts or single components.
the prompt, memory, hyperparameters, and tool-use rou-
3 THE JERICHO TEST-TIME LEARNING (J-TTL) BENCHMARK
To systematically measure and drive progress in on-the-fly agent learning, we introduce the Jericho Test-Time Learning (J-TTL) benchmark. This benchmark built upon the Jericho (Hausknecht et al., 2020) suite of Interactive Fiction (IF) games. IF games are fully text-based simulation en-
Jericho is available h hrtttiiii.cccoooooooorrreeerri

---

Published as a conference paper at ICLR 2026
vironments where agent issues text commands effect change in the environment and progress through a story. While the richness of these environments makes them a challenging testbed for AI, existing evaluation has primarily focused single-episode performance different games (Hausknecht et al., 2020; Gulcehre et al., 2020; Li et al., 2025). The J-TTI bench- mark refocuses the evaluation on different, critical axis: agent's ability to learn and improve its strategy through repeated attempts at the same complex task within single test session.
generalization across
Datasets. We use publicly available Jericho games that vary in difficulty and puzzle structure, including Detective, Library, Zorkl, Zork3, Balances, and Temple. Games are launched via Jericho with default scoring. Each episode is capped by step limit 110 unless stated otherwise).
The Jericho Game We model a Jericho game (Hausknecht et al., 2020) as a Partially Observable Markov Decision Process (POMDP), defined by the tuple AS T, R, 2,, T). Here, S is the latent state space, and A is the (infinite) combinatorial action space of natural language commands. At each step 6, an agent in latent state St E S takes an action a, A, causing transition state $t+1 T- St. Qz) and yielding a scalar reward r2 R(st, 0e). The agent does not observe the true state s but instead receives a textual observation o2 ~ -2(- st). An episode is trajectory of interactions with a finite horizon of 'I' steps:
new
(e) (e)
(e)
(e) (c) (e)
01
aT TT
The total return for an episode e is the sum of its rewards, R(e) - EE
Test-Time Learning Process on Jericho. The J-TTL benchmark protocol consists of session of K consecutive episodes played in a single game. After each episode concludes, the game environment is reset to its identical initial state, (e) sinit for all E {l..... .L.....K) K}. This ensures that any performance improvement solely attributable to the agent's internal learning process.
Let the agent be parameterized by set of learnable components 0 E O. In episode e, the agent's behavior is governed by a policy conditioned on its current learnable components, Tg(-).
The core of test-time learning lies in the update rule agent applies between episodes. After com- pleting episode and collecting the trajectory data r(*), the agent updates its learnable components for the next episode:
( U((()), (()))
where U : x Thist > ② is the agent's 's learning algorithm.
Evaluation Metrics, For a test-time learning session consisting of K episodes, we record total score for each episode, R(e). This yields a performance sequence for the entire session:
(R(1), R(2). R(K)).
From this sequence, we derive two metrics:
Learning Curve. A plot of the final return R(e) against the episode index e. This curve provides a direct visualization of an agent's learning progress.
Area Under the Curve (AUC). For quantitative comparison, we define the AUC as normalized
ratio. It measures the agent's total achieved score against the maximum possible score over the session:
ΣK R(e) AUC
(4)
K Rmmax
where Rmax is the maximum achievable score in single episode of the game. This metric yields a value between and 1, facilitating comparison across games with different scoring scales.
THE EVOTEST FRAMEWORK
To address the challenges posed by the J-TTL benchmark, we introduce EvoTest, evolutionary test-time learning framework, as illustrated in Figure 1. Unlike methods that perform gradient-basee updates model weights, EvoTest operates on fixed, non-trainable backbone LLM. It achieves learning by evolving the agent's 's entire high-level configuration between episodes, leveraging the rich, narrative feedback from game transcripts rather than just sparse, scalar rewards.
4

---

Published as a conference paper at ICLR 2026
4.1 THE AGENTIC CONFIGURATION
In the context of EvoTest, the general learnable components 0 from the J-TTL. formulation (Sec- tion 3) are instantiated as a holistic agentic configuration, denoted by E X. This configuration is a tuple X = (p, M, h, u) that defines the agent's s complete operational strategy:
Policy Prompt (p): A system prompt that provides high-level strategic guidance, heuristics, and behavioral guardrails g the backbone LLM.
Deployment-time Memory (Af): A structured, queryable database populated by the Evolver Agent after each episode. It stores the agent's prior experiences, effectively creating persistent knowledge base. The memory is organized into distinct components, such as: (a) a success mem- ory that logs state-action pairs which led score increases (e.g.. (state-hash, action) -> score www.edeelaaa delta), and (b) a failure memory that records patterns associated with stalls or neg- ative outcomes (e.g., repetitive action loops in a specific location). This allows the agent to recall specific, effective actions and avoid known pitfalls from past playthroughs. For detailed break- down of the memory's s data structure and concrete examples its contents, please see Appendix Hyperparameters (h): A set of parameters controlling the LLM's inference and the agent's decision-making, such temperature, exploration strength, and stopping criteria.
Tool-Use Routines (u): Active components executed at each decision maaiinn step to operational- ize knowledge and create useful state abstractions. These routines consist of two functions:
- Memory Interaction Logic: A set of rules governing how to query the memory (M). Before deciding on an action, this routine might check if the current game state exists in memory. If match is found in the success memory, the routine can inject the proven action into the LLM's prompt strong suggestion (e.g.. "Hint: this exact situation before, the action 'unlock door with key" worked well.").
State Abstraction Logic: the raw, verbose game history into a concise summary of progress. Instead of forcing the LLM to re-read the entire history, this tool parses the log for key environmental cues (e.g., g. the text "The anci ent scroll di sintegrates, revealime ling a map. ") and returns short, meaningful milestone string (e.g., "Mi Mestore::: Found the map. "). This ab- stracted state is included in the prompt at every step, providing efficient situational awareness.
An evolvable Python function- the state extractor that processes
The agent's policy in episode e, *x(e), is therefore function of the fixed LLM'' behavior modu- lated by this entire configuration.
4.2 THE TWO-AGENT LEARNING LOOP
EvoTest operationalizes the test-time learning update rule, x(+11) ((x(e)), ,(e)), through coop- erative two-agent design that separates acting from adaptation.
1. The Actor Agent. For given episode the Actor Agent is provided with single, fixed config- uration (=). It plays through the episode by repeatedly querying the backbone LLM, conditioning its actions on the current observation or and any information retrieved from its memory M(e) as dictated by its tool-use routines (e)) The result of its playthrough is the trajectory ((e) and the final return R(e).
2. The Evolver Agent. After the episode, the Evolver Agent takes the trajectory transcript (e)) and the parent configuration (<) as input. It performs whole-system evolution by generating of new candidate child configurations, ((e+1) (e+1)) Xm ((+1)). This the core learn- ing step, where the Evolver uses an LLM to analyze the previous episode's trajectory and propose improvements to the agentic configuration. The evolutionary operators include:
Prompt Mutation:
The Evolver rewrites the policy prompt ((e) create new prompt p. It incor- porates strategies that proved effective (c.g., adding "always examine objects before taking them") adds explicit rules to prevent observed failures (e.g., "avoid repetitive navigation loops").
Memory Update: The Evolver programmatically parses the transcript (()) to identify and log im- portant events. It records state-action pairs (oc, ar) that immediately preceded score increase in
5

---

Published as a conference paper at ICLR 2026
"success" table. It also identifies sequences of interactions that led to progress and records them in a "failure" table. This updated memory M(++1) is then inherited by all child configurations.
Hyperparameter Tuning: The Evolver proposes adjustments to the hyperparameters /(() to cre- ate h. For example, if the transcript shows the agent was stuck in a repetitive loop, the Evolver might suggest increasing the LLM's 'temperature" to encourage more diverse actions.
Tool-Use Refinement: The Evolver can modify the logic in u(() to create ū, changing when how the agent consults its memory. For example, it might strengthen rule in the prompt from a general suggestion to a direct instruction, such as telling the agent it must check its success memory before every action. The specification of these tools and the required JSON output format are detailed in the master prompt provided in Appendix H.
A child configuration is new combination of these potentially mutated components, representing a new hypothesis for a more effective agent.
4.3 CONFIGURATION SELECTION VIA UCB
After the Evolver generates a of new "child" configurations, EvoTest decide which one use for the next episode. This creates a classic dilemma: should we exploit configuration that has worked well in the past, or should we explore a new, untested one that might be even better?
To manage this trade-off, select the next configuration, (e-1), from candidate pool containing the parent from the previous episode. (()), and its new children, C(++1), The selection is guided by the Upper Confidence Bound (UCB) algorithm, a strategy from multi-armed bandit theory ( (Vermorel & Mohri, 2005) designed to manage the exploration-explootaaoo dilemma. The rule selects the configuration that maximizes the following score:
log N
X (o+1)
arg max
m(x)+ + 8
(5)
X XE(x()))UC--n)
1 + n(x)
The score for each configuration X is sum of two parts:
Performance Term (A(X)): This is simply the average score the configuration has achieved so far. It encourages us to re-use configurations that have good track record.
Exploration Bonus (..::): This term gives boost to configurations that are less certain about. n(x) is the number of times configuration has been tried, N is the total number of episodes completed, and is a hyperparameter controlling the exploration strength. The bonus is higher for configurations that have been tried fewer times (n(x) is small), making novel mutations attractive.
At the start of each new episode, the UCB algorithm selects a single configuration, which is then used for the entire duration of that episode. The episode's 's final score is then used to update the statisties for that chosen configuration. Crucially, this UCB approach also makes our learning process more stable and helps prevent sharp drops in performance. A simpler, greedy strategy might get fooled by a new configuration that gets a lucky high score and then repeatedly fails. UCB avoids this pitfali. Because the reliable parent configuration poorly after its initial trial, its average score (î) will drop. The UCB rule can then naturally "fall back" to the tested parent, which retains its high score. This acts as safety net, preventing the system from getting stuck on bad evolutionary path and ensuring more consistent improvement.
always in the selection pool, new child performs
EXPERIMENTS
Our experiments are designed to answer three central research questions regarding test-time learning on our J-TTL benchmark:
RQ1: Does test-time learning (TTL) lead to meaningful performance improvements on complex tasks compared to a non-learning agent?
RQ2: Does our proposed EvoTest framework enable more effective test-time learning compared existing methods, such as those based memory, reflection, and prompt optimization?

---

Published as a conference paper at ICLR 2026
Comparison of Area Under the Curve (AUC) scores on the J-TTL benchmark 5-flash across six Jericho games for two backbone LLMs: google/gemintn -.. (G) and anthropic/ cwtrrpoccccauue---101 clauue-4--onee--22333 backbone and are not distinguished by model. Higher values indicate better overall performance throughout the test-time learning session. The best performance in each column is highlighted in bold. Our method, EvoTest, consistently outperforms all baselines across both backbones.
Table 1:
(C). Weight-update methods use separate
Detective
Library
Zorkl
Zork3
Balances
Temple
Avg.
Method G
C
G
C
G
C
G
C
G
C
G
C
G
C
Non-learning Baseline
Static
0.21 0.23 0.15 0.16 0.03
0.04
0.05
0.06 0.11
0.12
0.08
0.09
0.11
0.12
Memory-based & Reflection-based Methods
Memory
0.25
0.26
0.18
0.20
0.04
0.05
0.06
0.07
0.13
0.14
0.10
0.11
0.13
0.14
RAG
0.32
0.34
0.24 0.25
0.07
0.08
0.09
0.10
0.18
0.20
0.15
0.16
0.18
0.19
Summary
0.45
0.47
0.33
0.35
0.12
0.13
0.15
0.16
0.25
0.27
0.21
0.22
0.25
0.27
Reflexion 0.58
0.60 0.41
0.44
0.09
0.11
0.25
0.27
0.30
0.32
0.29
0.31
0.32
0.34
Automated Prompt Optimitation Methods
TextGrad
0.61
0.62
0.45
0.47
0.11
0.13
0.28
0.30
0.16
0.18
0.23
0.25
0.31
0.33
Promptbreeder
0.63
0,65
0.47
0.49
0.10
0.12
0.29
0.31
0.23
0.25
0.30
0.32
0.34
0.36
EvoPrompt 0.65
0.67
0.48 0.50
0.10
0.12
0.30
0.32
0.24
0.26
0.27
0.29
0.34
0.36
Weight- Update Methods (Online Fine-Tuning,
SFT (online)
0.40
0.30
0.11
0.18
0.22
0.19
0.23
GRPO (online) 0.55
0.38
0.07
0.22
0.30
0.26
0.30
Tvo (Ours)
0.94
0.95
0.77 0.80
0.14 0.16
0.35
0.38
0.32
0.35
0.31
0.34
0.47
0.50
RQ3: How does the gradient- evolutionary approach of EvoTest compare to traditional gradient-based RL methods in the context of test-time learning?
5.1 SETUP
Backbone LLM. We use two powerful API models for the Actor Agent: and the highly capable
the cost-effective 2 The Evolver Agent, which performs the The fine-tuning base-
anthropic/ http:/cllaee---ooeeee----
most complex reasoning, is powered by openai/o3-2025-44-1 lines (SFT and GRPO) are implemented on /wen///een--a 32b, a large open-source model. Experiments with more LLM backbones can be found in Appendix N
Baselines. As foundational reference, a non-learning Static agent with fixed configuration es- tablishes zero-shot performance. The learning baselines are grouped into four main categories: (1) Memory-based Methods, which include (1a) Memory, which places the complete session tran- script history into the context for in-context learning, automatically truncating the oldest parts if the context limit is exceeded, and (1b) RAG, which retrieves relevant snippets from past trajectories; (2) Reflection-based Methods, which learn by appending textual feedback to the prompt, including (2a) Summary, which uses an LLM to progressively summarize the entire history of all past tran- scripts and feeds this condensed summary into the context, and (2b) Reflexion (Shinn al., 2023), which generates structured textual self-refiections after each episode to critique performance and guide the next attempt; (3) Automated Prompt Optimization Methods, which iteratively refine the guiding prompt, including (3a) TextGrad (Yuksekgonul et al., 2024), where after each episode, an optimizer LLM analyzes the trajectory to generate "textual gradient"-a critique describing how to improve the prompt-which is then applied for refinement, and two evolutionary approaches, (3b) Promptbreeder (Fernando et al., 2023) and (3c) EvoPrompt (Guo al., 2024), which evolve population of prompts; and (4) Weight-Update Methods, which contrast with our gradient-free free approach by performing online fine-tuning, including (4a) SFT (online), which performs Super- vised Fine-Tuning the actor model after each episode using the state-action pairs collected from the trajectory, and (4b) GRPO (online) (Shao et al., 2024), which applies gradient- based Reinforce- ment Learning policy updates to the model using the scalar rewards collected during the episode. All methods are evaluated under the same step budget and use the same backbone LLMs for their re- spective roles to ensure fair comparison. The detailed introduction be found in appendix C, D.
7

---

Published as a conference paper at ICLR 2026
Detective (Rmac 360)
Library (Rmax = 30)
Zorkl (Rmax 350)
360
Final score per prrepsode Rie)
Final score per apisdde Rie)
500
60
Final score episede
50-
240
-0
40
120-
10
zo-
20
50
50
1.0
30 Episode index
Episade index
Episode
Zork3 (Rmax =
Balances (Rmax = 51)
Tem (Rmax = 35)
Final score per ephocee Rie)
Final score per epšode Rle)
35
-00
6-
30
Final per episode
20
20
15
15
20
a0-
50
40
Episode index
Episode index
Episode index
EvoTest
EvoPrompt
Reflexion
GRPO
Memory
Static
Figure 2:
Learning curves showing final score per episode across six Jericho games with google het///emiiiiiiii 1aaiini--.------- 5-flash as LLM. EvoTest consistently demonstrates steeper and more stable learning trajectory compared to basclines, validating its effectiveness for test-time learning.
5.2 RESULTS
Table presents the AUC scores for all methods across the six selected Jericho games, while the learning curves in Figure 2 illustrate the per-episode performance progression. We analyze these results through the lens of our three research questions.
Test-Time Learning is Effective. The results provide a clear affirmative answer. Across all games, every learning-based method achieves a higher average AUC score than the Static baseline (Table 1). The learning curves (Figure 2) further confirm this, showing that methods capable of learning from experience consistently exhibit upward- trending scores, whereas the Static agent's s performance remains flat. This demonstrates that even with just few attempts, test-time learning is a valid and effective paradigm for improving agent performance on complex, long-horizon tasks.
RQI:
EvoTest Outperforms Existing Adaptation Methods. EvoTest consistently and substan- tially outperforms all other gradient-free baselines. In Table 1, EvoTTest achieves the highest AUC score all six games, with average score of 0.47/0.50-aa significant improvement the next best bascline, EvoPrompt (0.34/0.36). The learning curves reveal not just higher final scores but steeper rate of improvement, indicating more efficient learning.
RQ2:
The key insight lies in the limitation of single-channel adaptation. Memory and RAG provide raw information but offer no strategic guidance, leading low performance ceiling. Reflexion and 1 Promptbreeder perform better by refining strategy, but they are constrained single axis: the prompt. An agent can have a perfect prompt but still fail due to poor exploration (e.g., low temperature) its holistic, whole-system evolution. By concurrently optimizing the prompt, memory-acces rou- tines, and decision hy hyperparameters, it discovers and resolves complex performance bottlenecks that single-channel adaptations cannot. For example, it can learn to increase exploration temperature in early episodes and simultaneously add new strategic heuristic to its prompt based on its findings, multi-faceted adaptation that other methods are incapable of, as detailed and plotted in Appendix M.
other prompt-fouused optimizers like
inefficient use of its knowledge. EvoTest's strength is
RQ3: Evolutionary Adaptation is More Data-Efficient than RL at Test Time. The comparison between Evotest and the weight-updane methods clearly favors the evolutionary approach this set- ting. Evo average AUC (0.47/0.50) is substantially higher than that of GRPO (online) (0.30), the gradient-bart based RL baseline. This improvement indicates that Evotest successfully addresses a fundamental challenge in test-time learning: extreme data scareity.
8

---

Published as a conference paper at ICLR 2026
Table 2: Practical costs for a single leamning up- date.
Ablation Study Detective Game (Rmax 360)
Final Score per Episode R(e)
Method
Update Time
LLM Calls
SFT (online)
5-10 min
0
GRPO (online)
5-10 min
190
TextGrad
30-50 sec
2
Suollen (Pul Systems)
weo Prompr Eeciuction
EvoTes (Ours)
20-30 sec
wéo UCB Satection
weo Memoery Deslucton
EvoPrompt
20-30 sec
wes Hyperpuraretir Tuning
Reflexion
15-25 sec
wes Tock-ise Refinement
RAG
5-15 sec
0 (emb.)
Episode Index
Figure 3: Learning curves from the component ablation study on the Delective game.
Memory
<1 sec
Table 3: Ablation study on EvoTest compo- nents, showing Area Under the Curve (AUC) scores.
Table 4: Ablation on the Evolver agent's LLM. Performance, measured by AUC, correlates with model quality.
Detective Zorkl Balances
Evolver LLM
Detective Zorkl Balances
Evo
0.94
0.14
0.32
spenai/o3
0.94
0.14
0.32
w/o Prompt
0.52
0.05
0.16
deepseek/r)
0.90
0.12
0.29
w/o UCB
0.68
0.08
0.22
qwen3-32b
0.82
0.10
0.25
w/o Memory
0.82
0.11
0.28
qwen3-8b
0.68
0.07
0.20
w/o Hyperpara.
0.89
0.12
0.30
w/o Tool-Use
0.91
0.13
0.30
Static
0.21
0.03
0.11
Moreover, EvoTest alleviate traditional RL's reliance scalar rewards for credit assignment. In sparse-reward environments like Jericho, single episode provides a noisy and insufficient signal for effective gradient updates. It is inefficient way to learn from one complex success or failure. In contrast, Test bypasses this issue by leveraging the entire episode transcript as a rich, nar- rative feedback signal. The Evolver Agent performs credit assignment through semantic analysis of the game's story, identifying causal chains of failure (e.g., "the agent got stuck in a loop here") and success. This allows to make explicit, targeted, and structural edits to the agent's configura- tion. In essence, EvoTest shifts from credit assignment via backpropagation to credit assignment via narrative analysis, a more data-efficient mechanism for learning from a single experience.
5.3 MODEL. ANALYSIS
Ablation on Key Components. Our ablation study reveals the distinct contributions of each com- ponent in the EvoTest framework. The AUC scores in Table quantify the overall impact, showing that removing any component degrades performance, The largest performance drop occurs when re- moving prompt evolution (w/o Prompt), confirming that evolving the high-level policy is the primary driver of strategic adaptation.
The learning curves in Figure 3 offer deeper insight into the dynamics of these failures, particularly for the UCB ablation. While removing UCB also causes significant drop in AUC, the curve reveals more nuanced issue: instability. The w/o UCB agent, which uses greedy selection strategy, is prone to catastrophic drops in performance. This happens when the agent over-commits to a high- risk mutation that achieved a lucky high score in one episode but is not robust. Without UCB's exploration mechanism-whii encourages revisiting more reliable past configurations- gets stuck on a suboptimal evolutionary path and unable to correct its course after a poor choice. In contrast, the full Evo Test model leverages UCB maintain stable learning trajectory.
agent
Efficiency Analysis. For learning at test-time, the update step between episodes is a major bottle- neck. Our experiments show a clear divide in practicality between different approaches, as detailed in Table 2. Weight-update methods like online RL are not practical for this setting. A fine-tuning pass episode's data took 5 to 10 minutes expensive hardware, making it a non non-starter for a system that needs to learn at test time. In contrast, EvoTest and other gradient-free methods operate different timescale. Instead of costly training
H100 GPUs. This is not just slow: it demands

---

Published as a conference paper at ICLR 2026
Table 5: Ablation on the structure of prompt evolution. "Simple Mutation" uses a generic improve- ment instruction, while "Full" uses our structured, multi-part master prompt. Scores are AUC.
Detective
Zorkl
Balances
Evst (Full Structured Evolution)
0.94
0.14
0.32
EyoPrompt
0.65
0.10
0.24
EvoTest (w/ Simple Mutation)
0.65
0.07
0.20
EvoTeest (w/o Prompt Evolution)
0.52
0.05
0.16
Table 6:
Comparison of AUC scores in a setting where the Actor LLM is fixed to
qwen/qwen3
-2bb for all methods.
Method
Actor LLM
Evolver LLM
Detective
Zork1
Balances
Avg.
SFT (online)
qwen3-32b
N/A
0.40
0.11
0.22
0.24
GRPO (online)
qwen3-325
N/A
0.55
0.07
0.30
0.31
Evo'Test (Ours)
qwen3-32b
qwen33-32b
0.68
0.10
0.28
0.35
EvoTest (Ours)
qwen3-325
openai/o3
0.78
0.12
0.31
0.40
run, our learning step is single API call to an LLM, which takes about 20-30 seconds. We provide more formal analysis of the computational complexity of EvoTest in Appendix E
Impact of the LLM of Evolver Agent.
To assess the sensitivity of our framework to the Evolver's reasoning capabilities, conducted an ablation study its underlying LLM (Table 4). The results reveal a clear correlation between model scale and agent performance; more powerful models like openai/ 1/33////////// consistently yield higher scores, likely due to their superior ability to distill complex strategic insights from raw episode transcripts. Notably, even with a significantly smaller model such 3wen3-8b, performance remains substantially above the non-learning Static baseline. This finding demonstrates the robustness of the EvoTTes framework: while a more capable Evolver LLM acts performance amplifier, the fundamental act-evolve loop effective in its own right.
Ablation on the Structure of Prompt Evolution. To further isolate the contribution of our prompt- evolving logie, create baseline, "EvoTTes (w/ Simple Mutation))" which replaces our detailed Evolver prompt with a generic instruction to simply "analyze the trajectory and generate an im- proved prompt." " The results in Table show that while this simpler mutation improves over no prompt evolution, it significantly underperforms our system, with the AUC score on Detective drop- ping from 0.94 0.65. This demonstrates that the performance gains are not just from evolving the prompt, but from the Evolver's sophisticated analysis, which is core contribution of our framework.
A Fairer Comparison with Fine-Tuning Methods. In this setup, we normalize the underly- ing model capabilities by using qwen / /Wenn-33bbb As shown in Table 6, we evaluate two versions of our method: one using qwen/ (aeen//wen33332 for both the Actor and Evolver, and a second version that pairs the qwen/qwen3 3-wm/333n333 Actor with the stronger opena /o3 Evolver. The results confirm that even when both components use Evoeest (Avg. AUC 0.35) outperforms the strongest weight-updat baseline, GRPO (0.31). Furthermore, the performance leap when using a more capable Evolver (Avg. AUC 0.40) underscores the impact of the optimizer's reasoning ability within framework.
the backbone for all compared methods.
qwen/ swen/qwen3-a
6 CONCLUSION
We introduce the J-TTL benchmark to measure test time agent learning and proposed EvoTest, novel evolutionary framework that improves agentic systems at test-time without gradients. By analyzing entire episode transcripts, EvoTest evolves the complete agent configuratio memory, and hyperparameters- rapidly adapt. Our experiments show EvoTest significantly outperforms strong baselines, including reflection, prompt optimization, and online fine-tuning. Its strength lies in using rich, narrative feedback for credit assignment, far more data-efficient paradigm than relying on sparse rewards. This work provides a concrete step toward building truly autonomous agents that learn and self-improve from experience.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This research is supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2025) (Grant MOE-T2EP20124-0009)
ETHICS STATEMENT
All authors of this paper have read and adhered to the ICLR Code of Ethics. Our work focuses on foundational research into the learning capabilities of Al agents within simulated, text-based environments. We have identified and considered the following potential ethical dimensions of this research:
Inherited Bias in Language Models Our framework, along with the baselines, utilizes large lan- guage models (LLMs) backbones. It is well-documented that LLMs can inherit and amplify societal biases present in their training data. While the fictional context of the Jericho games is unlikely to surface common social biases, we acknowledge that the agents' generated language and decisions are fundamentally shaped by the underlying models. Our research does not introduce new sources of bias but operates within the existing limitations current LLM technology.
Dual-Use and Long-Term Implications Research into self- seff-improving autonomous agents con- tributes to long-term vision of more capable and independent AI. Such technology could, in distant future, have dual-use potential. However, our work is situated at very early, foundational stage and is confined to a controlled, non-phin physiccl,,,,, and non-real-word gaming environment. The primary goal is to understand and measure test-time learning in sandboxed setting, which is critical step for developing safer and more reliable AI systems.
This research does not involve human subjects, personal data, any form of deception. The datasets (Jericho games) and software are publicly available. We have been trans- parent about the use of LLMs for language polishing in the manuscript, detailed in Appendix B. We have no conflicts of interest that could have influenced the results or their interpretation.
Research Integrity
REPRODUCIBILITY STATEMENT
We are committed to the full reproducibility of our work. To this end, we have made extensive efforts to document our methodology and provide all necessary artifacts. The primary resources for reproducing our results are detailed in the appendices.
Our complete source code, which includes the implementation of the EvoTest framework. the J- TTL benchmark setup, and all baseline methods described in the paper, is provided in https: //github. com/ hom/yf-Bi /ete///fe/e//eeets............)) Appendix C provides comprehensive implementation de- tails, including the specific Jericho games used, the environment configuration (e.g., episode step limits), hardware requirements, and the exact model identifiers for all LLMs used. Appendix D) offers detailed descriptions of all baseline methods, enabling faithful re-implementation of comparisons, The core of our method's 's learning mechanism, the Evolver Agent's 's master prompt, is provided in its entirety in Appendix H). Further experimental details, including a controlled com- parison on the Qwen3-322 backbone, are available in Appendix L.. All key hyperparameters and random seeds will be included in our public code release to ensure deterministic replication. We believe these resources provide clear and complete pathway to reproduce experiments and validate our findings.
EvoTestt.
REFERENCES
Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, and Bryan Hooi. Mlr-bench: Evaluating ai agents on open-ended machine learning research. arXiv preprint 200.::::505.99555. 2025a.
Yulin Chen, Haoran Li, Yuan Sui, Yufei He, Yue Liu, Yangqiu Song, and Bryan Hooi. Can indirect prompt injection attacks be detected and removed? arXiv preprint arXiv:2502. /6580, 2025b.
11

---

Published as a conference paper at ICLR 2026
Yulin Chen, Haoran Li, Yuan Sui, Yue Liu, Yufei He, Yangqiu Song, and Bryan Hooi. Robustness via referencing: Defending against prompt injection attacks by referencing the executed instruction. arXiv preprint arXiv:2504.20442 2025c.
Jinyuan Fang, Yanwen Peng, Xi Zhang, Yingxu Wang, Xinhao Yi, Guibin Zhang, Yi Xu, Bin Wu, Siwei Liu, Zihao Li, et al. A comprehensive survey of self-evolving ai agents: A new paradigm bridging foundation models and lifelong agentic systems. arXiv preprini arXiv:2508. 07407, 2025.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309. 16797, 2023.
Stan Franklin and Art Graesser. Is it an agent, or just a program?: A taxonomy for autonomous agents. In International workshop on agent theories, architectures, and languages, pp. 21-35. Springer, 1996.
Hongcheng Gao, Yue Liu, Yufei He, Longxu Dou, Chao Du, Zhijie Deng, Bryan Hooi, Min Lin, and Tianyu Pang. Flowreasoner: Reinforcing query-level meta-agents. arXiv preprint arXiv:2504. 15257, 2025a.
Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, et al. A survey of self-evolving agents: On path to artificial super intelligence. arXiv preprint 27X1v:250..21046, 2025b.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances in neural information process- ing systems, 33:7248 72599, 2020.
Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Evoprompt: Connecting Ilms with evolutionary algorithms yields powerful prompt optimizers. ICLR, 2024.
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. Interac- tive fiction games: colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7903-7910, 2020.
Yufei He, Zhenyu Hou, Yukuo Cen, Feng He, Xu Cheng, and Bryan Hooi. Generalizing graph trans- formers across diverse graphs and tasks via pre-training on industrial-scale data. arXiv preprint arXiv:2407.03953, 2024.
Yufei He, Ruoyu Li, Alex Chen, Yue Liu, Yulin Chen, Yuan Sui, Cheng Chen, Yi Zhu, Luca Luo, Frank Yang, et al. Enabling self-improving agents to learn at test time with numan-in-the-loon guidance. arXiv preprint arXiv:2507.1711 2025a.
Yufei He, Yuexin Li, Jiaying Wu, Yuan Sui, Yulin Chen, and Bryan Hooi. Evaluating the paper- clip maximizer: Are rl-based language models more likely to pursue instrumental goals? arXiv preprint wwxXix:2502.1220000 2025b.
Yufei He, Yuan Sui, Xiaoxin He, and Bryan Hooi. Unigraph: Learning a unified cross-domain foundation model for aext-attributed graphs. In Proceedings of the 31 ACM SIGKDD Conference Knowledge Discovery and Data Mining V. I, pp. 448 4589459, 2025c.
Yufei Hc, Yuan Sui, Xiaoxin He, Yue Liu, Yifei Sun, and Bryan Hooi. Unigraph2: Learning unified embedding space to bind multimodal graphs. In Proceedings of the ACM on Web Conference 2025, pp. 1759-17781 1770, 2025d.
Yufei He, Juncheng Liu, Zhiyuan Hu, Yulin Chen, Yue Liu, Yuan Sui. Yibo Li, Nuo Chen, Jun Hu, Bryan Hooi, et al. Evoclinician: A self-evolving agent for multi-turn medical diagnosis via test-time evolutionary learning. arXiv preprint N601:2200..29660 2026.
Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. Graphmae2: A decoding-enhain eehhancd masked self-supervised graph learner. In Proceedings of the ACM web conference 2023, pp. 737-746, 2023.
12

---

Published as a conference paper at ICLR 2026
Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See- Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty- aware planning enhances information secking in large language models. arXiv preprint arXiv:2402.03271, 2024.
Zhiyuan Hu, Yibo Wang. Hanze Dong, Yuhui Xu, Amrita Saha, Caiming Xiong, Bryan Hooi, and Junnan Li. Beyond' aha!": Toward systematic meta-abilities alignment in large reasoning models. arXiv preprini arXiv:2505. 10554, 2025.
Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See-Kiong Ng. Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, and Bryan Hooi. Rewarding the rare: Uniqueness-awan rl for creative problem solving in Ilms. arXiv preprint wrXiv:2601.086.. 2026.
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang. Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of Ilm agents: A survey. arXiv preprint XXXx:24422.02116 2024.
Yibo Li, Zijie Lin, Ailin Deng, Xuan Zhang, Yufei He, Shuo Ji, Tri Cao, and Bryan Hooi. Just-in- time reinforcement learning: Continual learning in Ilm agents without gradient updates. arXiv preprint arXiv:2601. 18510, 2026.
Zhengyang Li, Qijin Ji, Xinghong Ling, and Quan Liu. A comprehensive review of multi-agent reinforcement learning in video games. IEEE Transactions on Games, 2025.
Yue Liu, Hongcheng Gao, Shengfang Zhai, Yufei He, Jun Xia, Zhengyu Hu, Yulin Chen, Xihong Yang. Jiaheng Zhang, Stan Z Li, et al. Guardreasoner: Towards reasoning-basee Ilm safeguards. arXiv preprini arXiv:2501. 18492, 2025a.
Yue Liu, Jiaying Wu, Yufei He, Hongcheng Gao, Hongyu Chen, Baolong Bi, Ruihan Gong, Jiaheng Zhang, Zhiqi Huang, and Bryan Hooi. Efficient inference for large reasoning models: A survey. arXiv preprint arXiv:2503.23077, 2025b.
Pattie Maes. Modeling adaptive autonomous agents. Artificial life, 1(1-2):135-162 1993.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
Alexander Novikov, Ngån Vü, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025.
Charles Packer, Vivian Fang. Shishir.C G Patil, Kevin Lin, Sarah Wooders, and Joseph_E Gonzalez. Memgpt: Towards lims as operating systems. 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances Neural Information Processing Systems, 36:8634 865623366666,,,66 2023.
Yuan Sui, Yufei He, Zifeng Ding, and Bryan Hooi. Can knowledge graphs make large language models more trustworthy? an empirical study over open-ended question answering. arXiv preprint arXiv:2410.08085, 2024a.
Yuan Sui, Yufei He, Nian Liu, Xiaoxin He, Kun Wang, and Bryan Hooi. Fidelis: Faithful reasoning in large language model for knowledge graph question answering. arXiv preprint 138.332405.13873. 2024b.
Yuan Sui, Yufei He. Tri Cao, Simeng Han, Yulin Chen, and Bryan Hooi. Meta-reasoner: Dy- namic guidance for optimized inference-tine reasoning in large language models. arXiv preprint arXiv:2502. 19918, 2025.
13

---

Published as a conference paper at ICLR 2026
Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Hamessing the power of intelligent Ilm agents. arXiv preprint arXiv:2306.03314 2023.
Joannes Vermorel and Mehryar Mohri. Multi-armed bandit algorithms and empirical evaluation. In European conference machine learning, pp. 437 448.. Springer, 2005.
Cheng Wang, Yue Liu, Baolong Bi. Duzhen Zhang, Zhong-Zhi Li, Yingwei Ma, Yufei He, Shengju Yu, Xinfeng Li, Junfeng Fang, et al. Safety in large reasoning models: survey, arXiv preprint arXiv:2504. 17704, 2025.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential large language model based agents: A survey. Science China Information Sciences, 68(2):121101 2025.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2023.
Xianglin Yang, Yufei He, Shuo Ji, Bryan Hooi, and Jin Song Dong. Zombie agents: Persistent trol of self-evolving Ilm agents via self-reinforcing injections. arXiv preprint arXiv:2602.1550 2026.
Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang. Evoa- gent: Towards automatic multi-agent generation via evolutionary algorithms. arXiv preprint wrX1::2406.14228 14228, 2024.
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation" via text. arXiv preprint 2400:2400..77466 2024.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19724-19731, 2024.
Han Zhou, Xingchen Wan, Ruoxi Sun, Hamid Palangi, Shariq lqbal, Ivan Vulić, Anna Korhonen, and Sercan Ô Ank. Multi-agen design: Optimizing agents with better prompts and topologies. arXiv preprint 200.00550... 2025.
Shuyan Zhou, Frank F Xu. Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307. 13554, 2023.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The eleventh interna- tional conference on learning representations, 2022.
Zhenhao Zhu, Yue Liu, Yampei Guo, Wenjie Qu, Cancan Chen, Yufei He, Yibo Li, Yulin Chen, Tianyi Wu, Huiying Xu, et al. Guardreasoner--omii: A reasoning-based multi-modal guardrail for text, image, and video. arXiv preprint arXiv:2602.03322, 2026.
CONTENTS
1 Introduction
Related Work
3 The Jericho Test- Time Learning (J-TTL) Benchmark
14

---

Published as a conference paper at ICLR 2026
4
The EvoTTs Framework
4.1
The Agentic Configuration
4.2
The Two-Agent Learning Loop
4.3
Configuration Selection via UCB
5
Experiments
5.1
Setup
5.2
Results
5.3
Model Analysis
6
Conclusion
10
Appendix
16
Limitations
16
Use of Large Language Models for Language Polishing
17
C Implementation Details
17
C.I Datasets and Environment
17
C.2 Hardware
17
C,3 Reproducibility
17
Baselines.
18
Computational Complexity Analysis
20
F Detailed Multi-Seed Experimental Results
22
G The Test-Time Learning Problem: A Detailed Analysis
22
G.1 The Task: Learning and Mastery in Text-Adventure Games
22
G.2 A Concrete Failure Case: The Navigation Loop
23
G.3 The EvoTest Solution
23
H The Evolver Agent's Master Prompt
23
I Case Study: Learning to Navigate the Library
25
J The Memory Component in Practice: Concrete Examples
26
J.1 Success Memory: Building Database of What Works
26
J.2 Failure Memory: Identifying and Pruning Unproductive Actions
31
K Evolution of Agent Behavior: Key Transcript Snapshots from Detective
32
K.1 Episode 3: Early Exploration and Basic Mistakes
32
15

---

Published as a conference paper at ICLR 2026
K.1.1 Key Transcript Steps: Episode 3
32
K.2
Episode 22: Mid-Stage Execution a Learned Plan
33
K.2.1 Key Transcript Steps: Episode 22
34
K.3 Episode 49; Near-Perfect Execution and Refinement
34
K.3.1 Key Transcript Steps: Episode 49
35
L Experiment with Qwen3-32B Backbone
36
M Dynamic Hyperparameter Evolution: A Case Study
37
Scalability with More Capable Models
37
A
LIMITATIONS
Our work introduces novel evolutionary framework for test-time learning, but it is important to acknowledge the inherent trade-offs and limitations of this approach.
The Drawback of Not Modifying Neural Weights. Our framework's foundation is its gradient- free nature, which is key to its practicality and data efficiency at test time. This strategic decision, however, introduces fundamental trade-offs by constraining learning the symbolic level, rather than the parametric level of the model's weights. 1) Bounded by Pre-trained Capabilities: Our approach, like other LLM-based agents, assumes the backbone model possesses latent knowledge sufficient for the task. The central challenge we address is not the knowledge itself, but the devel- opment of an efficient test-time learning mechanism to surface and apply this knowledge the fly. EvoTest operates as a high-level orchestrator; it learns to discover and refine the existing capabilities of its backbone LLM. However, it cannot instill fundamentally new, low-level reasoning patterns that lie outside the model's pre-trained knowledge base. For instance, if a game required a form of complex spatial reasoning that the LLM has never mastered, no amount of prompt evolution or memory management could create this capability from scratch. Fine-tuning, in contrast, holds the long-term promise of teaching model truly novel skills. The performance of EvoTest there- fore capped by the inherent intelligence of its frozen backbone. Future work could explore hybrid approaches that use EvoTest for rapid, ppisode-to-episodd adaptation while employing very slow, background fine-tuning to gradually enhance the agent's core competencies over hundreds of ses- sions. 2) Posk-Specific Overfitting vs. Generalizable Skill Acquisition: The J-TTL benchmark and our method are designed for rapid improvement on a single task. The highly specific strategies and "verbal guardrails" that EvoTest learns (e.g., unlock door with key) are a form of strategic overfitting to the current task instance. This is highly effective for the TTL setting, but the learned knowledge is brittle and may not generalize to even minor variations in the task environment (e.g.. in different room). In contrast, traditional RL and fine-tuning, when applied across
if the key a diverse distribution of tasks, aim to learn more robust and generalizable policies. A key direction for future research is to investigate how the symbolic strategies evolved by Tvotest for specific tasks could be collected and abstracted to build a library of general-purpose skills.
Dependency on a Powerful Evolver Agent. Our framework decouples acting from adaptation, but in doing so, it creates a strong dependency on the reasoning capabilities of the Evolver LLM. As our ablation in Table 4 demonstrates, the quality of the evolutionary step is correlated with the power of the model performing the analysis. This means that the success of the entire system hinges on access to capable-and potentially expensive-"oppieee "optimizer" model.
Complexity of the Evolutionary Search Space. The agentic configuration
(p. M, h, u) forms
a vast and complex combinatorial search space. Our current framework employs a simple (1 + m) evolutionary strategy with UCB selection, which, while effective, is relatively simple search heuristic. It may be prone to converging on local optima, especially in more complex games. Future work could explore more sophisticated population-base evolutionary algorithms, quality- methods, or techniques from automated program synthesis to more effectively navigate this complex strategic landscape.
16

---

Published as a conference paper at ICLR 2026
B USE OF LARGE LANGUAGE MODELS FOR LANGUAGE POLISHING
In the preparation of this manuscript, we used Large Language Models (LLMs) as a writing assis- tance tool to enhance language, clarity, and readability. This usage strictly confined polishing text that was already drafted by the human authors.
Our process interactive. After writing the content, used LLMs with specific prompts to refine the text. These prompts included requests to "check for grammatical errors," "rephrase this sentence for clarity," "make this paragraph more concise," or "suggest alternative phrasing to improve flow."
All suggestions generated by the LLM were critically reviewed, and the human authors retained full editorial control, making all final decisions regarding the manuscript's content and wording. The LLMs were not used to generate any scientific ideas, experimental results, data analysis, or other core intellectual contributions of the paper. The role of the LLM was analogous to that of an advanced grammar and style checker, and all research and conclusions presented are entirely the work of the authors.
C IMPLEMENTATION DETAILS
C.I DATASETS AND ENVIRONMENT
Our evaluation is conducted six publicly available Interactive Fiction (IF) games from the Jericho suite (Hausknecht et al., 2020). These games were chosen to represent a di- verse range of puzzle structures and difficulties: Detective, Library, Zorkl, Zork3, Balances, and Temple.
Datasets:
using Jericho's standard Python API, Zrov.Env. For each game, a test-time learning session consists = 50 consecutive episodes. To ensure fair comparison, each episode is capped maximum of T = 110 interaction steps. The game environment is reset to its identical initial state after each episode, meaning any performance gains are solely attributable the agent's 's learning algorithm. All textual observations from the game are converted to lowercase before being processed by the agent. All methods, including our own and all baselines, operate under these identical environmental constraints.
Environment: All experiments
C.2 HARDWARE
There is significant difference in hardware requirements between the gradient-free and gradient- based methods evaluated.
Gradient-Free Methods: EvoTest and all non- weight-update baselines (e.g., Static, Memory, RAG, Reflexion, TextGrad, EvoPrompt) do not require specialized local hardware. Their learning steps are executed via API calls to external LLMs. As such, these methods can be run on a standard machine with a CPU and a stable internet connection.
Gradient-Based Methods: The weight-update methods, SFT (online) and GRPO (online). have substantial hardware demands. The online fine-tuning and policy gradient updates were performed on a dedicated cluster equipped with 4 NVIDIA H100 GPUs. This hardware is necessary to accommodate the model's weights, gradients, and optimizer states in memory and to complete the training step in a reasonable time frame.
C.3 REPRODUCIBILITY
We are committed to ensuring that our results are fully reproducible. To this end, we will make our code and experimental artifacts publicly available.
The complete source code for the EvoTest framework, the J-TTL benchmark setup, and all baseline implementations will be released at public GitHub repository upon publication.
Code Release:
The current code is available for review at http://iitub..ommfffhheehitt
Model and Environment Identifiers: We have specified the exact model identifiers used for all
roles and baselines (e.g., google e/ gemi ni -2. 5-.55-1lamm flash, opena
6). We
17

---

Published as a conference paper at ICLR 2026
will also provide the version of the Jericho library and specific game files used in experiments to prevent discrepancies arising from environment updates.
Configuration and Prompts: The initial configuration files, including the generic starting prompts for the Actor Agent and the detailed master prompts used to guide the Evolver Agent, will be included in the code release. This is essential for reproducing the evolutionary trajectory of the agents.
To facilitate detailed analysis and verification, we will release the com- plete logs from all our experimental runs. These logs will include the per-episode transcripts, the sequence of evolved configurations (prompts, hyperparameters, etc.), UCB selection scores, and final episode scores for every method every game.
Full Experimental Logs:
Seeds and Hyperparameters: All random seeds used for LLM sampling and environment ini- tialization will be provided. Additionally, all fixed hyperparameters, such as the UCB exploration constant 3 and the number of children m generated per evolution step, will be documented in the repository.
Algorithm EvoTest: Evolutionary Test-Time Learning
1: Input: Number episodes K, initial configuration x (1)
2: Initialize history of returns H <- 0
3: for = ,,....K do
4:
Acting Phase
5: Select configuration (*) for the current episode.
6: Actor Agent: Execute episode using x(() to get trajectory 7(0) and return R(e).
7:
Update statistics for (*); n=x((1) + n(x (o)) + 1, update ((x(())
8:
Evolution Phase
9:
Evolver Agent:
10:
Update Memory: ((++1) + UpdateMemory (actnmorryMMM (M(c),,
Generate child configurations C(++1) (Xi.... Xm) by applying evolutionary operators (prompt mutation, etc.) to (+) and incorporating M(++1).
12:
- Selection Phase
13: Select next configuration x (e+1) from ( U ((e+1) using the UCB rule (Eq. 5).
14:
end for
15:
Return: Sequence of episode returns (R( R(K)).
D BASELINES.
To rigorously evaluate the performance EvoTest on the J-TTL benchmark, we compare it against comprchensive suite of bascline methods. These baselines are organized into four distinct categories based on their underlying learning strategy: memory-bases methods, reflection-batt based methods, au- tomated prompt optimization methods, and weight-update methods. We also include non-learning static agent to establish zero-shot performance floor. For all comparisons, methods are allocated the same step budget per episode and, where applicable, utilize the same backbone LLMs to ensure a fair and controlled evaluation environment.
Non-Leanning Baseline.
Static: This agent serves as the fundamental, non-learning baseline. operates with a single, fixed intggurraioonnccuudi a generic, hand-crafted prompt (e.g., "Explore the environment and try to score points.") and default hyperparameters- the entire duration of the test session. It performs updates between episodes and has mechanism for cross-episode memory. Its purpose is to measure the zero-shot performance of the backbone LLM on the task and establish a reference point against which all learning- baaedd based improvements its score across episodes is attributable solely to the inherent stochasticity the LLM's generation process.
be quantified. Any variation in
18

---

Published as a conference paper at ICLR 2026
Memory-based Methods. These methods learn by accumulating and accessing information from past interactions. However, they do not modify the agent's 's core strategic prompt or its decision- making logic.
Memory (Full History): This method attempts to learn by providing the agent with maximum historical context. After each episode, the complete transcript of that episode is appended growing history of all previous transcripts in the session. This full session history is then placed into the LLM's context window for the subsequent episode. The primary learning mechanism is in-context learning, where the LLM is expected to identify patterns and successful strategies from the raw text of past attempts. Its main limitation is the finite context window of the L.LM; when the session history exceeds the context limit, the oldest parts of the history are automatically truncated.
Generation): This agent enhances its decision-making by actively retrieving relevant information from past experiences. All trajectories from the current session are stored in a vector database. At each step within an episode, the agent's current observation is used to query this database to find the most similar relevant snippets from past trajectories. These retrieved snippets, which may contain successful or failed state-action sequences from similar situations, are then injected into the prompt additional context. This allows the agent dynam- ically access pertinent past knowledge without being constrained by fixed context window. The base prompt and hyperparameters, however, remain static throughout the session.
RAG
Reflection-based Methods. This category includes methods that use an LLM to generate high-level textual analyses of past performance, which are then used to guide future behavior.
Summary: This agent learns by creating a condensed narrative of its experiences. After each episode, an LLM is prompted to progressively summarize the entire history of all past transcripts. This summary is updated after every episode to incorporate the latest attempt, creating a concise, high-level overview of the session's progress, including key discoveries and persistent challenges. This condensed summary is then prepended to the agent's ' prompt for the next episode, aiming to provide strategic context without consuming the entire context window with raw transcripts.
Reflexion (Shinn et al., 2023): prominent "verbal reinforcement learning" baseline. After each episode concludes, the agent reflects on its performance by analyzing the trajectory transcript. It generates a structured self-reflection that identifies specific failures, hypothesizes their root causes, and formulates explicit, actionable plan to avoid those mistakes in the future (e.g.. "I got stuck in the kitchen because I kept trying to 'open the locked pantry'. In the next attempt, must first 'indd the pantry key" in the living room."). This textual reflection is then added to the agent's prompt, accumulating over episodes to build a rich, strategy-focused memory that directly informs future decision-making.
Automated Prompt Optimization Methods. These methods focus on iteratively refining the agent's 's core policy by directly modifying its guiding system prompt.
TextGrad (Yuksekgonul et al., 2024): This method is adapted for our test-time learning setting by treating the prompt as set of "textual parameters" to be optimized. After each episode, the trajectory and the prompt that generated it are passed to a separate "optimizer" LLM. This optimizer generates a "textual gradient" short, critical analysis describing flaw in the prompt and suggesting a direction for improvement. A subsequent LLM call then "applies" this gradient by editing the original prompt based the critique. This creates refined prompt for the next episode, directly evolving the agent's 's high-level strategy.
Promptbreeder (Fernando et al., 2023) & EvoPrompt (Guo et al., 2024): These two methods are adapted from their original formulations to our sequential, single-session setting. Both employ an cvolutionary algorithm to optimize a population of prompts. The process begins with a set of initial seed prompts. For each episode, a prompt is selected from the population, and its performance is evaluated based on the final episode score, which serves as its "fitness." After the episode, this fitness score is used to guide evolutionary operations. High-performing prompts are selected for "mutation" (where LLM makes small modifications to the prompt) and "crossover" (where an LLM combines two successful prompts). The prompt for the next episode is then selected from
19

---

Published as a conference paper at ICLR 2026
this newly evolved population. This creates competitive, population based search for the most effective guiding instruction.
Weight-Updare Methods (Online Fine- Tuning). In contrast to the gradient-free methods above, this category includes baselines that directly modify the weights of the backbone LLM via online fine-tuning. These methods represent the traditional approach to model adaptation.
SFT (online): This agent learns by imitating its own past behavior. After each episode, the trajectory is converted into dataset of (state, action) pairs. The backbone LLM is then fine-tuned on this dataset using a standard Supervised Fine-Tuning (SFT) objective. This update adjusts the model's weights to increase the likelihood of generating the actions it took in the previous episode, given the same states. This approach reinforces the entire trajectory, which can be effective for successful runs but risks strengthening poor decision madiiiin-making patterns from failed attempts.
GRPO (online): (Shao et al., 2024) This agent uses gradient-based Reinforcement Learning (RL) approach to update its policy. After each episode, the trajectory's 's state-action pairs and their associated rewards are used to compute a policy gradient. The model's weights are then updated to "reinforce" actions that led to positive rewards and suppress those that did not. This allows for more nuanced, reward- -eriddg credit assignment than SFT. However, its effectiveness highly dependent on the quality and density of the reward signal, and it is computationally intensive, requiring significant GPU resources for backpropagation through the model.
E COMPUTATIONAL COMPLEXITY ANALYSIS
In this section, we provide a more formal analysis of the computational complexity of EvoTest compared to the baselines, focusing the cost per test-time learning cycle (one episode of acting and one phase of learning). We define the following variables for our analysis:
K: The total number of episodes in session.
T: The maximum number of steps per episode.
C;: The context length (in tokens) provided the Actor LLM at step t.
La: The average length (in tokens) of a generated action.
Lo: The average length (in tokens) of an environment observation.
Tz. = T (L, + L.): The approximate total length of an episode transcript.
m: The number of child configurations generated by the Evolver in EvoTest.
Pactor: The number of parameters in the actor LLM.
Pevalver: The number of parameters in the evolver/optimizer LLM.
The hidden dimension of the actor LLM's transformer architecture.
We model the cost of LLM forward pass for generating Loui tokens from input of Li tokens as CostLIM(Lin. Low).. This cost primarily dependent on the model size and the total number of tokens processed.
Complexity of EvoTest, The cost of a single EvoTest cycle can be decomposed into the Acting Phase and the Evolution Phase.
1. Acting Phase: In each episode, the Actor Agent takes T steps. At each step t, it queries the backbone LLM.
Cost
CostLIM Ia) so T COStLLM(C. Ia)
where Ö is the average context length. This cost is dominated by T forward passes through the actor model.
Evolution Phase: After the episode, the Evolver Agent performs single, large query generate new configurations. The input is the full episode transcript (Tz.).
CostE COStLIM(TI, Feconfig)
20

---

Published as a conference paper at ICLR 2026
where Leonfig is the length of the generated configuration text. The UCB update step is (()), which negligible compared to the LLM call.
The total cost for cycle of Evo is thus:
CostEvoTest T COstLIM La) + CostLIM(TL: Leanlig)
(8)
This cost is entirely composed of LLM forward passes, which can be served via APIs without requiring local GPU memory for gradients.
Complexity of Baselines.
Static/Memory: The cost is simply the acting phase, CostAa- These the most efficient but least effective methods.
Reflexion/EvoPromppe These methods have similar complexity profile to EvoTest. Their learn- ing phase also consists of single large LLM call that takes the transcript Tz as input to generate a reflection or new prompt. Their total cost is structurally identical to Equation 3.
Online RL (GRPO): This is where the complexity profile differs fundamentally. The cycle con- sists of an acting phase and a weight-update phase.
1. Acting Phase (RL): The cost is identical to other methods: Costact T COSILLM( La).
Weight-Update Phase (RL): This phase involves backpropagation to update the model weights. The computational cost of training step for transformer model is approximately proportional to the number parameters and the total sequence length processed. For an entire episode trajectory of length T, this cost is:
CostUpdate ((Pactor T)
This cost reflects the computation for a full forward and backward pass through the trajectory to compute gradients. More critically, this step has substantial hardware requirements. The GPU VRAM must be large enough to store:
Model Weights: ((Pactor
Gradients: ((Pactor)
Optimizer States (e.g., Adam): O(2 Pactor)
Activations: OTT d batch_siza
The memory for activations scales with the episode length T, making online fine-tuning long trajectories very demanding. The total cost for one cycle of online RL is:
COStRI. = T COstLIM La) + O(Pacto T)
(10)
Comparative Summary. As shown in Table 7, EvoTest's architecture trades the expensive, hardware-intensive backpropagation step of online RL. for an additional LLM forward pass. While a large LLM call is free, computationally cheaper than full fine-tuning pass and, most importantly, can be offloaded to an API. This obviates the need for specialized local hardware (high- VRAM GPUs) and makes EvoTest a more data-efficient and practical solution for the test-time learning paradigm.
Table 7: Complexity comparison of single learning cycle.
Aspect
EvoTest (Ours)
Online RL (GRPO)
Acting Cost
T. COSTLLM
T COSTLLM
Learning Cost
Cost Leontig)
O T)
Mechanism
Gradient-Free (Forward Pass)
Gradient-Baset Based (Backprop.)
Hardware Req.
CPU + Network
High-VRAM GPU
Scalability Driver
API Latency
GPU Compute & Memory
21

---

Published as a conference paper at ICLR 2026
F DETAILED MULTI-SEED EXPERIMENTAL RESULTS
This section provides the detailed multi- results mentioned in our response to the reviewers. The table 8 reports the mean Area Under the Curve (AUC) and standard deviation over 5 random seeds for all methods using the googl e/ e/gemin cimmini-- -2. 5- flash backbone, validating the robustness of our findings.
Table 8: Detailed comparison of Area Under the Curve (AUC) scores on the J-TTL benchmark, showing mean std over 5 seeds for google/ / ieemini--i -2. 5- flash backbone. Higher values indicate better overall performance. The best performance in each column is highlighted in bold. These results robustly confirm that Test consistently outperforms all baselines.
Method
Detective
Library
Zorkl
Zork3
Balances
Temple
Avg.
Non-learning Raseline
Static
0.22 + .22 0.14 01 0.03 + .01
0.05 + .01
0.11
.02
0.08 + .01
0.11 + .01
Memory-based & Reflection-based Methods
Memory
0.56 + 02 0.19 + .02 0.04 + .01
0.06 + .20
0.14
+ .02
0.11
+ .01
0.13
+ .01
RAG
0.35 1 003
0.25 1 003 0.07 + .02
0.10
1
022
0.19
1
-03
0.16
.00
0.18
1 .02
Summary
0.46 + 04
0.34
.03 0.11
1 .02
0.16
1
.03
0.26
1
-03
0.22
1
003
0.26
1 .03
Reflexion 0.59 .05 0.42 .44 0.09 1 .22
0.26
.0
0.31
00
0.30
1
0.33
1 .03
Automated Prompt Optimization Methods
TextGrad
0.62 .05 0.46
0.12 .03
0.29 4 .04
0.17
.03
0.24
.03
0.32 + .03
Prompthreeder
0.64 + 04
0.48 + 04
0.11
02
0.30
+
.04
0.24
003
0.31
+ 003
0.35
+ .03
EvoPrompt
0.64 + 04 0.48 + .03 0.13 + .02
0.31
+ .03
0.25
-
.0
0.28
= 00
0.35 + .03
Weight- Update Methods (Ouline Fine-Tuning)
SET (online)
0.41 + .06
0..1 + .05
0.11 + -04
0.19 + -04
0.23 + 005
1.20+.000
0.24 -04
GRPO (online)
0.58 + 005
0.39 .04
0.08 + .03
0.23 + .04
0.31
+ 004 0.25 - .03
0.31 - .03
Test (Ours)
0.93 + .02 0.78 + .03
0.15 + .02
0.36 - 020
0.33 + .02
0.32 + 02
0.48 + .01
G THE TEST-TIME LEARNING PROBLEM: A DETAILED ANALYSIS
This section provides more detailed analysis of the core research problem addressed by our J- TTL benchmark, illustrating why traditional learning paradigms fail and motivating the design of EvoTest.
G.1 THE TASK: LEARNING AND MASTERY IN TEXT-ADVENTURE GAMES
The task given to the agent in our J-TTL benchmark is designed to be challenging test of on-the-fly learning and adaptation.
The agent interacts with a classic text-based adventure game from the Jericho suite (e.g., Detective, Zork/). In these games, the entire world is described through text. The agent receives textual observations (e.g., "You are in the Chief's s office. A piece of white paper is on the desk. ") and must issue text commands (c.g., TAKE PAPER) to act. These games are notoriously difficult for Al due to complex puzzles, long-range planning dependencies, sparse rewards, and irreversible consequences.
The Environment.
The Goal. The agent's 's goal is structured at two levels:
The Episodic Goal (Maximize Score): Within a single playthrough (one "episode"), the agent's objective is to take actions that maximize its final score, which is awarded for discovering areas, solving puzzles, and advancing the plot.
The Learning Goal (Improve Across Episodes): The ultimate task is to play the same game repeatedly and demonstrate learning by progressively increasing its final score from episode to the next. The agent must use its experience from failed or suboptimal attempts to build better strategy for subsequent attempts.
22

---

Published as a conference paper at ICLR 2026
G.2 A CONCRETE FAILURE CASE: THE NAVIGATION LOOP
To illustrate why this task is hard, consider agent attempting play Detective. It might correctly execute GO WEST, but from the next location, it gets stuck by repeatedly attempting GO WEST again, an invalid move the game rejects with "You can't go that way. This simple failure highlights the limitations of existing methods.
The Static Agent. A non-learning agent with fixed, generic prompt has mechanism to correct this error. It will likely repeat the same mistake every episode, resulting in flat, low-scoring performance curve. It cannot adapt.
The Online SFT Agent. A more sophisticated agent might use Supervised Fine- Fine-Tuning (SFT) on its prior trajectory. Our SFT baseline intelligently filters for "positive" actions (those that yielded a score increase). This approach fails for two reasons:
where the agent got stuck, it generated very few, if any, positive actions. The dataset for fine-tuning is therefore either empty or extremely small. With good data to learn from, the agent cannot improve.
the low-scoring episode
Many critical actions in these games are "neutral" and provide no immediate reward (e.g., UNLOCK DOOR WITH KЕY). An SFT agent that only trains on score-increasing actions will never learn these essential intermediate steps, rendering it incapable of solving complex puzzles.
The Online RL Agent. A Reinforcement Learning (RL) agent receives a reward of "0" for the invalid GO WEST action. In a sparse-reward environment like Jericho, this signal is incredibly weak and ambiguous. It is indistinguishable from the 'reward=0" received for neutral but necessary action. A single gradient update based on this noisy signal is insufficient to meaningfully correct the agent's policy for the next attempt, demonstrating failure of credit assignment in low-data regime.
G.3 THE EVOTEST SOLUTION
EvoTest is designed overcome these failures. Its Evolver Agent analyzes the entire episode just
transcript, not just scalar rewards or positive actions.
It Learns from Failure: Unlike SFT, EvoTest learns most effectively from failures. The Evolver semantically identifies the unproductive loop by reading the game's 's textual feedback ("You can't t go that way. ") paired with the repeated action and recognizes it as problem to be solved.
Performs Whsle-System Evolution: Based on its analysis, the Evolver directly rewrites the agent's prompt, generating targeted, structural edit correct the error (e.g., adding new rule: Step 5: From the steet,, GO EAST to enter the Mayor's s house.). This far more direct and data-efficient learning mechanism than small, gradient-based weight up- date.
H THE EVOLVER AGENT'S MASTER PROMPT
The core of EvoTest's learning capability resides in the Evolver Agent, which is guided by com- prehensive "master prompt." This prompt structures the analysis of a completed episode transcript, enabling the Evolver's LLM to perform holistic, multi-faceted updates across the entire agentic configuration X (p, M, h, u). Unlike simpler approaches that only modify the policy prompt, master prompt instructs the Evolver to act full-system optimizer, proposing changes to the agent's 's high-level strategy, its structured memory, its low-level decision- making parameters, and its internal tool-use logic.
The prompt is divided into four distinct parts, each targeting a specific component of the agentic system.
23

---

Published as a conference paper at ICLR 2026
EvoTest Master Prompt: Preamble and Context
You are an At agent eysten eptinizer, Your tack is to anslyze the transcript of improved configuration for
text-adventore seseion ageat that play the next episade,
game. The goal help agent higher
agent's configuration
componants:
guiding prompt agent's a high-Level strategyl.
Nonocy updatas (structured data for succass/failurs databassi.. ryperparametore llike temperatere, ror sacision-making). Tool-use routines (Python
asstraction
roles
You will receive the previous guiding prompt and the rull game history, Generete a new, complete contiguration by tollowing the tour parts below.
agent used tellewing guiding prompt may accurace; it 12 needed): "(car_pzonpt]"
history cane session:
GANE B1STORY START
icur_hiese hww._histttyy.ccc
BISTORY inogatave_cection;
24

---

Published as a conference paper at ICLR 2026
EvoTest Master Prompt: Detailed Generation Instructions
2XRT 1: Generate new improved guiding prompt.
This prorpt the agent's high-level policy. Stracture clearly.
Create "Mackthzough" oz "sssertial actions" section. isentsty actions tzon the hietory tast led to seare increases or were strictly necessery for game progression, Synthesize these into cleat, step-py-steg plan, he "unlosk
with actica phrasing Aeoid" section. Identicy actions stack, caused loops, produced rapeated crrera, or waze clearly unproduccave. Sist thess as negative constroince oz "quardreils." has Broinstorm passible not tully interacted with, sugrest a systemetic approach for the agent te follov once the known welkthrough is complete (e:g., "visit every roon, eystenaticelly EKAMINE, SEARCE,
key" instaad of rase
getcing
"Exploration Plan" section, roans or objects that have
steps.
like 2058, PULL, READ
cbjects."
2AR7 2: Generate nemosy updates for the database.
Analyze the tronscript and extract all state-action pairs that resulted in score will added
J5W 1ist objects.
aceatra auccess menary.
The "state_text' should be the tel: abservation text right bafore the action xas taken, Ine 'action" is the comnand the agent gave. "scora_delta"
ao new scoze increases Mere fourd, return empty '[]".
Zxample:
[("stato_texxt; thee stlayeu [[Matate_texi"; living roan so'nyou are standing ...,, Mastion": "geat wood*, "acore_delca's 10)3
...,, "acctonn: "take pistol", "scoro_dcitaa:
]
Buggest hyperparanele: mejas Analyze agent's Oukgutt a J5CN abject.
tune
balance.
If the agent was stuck in repetitive loops seened overly cautious, suggest increasing If the egent's sctions seemed cheolie, candon, or devisted foon guod exlating plan, auggest decreasing the comparature to promote adharence to the scracedy (e.a., ""Eenperature": 0.2").
diverse actions (e.g., ""comparatuee": 0.9%).
folloved the plan suggesi changes by seturning an enpty J20N cbject '(I". Example: 'Il "tenperature"" 0.75 }1"
If agent's behavior
intinal
2ART
46. State Ext ractor Code: Generate Pyshan code block for funct oon named "ssiiact_atale This function Lakea the full "gans_histoey" string as impas and seluria short aumaary of key nilestones completed.
presence specific rstrings ageat's actions.
login
gane's ousputa,,
Keep the code simple to avoid baes, Use "1E/ellt/else" with Nin" checks. Do not Include comeenty Inside the function
4b. Menory Interact "ogies Progose a single sentence be added the nain guiding proupt that refinen how the agent should use its nencry. This logie should evelve to becone noze diaciplined ovez cime.
Tarly-stag age suggestion: ORist: in faniliat lar aituatione."
your nenory past
Kid-stage suggest lons ""Surategy: In any soon you' visited before, consult your success nenory for proven act fons."
stage cules "Bules Before evesy for the cursent state and tollow ise suggestion it one exists."
you query your denory
Jurmal your reaponse as Iollows with additional Lext. The function name HUST be state".
[Feas generaled propl beze]
IYour JSCN for nenory updates herel
for hyperparaneter adjistmenss <oode>
def extract state Igane historyl:
[Return string summaixiig the current state]
|Your ore-sentence menory interaction logic herel
I CASE STUDY: LEARNING TO NAVIGATE THE LIBRARY
Analysis of EVOTEST's behavior in the library game reveals is performing form Verbal Reinforcement Learning. In this paradigm, the agent's natural language prompt is its policy, the full episode transcript is the rich reward signal, and policy updates are semantic edits to the prompt. The evolution of this policy is detailed in the prompts shown in Figures 4 through 8.
Episode The Credit Assignment Problem. The initial agent begins with a simple, high-level directive (Figure 4) and acquires the target biography, but gets stuck at the security alarm. It mis-
25

---

Published as a conference paper at ICLR 2026
attributes the failure, falling into loops of re-locking the rare books room, demonstrating classic credit assignment problem where a delayed, sparse negative signal is linked to the wrong proximate cause.
Episode 1-3: Positive Verbal Policy Update. The Evolver Agent analyzes the first transcript, iden- tifies the successful action sequence for retrieving the biography, and distills it into tic. This positive policy update is evident the updated prompt for Episode 1 (Figure 5) and its further refinement by Episode 3 (Figure 6). The Evolver performed credit assignment the score-increasing events and codified the successful trajectory into the agent's strategy, effectively abstracting a reusable skill from single experience.
heuris-
Episode 11: Latent Value Discovery. By now, the evolved prompt has discovered a non-obvious but critical action: immediate score increase but is prerequisite for solving later puzzle. This demonstrates la- tent value discovery. A traditional RL. agent would struggle to find such an unrewarded action. EVOTEST uses semantic reasoning on the narrative-"an NPC is blocking object; making them leave could useful"-to identify a state with high future value, bypassing the need for numerical value propagation.
TECHNICIAN ABOUT GATES (Figure 7). This command provides no
Episode Negative Policy Update and Guardrails. The agent's policy now a multi-stage plan. Crucially, it includes "Actions to Avoid" section (Figure 8), warning against interacting with the security gates because it "causes an endless clarification loop." This is a negative policy update. The system has learned not just what do, but what not to do, by identifying frustrating failure pattern (a parser bug) from the transcript. It created a "verbal guardrail" to prevent this specific failure, form of learning that is difficult to achieve with scalar rewards alone.
Prompt of Episode 0
Explore systematically and examine objects to make progress.
Figure 4: Case Study: Prompt of Episode 0.
J THE MEMORY COMPONENT IN PRACTICE: CONCRETE EXAMPLES
To illustrate precisely how the Evolver Agent constructs and utilizes memory, this section details the process using interactions from the Detective game. The memory is not monolithic block of text; is a structured database, programmatically populated by the Evolver after each episode.
J.1 SUCCESS MEMORY: BUILDING A DATABASE OF WHAT WORKS
The Success Memory functions as high-reward state-action lookup table. The Evolver parses the episode transcript for any step where the score increases and logs the preceding state and action.
For example, after a single episode, the Evolver identifies several score- ceerrincreasing events from the trajectory log.
26

---

Published as a conference paper at ICLR 2026
Prompt of Episode
Retrieve Graham Nelson biography efficiently: Step-by-step actions that proved essential so far 1. In lobby: talk to reference librarian first { west Ground-Floor Stacks, ASK LIBRARTAN ABOUT GRAHAM NELSON and then ASK LIBRARIAN ABOUT RARE.
2. Return to lobby and GIVE ID CARD TO ATTENDANT; you receive
the Rare-Books KEY.
3. Reach second-floor stacks (north, up, south) and UNLOCK RARE BOOKS DCOR WITH KEY, OPEN DOOR, ENTER.
4. TAKE BIOGRAPHY OF GRAHAM NELSON.
5. Leave the room, CLOSE RARE BOOKS DOOR, then LOCK RARE BOOKS
DOOR WITH KEY.
6. Back in lobby GIVE KEY TO ATTENDANT; he returns your ID card.
Avoid
Wandering in the dusty card cataloguu corner it contains no useful data. Trying to leave the library while still holding the key while the rare-books door is open/unlocked: this triggers alarms and extra trips.
Vague commands such as \talk to attendant"; instead use verbs ASK or GIVE.
Next places and ideas (not yet proven)
Northwest Duplicating Services room: may 1et you photocopy pages, reducing risk at security gate.
\Computer Room" east of 2F stacks. The \Private" door north of lobby.
Systematically examine new rooms, objects, signage and NPCs; look for legitimate ways to check out or copy the biography so the alarm will not sound when passing the security gates.
Figure 5: Case Study: Prompt of Episode
27

---

Published as a conference paper at ICLR 2026
Prompt of Episode 3
Goal: leave the library with the Biography of Graham Nel son legitimately checked-out.
Essential steps that advanced the story and/or scored points (use exactly these verbs { the game is picky):
1. From the Lobby ask attendant about book + he sends you to Reference
2. West to Ground-floor Stacks (Reference Desk) ask librarian about Graham Nelson ask librarian about rare she tells you key is at Circulation
3. Return to Lobby give id card to attendant you receive
key to rare books room (+5)
4. Reach the heavy wooden door (N + U 8) unlock door with
key open door enter take biography (+5)
5. Before leaving the second floor close rare books door
lock door with key
6. Back in the Lobby . give key to attendant + attendant
returns your
7. Still in the Lobby ask attendant to check out biography
(or \check out biography" / \borrow biography") east leave the building without the alarm
Actions that wasted tine or caused trouble avoid these: Re-examining the dusty card-catalog drawers (they contain unreadable parchment). Atalk attendant" the correct form is \ask attendant about ...".
Dropping the biography in the stacks: you must keep it and check it out.
Trying to exit before key is returned, door locked, and book checked-out { triggers alarms.
Where to explore next if something is still missing: Rooms not yet inspected in detail: Duplicating Services (NW from Lobby)
Private Room (N from Lobby)
Computer Room (E from 2nd-floor stacks)
Systematically: in every room LOOK, EXAMINE all obvious objects, ASK any NPCs about topics that appear in room descriptions, and INV and SCORE regularly to monitor progress.
Figure Case Study: Prompt of Episode
28

---

Published as a conference paper at ICLR 2026
Prompt of Episode 11
ESSENTIAL WALKTHROUGH (all steps verified by score increases)
1. Gain entry to Kare-Books ASK ATTENDANT ABOUT GRAHAM
WEST to Ground-Floor Stacks ASK LIBRARIAN ABOUT GRAHAM ASK LIBRARIAN ABOUT KEY EAST to Lobby . ASK ATTENDANT ABOUT KEY GIVE ID CARD тO ATTENDANT receive Rare-Books key (+5)
2. Retrieve the Graham Nelson biography WEST NORTH { UP
SOUTH to 2F Stacks UNlOCk DOOR WITH KEY; OPEN DOOR; ENTER TAKE BICGRAPHY
Return to 2F Stacks, CLOSE DOOR, LOCK DOOR WITH KEY NORTH DOWN { SOUTH EAST to Lobby GIVE KEY ATTENDANT (ID card returned)
Empty the Computer Lab and grab the encyclopedia WEST NORTH { UP { SOUTH EAST to Computer Lab ASK TECHNICIAN ABOUT GATES+ technician leaves (+2) EXAMINE PRINTERS- automatically take Encyclopedia Frobozzica
4. Please the Reference Librarian . WEST { NORTH { DOWN SOUTH EAST to Ground-Floor Stacks GIVE ENCYCLOPEDIA TO
LIBRARIAN (+2)
Confirmed score far: 19 points
ACTIONS тO AVOID Getting stuck in the \which do you mean, the howto manual or the security gates?" loop when it appears, simply type HOWTO MANUAL once and continue. Trying to open push / pry the \Private" north door Fussing with the burned-out copier after the explosion. on already-searched objects.
Repeating LOOK/EXAMINE
PLACES & IDEAS тO INVESTIGATE NEXT (unverified) Read the HOWTO MANUAL offered during the gate disambiquation; then SEARCH or LOOK IN the opened security gate Tom is fixing. Examine paintings in both stairwells and the damaged picture on the ground-floor stairs (MOVE, PUSH, PULL) Re-visit Duplicating Services after the smoke clears for anything the blast exposed. Scrutinise every object named in room texts (front doors, notices, posters, card-catalog drawers, etc.). ASK each NPC ABOUT any new topic you uncover.
Systematic approach: in every room LOOK, then EXAMINE and SEARCH every noun that appears. Try TAKE, OPEN, READ, PUSH, PULL, TURN and ASK <npe> ABOUT <noun/topic> to expose further clues.
Figure 7: Case Study: Prompt of Episode 11.
29

---

Published as a conference paper at ICLR 2026
Prompt of Episode 49
ESSENTIAL ACTION LIST { VERIFIED TO 24 POINTS
1.
Borrow the
key (+5) In the Lobby: GIVE ID CARD
TO ATTENDANT
2. Confirm the book's s location WEST to Ground-Floor Stacks
ASK LIBRARIAN ABOUT GRAHAM
3. Retrieve the Graham Nelson biography (+5) WEST [ NORTH
UP SOUTH to 2F Stacks UNLOCK DOOR WITH KEY ; OPEN DOOR ;
ENTER TAKE BIOGRAPHY NORTH (leave) CLOSE DOOR LOCK DOOR
WITH KEY
4. Return the key NORTH { DOWN { SOUTH EAST to Lobby GIVE ATTENDANT (get ID card back)
5. Clear the Computer Lab (+2 5) WEST NORTH { UP { SOUTH
EAST to Computer Lab ASK TECHNICTAN ABOUT GATES (+2)
EXAMINE PRINTERS (auto-take Encyclopedia Frobozzica, +5)
6. Donate the Encyclopedia (+2) WEST { DOWN SOUTH { EAST
WEST to Ground-Floor Stacks GIVE ENCYCLOPEDIA TO LIBRARIAN
7. Extra find in 2F Stacks (+5) NORTH { UP SOUTH to 2F
Stacks SEARCH SHELVES (get \Debt of Honor")
Confirmed score so far: 24 points.
Avoid: Interacting with the security gates (causes an endless clarification loop).
Repeatedly fiedling with the paintings/canvas until new clues appear. Non-standard verbs (USE, TYPE, TALK TO, etc.). Trying to leave the library while holding required books.
Unexplored areas worth checking once the above is complete (no points verified yet):
\Private" door in Lobby Duplicating Services room after explosion Stairwell paintings ('grue" canvas, damaged logo) Card-catalog drawers and miscellaneous shelving
After finishing the essentials, visit every room, LOOK, EXAMINE, SEARCH, and ASK NPCs about any noun you encounter, noting any score changes. Good luck!
Figure 8: Case Study: Prompt of Episode 49.
30

---

Published as a conference paper at ICLR 2026
Evolver Agent: Parsing Successes from Episode Log
The Evolver's parser identifies the following successful interactions:
1. STATE: '<< Chief's office >> You can see a piece of white paper...
ACTION: 'read paper"
REWARD: +10 points
2. STATE: '<< closet
there is gun on the floor..." ACTION: ' 'get pistol"
REWARD: +10 points
3. STATE: "<< living room you see battered piece wood...
ACTION: 'geet wood"
REWARD:
+10 points
Based on these observations, the Evolver programmatically updates the success memooryymmmoryy.. json file. This file stores a mapping from hash of the state's ' descriptive text to the action that proved successful. The resulting database entries would look like this:
Success Memory Database (success_memong jsor)
State Hash (Key)
Stored Action (Value)
Score Delta
hash ("<< Chief's office... hash ("<< closet >>...") hash ("<< living room >>...")
"read paper"
+10
"get pistol"
+10
"get wood"
+10
During the next episode, when the Actor Agent encounters the state << closet >>, its memory-access routine will hash the state description, find match in the database, and retrieve the action "get pistol". This information is then used to augment the prompt, for example: "Hint: In this exact situation before, the action 'get pistol worked well."
J.2 FAILURE MEMORY: IDENTIFYING AND PRUNING UNPRODUCTIVE ACTIONS
The Failure Memory's goal is to prevent the agent from repeating obvious mistakes, especially getting stuck in loops. The Evolver identifies these patterns by detecting sequences of actions that result in no change to the game state score.
From the provided logs, the Evolver can identify wasted move:
Evolver Agent: Parsing Non-Productive Action
The Evolver's analysis routine detects the following loop:
At Step t: Agent is in state << hallway >> you still in the hallway lallway....
Agent's Action: west.
At Step t+1: The environment returns the exact same state description:
hallway >> you are still in the hallway...
Analysis: The state did not change, and the score did not change. This action is flagged unproductive for this state.
31

---

Published as a conference paper at ICLR 2026
Unlike the Success Memory, this insight is not stored in database be queried. Instead, the Evolver uses it to directly mutate the agent's 's core policy- the prompt- by adding an explicit "verbal guardrail." This creates permanent and proactive change to the agent's strategy.
Policy Prompt Mutation: Creating a Verbal Guardrail
Original Prompt Section (Episode N):
Strategy Explore systematically.
Examine all objects.
Evolved Prompt Section (Episode N+1);
## Strategy
Explore systematically. Examine all objects.
Known Dead Ends / Wasted Actions
In "<< hallway >>", avoid the action 'west". It leads back to the same room.
This example shows how EvoTest moves beyond simple trial-and-error. It performs semantic credit assignment on the narrative the game, identifies specific unproductive behavior, and encodes rule to prevent it in the future. This form of learning, which prunes the search space by identifying and forbidding useless actions, is a key reason for its data efficiency compared to methods that rely solely scalar rewards.
K EVOLUTION OF AGENT BEHAVIOR: KEY TRANSCRIPT SNAPSHOTS FROM Detective
To demonstrate how the agent's behavior evolves over session, this section presents key mo- ments from the transcripts of early, mid-stage, and late-stage episodes for the Detective game. Each snapshot includes the agent's guiding prompt at that stage, highlighting the direct link between the evolved policy and the agent's actions.
K.1 EPISODE 3: EARLY EXPLORATION AND BASIC MISTAKES
In early episodes, the agent operates with a generic prompt and relies on broad exploration. Its behavior characterized by discovering simple, high-reward actions while also making fundamental errors, such getting stuck in loops.
Guiding Prompt: Episode 3
You detective playing text-based game. Your goal is solve the murder of the Mayor and maximize your score. Explore your surroundings, examine items, and talk to people to gather clues. Be systematic.
K.1.1 KEY TRANSCRIPT STEPS: EPISODE
The agent successfully discovers initial points but then gets stuck in a simple navigational loop, revealing lack of long-term strategy.
32

---

Published as a conference paper at ICLR 2026
Transeript Snapshot: Steps 2-9
[STEP 2]
[OBS] confi :en detective was created by matt barringer... [ACTION] west
[STEP 3]
[OBS] << closet >> there is gun on the floor. better get it. [ACTION] take pistol [REWARD] +10
[STEP 4]
[OBS] taken. [your score has just gone up by ten points. J [ACTION] east
[STEP 5]
[OBS] << chief's s office >> north or west.
[ACTION] north
[STEP 6]
[OBS] << outside >> to the east is dead end. to the west street.
[ACTION] west [REWARD] +10
[STEP 7]
[OBS] << outside east is home. [ACTTON] west
to the north is restraunt ... to the
(STEP 8]
[OBS] you can't go that way. << outside >> north is restraunt...
[ACTION] west
[STEP 9] [OBS] you can't t go that way. << outside >> north is restraunt...
[ACTION] look
Analysis: The agent successfully finds points by taking the pistol (Step 3) and exploring west (Step 6). However, at Step fails process the new information and tries go 'west' again, an invalid move. It repeats this mistake at Step 8, demonstrating a classic failure mode of early-stage agents: getting stuck in simple loop. This transcript provides clear signal for the Evolver create rule about how to navigate this specific "outside" area.
K.2 EPISODE 22: MID-STAGE EXECUTION LEARNED PLAN
By the middle of the session, the prompt has evolved into partial walkthrough. The agent can now flawlessly execute complex sequences it previously struggled with may fail on details not yet encoded in its strategy.
Guiding Prompt:
Episode 22
Goal: Get to the Holiday Inn. Known Path: 1. Office: 'read paper", "west", 'take pistol', 'cast", 'north' 2. Mayor's House: Go east living room for wood, then to dining room for note. Then go north to exit. 3. Vidco/Music Stores: Navigate past them to reach the alley. Avoid: Going west multiple times at the street hub.
33

---

Published as a conference paper at ICLR 2026
K.2.1 KEY TRANSCRIPT STEPS: EPISODE 22
The agent expertly navigates the Mayor's house but fails at crucial, un un-memorized interaction due incorrect action syntax.
Transcript Snapshot: Steps 8-13 and 30-32
[STEP 8]
the east is the living room [OBS] << mayor's house [ACTION] east
[STEP 9]
[OBS] << living room >> you see a battered piece of wood. [ACTION] take wooden wood [REWARD] +10
[STEP 10]
[OBS] taken. [your score has just gone up by ten points. [ACTION] west
[STEP 11]
[OBS] << mayor's s house [ACTION] west
the west is the dining room.
[STEP 12]
[OBS] << dining room >> see a note on the table.
[ACTION] take paper note [REWARD] +10
[STEP 13]
[OBS] taken. [your score has just gone up by ten points. ] [ACTION] east
[STEP 30]
[OBS] << back music store ... to the west there is
dazed looking man
[ACTTON] west
(STEP 31]
[OBS] << music store >> you need to kill him. best
chance: use your gun.
[ACTION] gun
[STEP 32]
[OBS] what do you want to use the gun on?
[ACTION] dazed man
The sequence from steps 8-13 shows perfect execution of a learned sub-plan, retrieving both items in the Mayor's house without error. This demonstrates successful memory and planning. The failure occurs later at Step 31. The prompt did not specify the exact syntax for using the gun, so the agent defaults to generic 'use gun' command, which leads to clarification question from the game and wastes a turn. The Evolver will parse this failure and refine the prompt to use the correct syntax: 'shoot dazed man with gun".
Analysis:
K.3 EPISODE 49:
EXECUTION AND REFINEMENT
In late-stage episodes, the agent possesses a nearly complete walkthrough. Its behavior is precise, confident, and focused on executing the optimal path.
34

---

Published as a conference paper at ICLR 2026
Guiding Prompt: Episode 49 (abridged)
Follow this exact sequence to score 330 360 and win... 8. Holiday Inn capture north - elevator to 15th floor (+10) west, north, west, north, north - Room 30 enter room + automatic gun-fight, arrest killer (+100) up - "info" room - *** YOU HAVE WON
K.3.1 KEY TRANSCRIPT STEPS: EPISODE 49
The agent confidently executes the final, complex maze-like sequence of the game without a single misstep.
Transcript Snapshot: Steps 43-49
(STEP 43]
[OBS] << holiday inn >> you have to find it. to get started, go north. [ACTION] north [REWARD] +10
[STEP 44]
[OBS] << holiday inn 15th floor >> you can go east or west. [ACTION] west
[REWARD] +10
[STEP 45]
[OBS] << hal hallway >> you can only go north. [ACTION] north
[REWARD] +10
[STEP 46]
[OBS] << hallway >> ... you go west east.
[ACTION] west
[REWARD] +10
[STEP 47]
[OBS] << hallway >> ... you are getting close! you can only go north.
[ACTION] north
[REWARD] +10
[STEP 48] [OBS] << hallway >> ... to the north is hall. [ACTION] north [REWARD] +100
[STEP 49]
[OBS] << room 30 >> after a harrowing gun battle you conk him on the head...
to leave this darned game, up. [ACTION] up
This transcript shows the agent in its most effective state. The sequence 'west, north, west, north, north' (Steps 44-48) is executed perfectly, demonstrating that the agent no longer exploring but is following precise, validated plan. Each action leads directly to progress. This flawless execution of a complex, non-obvious path is a hallmark of the converged EvoTest agent and highlights the power of evolving detailed, procedural policy.
Analysis:
35

---

Published as a conference paper at ICLR 2026
Table 9:
Comparison of AUC scores on subset of games where all methods use qwen/ qwen3-322 as the actor backbone LLM. EvoTest continues to outperform both weight- update baselines, highlighting the robustness of the evolutionary learning algorithm.
Method
Detective
Zorkl
Balances
Avg.
SFT (online)
0.40
0.11
0.22
0.24
GRPO (online)
0.55
0.07
0.30
0.31
EvoTes (Ours)
0.78
0.12
0.31
0.40
Evolution of LLM Decoding Temperature vs. Performance
350
300
12
Final Score per Eptratee
Frai Score
Decoding Temperature
100
0.4
30
02
0.0
Episode Index
Figure 9: Evolution of the Actor's LLM temperature versus its performance on the Detective game.
L EXPERIMENT WITH QWEN3-32B BACKBONE
To provide more direct and controlled comparison against the weight-update baselines, we con- ducted an additional experiment using gwen/qwen3-tt /////-wwn333 as the backbone LLM for all com- pared methods. In main experiments, the weight-update methods (SFT and GRPO) use swen/qwents mwen//wee33322. while EvoTest uses proprietary models. This supplementary experiment aims to normalize the underlying model capabilities to better isolate the effectiveness of the learning algorithms themselves.
In this setup, the Actor Agent in the EvoTest framework google/ wew.eemii...--111 use spenai/o3 1atpaaaioo22225--22-21 2022204222222 EvoTest's performance reflects its architectural strengths rather than being limited by a weaker optimizer. The results, presented in Table 9, compare EvoTest against SFT and GRPO on rep- resentative subset of games, with all three methods now relying on the same qwen 3wtnn//wnn332 backbone for their acting component.
switched from to 3ww32 The Evolver Agent continued to maintain its strong analytical capabilities, ensuring that
The results confirm that Evo performance advantage holds even in this controlled setting. As shown in Table EvoTest surpasses both SFT (online) and GRPO (online) on all three tested games. The average AUC score for EvoTest (0.40) is substantially higher than GRPO (0.31) 1) and SFT (0.24). This indicates that the superiority of our evolutionary approach not merely an artifact of using more powerful backbone model in the main experiments. Instead, it underscores the fundamental data efficiency of EvoTest's 's learning mechanism. By leveraging rich, narrative feedback from the entire episode transcript for whole-system evolution, EvoTest make significant and tar- geted improvements from a single experience than gradient-basee methods relying on sparse scalar rewards.
36

---

Published as a conference paper at ICLR 2026
Comparison of AUC scores on subset of games different LLM backbones. While stronger models moderately improve the Static agent's performance, the benefit is significantly greater for EvoTest, demonstrating the scalability of learning framework.
Table 10:
Method
Backbone LLM
Detective
Zork1
Balances
Avg.
Static
gemini-2.5
0.21
0.03
0.11
0.12
EvoTest
1omini-2.5-flas
0.94
0.14
0.32
0.47
Static
claude-4-sonnet
0.23
0.04
0.12
0.13
EvoTest
-laude-4-sonne
0.95
0.16
0.35
0.49
Static
gemini-3-pro-preva.
0.25
0.05
0.14
0.15
EvoTest
gemini-3-pro-previie
0.96
0.19
0.40
0.52
Static
gpt-5.1
0.28
0.06
0.17
0.17
EvoTest
gpt-5.1
0.98
0.24
0.48
0.57
M DYNAMIC HYPERPARAMETER EVOLUTION: A CASE STUDY
Figure illustrates how the Evolver intelligently manages the Actor's LLM decoding temperature in response to its performance. The Evolver's 's logic is not random; it follows an adaptive strategy to balance exploration and exploitation.
N SCALABILITY WITH MORE CAPABLE MODELS
To investigate how our framework scales with model improvements, evaluated EvoTest and the non-learming Static baseline using two next-generation models: openai/gpt-5.. and google gemi ni-3-pro-previ ew. The results, presented in Table 10, show positive trend: as the underlying model's capability improves, the performance of both agents increases. As models become more capable, EvoTest has more raw material to work with, leading to disproportionately larger gains compared to simple zero-shot approach.
37


Reviews:
[{"review_id": "cQmWgEV8aD", "reviewer": "Reviewer_m2c3", "summary": "The paper presents Jericho Test-Time Learning (J-TTL), a few-shot benchmark based on Jericho, and EvoTest, an LLM-based test-time learning framework for J-TTL.\n\nEvoTest comprises of two LLM agents, an actor and an evolver. An actor agent interacts with the Jericho environment given its configuration. An evolver agent proposes new configurations based on the interaction history. An actor agent's configuration includes prompts, interaction memories, LLM inference hyperparameters, and game-specific tools.\n\nThe evaluation results on J-TTL demonstrate that EvoTest improves over the non-learning baseline and other self-improvement baselines.", "questions": "1. In Section 4.3, do you discard all the options that were not selected? Wouldn't that make $\\chi^{(e)}$ the only candidate with $n(\\chi)>1$? Or will the evolver agent generate repeated child configurations?\n2. How is the initial configuration determined? Do you use a generic one for all games or do you use specific ones for different games? Does the initial configuration affect the final performance?\n\nMinor issues / Typos\n\n- Eq 5, fix the argmax and define $n$ and $N$.\n- In multiple places, the quotation marks aren't correctly used (``` `` ``` and `''`)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper reads well with a clear logic flow. The concept of few-shot learning and test-time learning is not new, but the proposed LLM-based method in a text-based environment is novel to my knowledge.\n- The experiments are thorough with a good variety of baselines and key ablations. Empirically, EvoTest seems to be meaningfully better than previous methods.\n- Code and prompts with examples and interaction logs are provided for reproducibility.", "weaknesses": "My main concern is the potentially limited scope. The evaluations are limited to a subset of Jericho games. This raises questions of 1) whether the set of games is cherry-picked, and 2) whether the method is over-engineered to work specifically on Jericho games. Despite that EvoTest is empirically superior to previous baselines, it is compared with methods with a broader scope.\n\nBranching off of this, from the ablation (Table 3), it seems that prompt optimization and the evolution selection strategy (UCB) are the two major factors for the improvement. I would like to see how much the specific prompt-evolving prompt from the evolver contributes to this. This should be verifiable by transplanting the prompt-evolving part in the evolver master prompt to other prompt optimization methods, such as EvoPrompt.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L7alIugr5R", "reviewer": "Reviewer_QWGa", "summary": "The paper introduces EvoTest, an evolutionary framework designed to solve the problem of AI agents being unable to learn on the fly once deployed. EvoTest uses a gradient-free, two-agent system: an Actor agent attempts a task, and an Evolver agent analyzes its transcript. Based on this analysis, the Evolver evolves the entire agentic system for the next episode by:\n* rewriting its prompt, \n* updating its memory, \n* tuning hyperparameters, \n* refining its tool-use routines. \n\nOn the newly introduced J-TTL benchmark, EvoTest demonstrated superior performance, significantly outperforming reflection-based and memory-only baselines and proving capable of solving tasks that all other methods failed.", "questions": "The perfect writing quality of this paper leads me to not having any question, everything is clearly defined.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper addresses a novel problem. With the emergence of LLM as agents, new problems appear, that include allowing such agents to quickly adapt to tasks unseen during training. \n\nSolving a problem is a core task that many industrial actors are currently trying to tackle.\n\nThe authors insist on their gradient free approach, which indeed does not require retraining/finetuning of the weights.\n\nThe benchmark is well motivated and established.\n\nThe result appear strong, and the ablation study provides intuition to the reader on the relevance of the difference components, even if second order results would allow to distinguish the true performance gaps of the compared methods.", "weaknesses": "My main concern is the relevance of the paper for the readers of ICLR. While clearly framed as a scientific paper (with e.g. the outlined list of contributions, of research questions), the paper still feels like an engineering paper in the sense that it primarily aims at pushing up a metric. A clear description of the scientific problem in the introduction could help change this impression, given to the reader. I am not sure how learning representations are here (even implicitly) discussed.\n\nMy second concern is the lack of introductory examples (can be done in one example) of:\n1. The addressed research problem, show how current agent fail at addressing adapting to one of the existing games.\n2. the task given to the agent. It would greatly help understanding if the (current) Figure 1 could incorporate an example of the tasks provided in the Benchmark. Even if these have been introduced in an existing paper, it would make this work more comprehensible on its own.\n\nThis can be done by referencing content placed in the appendix, if space is limited, but is important to help the reader understand the problem, and assert that it is relevant.\n\nNo limitations discussion. I am quite sure that a discussion could be made on e.g. the drawback of not letting the agent learn (in the primal sens of modifying its neural weights).\n\nLack of second order reported metrics (mean +/- std)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YM7nOtexoe", "reviewer": "Reviewer_S8oG", "summary": "The work introduces a benchmark and a framework focusing on test-time learning. The new benchmark is created from an existing set of text-based games, where the capacity of improvement during learning is considered. The proposed framework, EvoTest, consists of an actor and an evolver that changes the actor's learning configuration by altering the policy prompt, memory, hyperparameters, and available tools.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A. The work investigates an important problem of decision-making, which is online learning for fast adaptation.\n\nB. The content of the submission is clearly presented, and the reading flow is good.", "weaknesses": "I. The work contains some hidden assumptions and unjustified claims:\n\n   a. The proposed framework relies on the assumption that the LLM backbone contains sufficient information for solving the task of interest. While this assumption is realistic, it is never clearly stated.\n\n   b. The comparison with fine-tuning methods is incomplete as the performance of EvoTest with qwen/qwen3-32b for the actor and the evolver is never reported. This is the only way to make a fair comparison.\n\n   c. The work is missing a limitation section.\n\n   d. There seems to be a hidden assumption on credit assignment because in Line 258, it is stated: \"It records state-action pairs ($o_t, a_t$) that immediately preceded a score increase in a 'success' table.\" This means the framework assumes the credit should be given to the immediate state-action pair, which is not true in the general case. Discussing this aspect would be useful. \n\n   e. Several claims are made about the possible behavior of the evolver: \"[the evolver] can learn to increase exploration\ntemperature in early episodes and simultaneously add a new strategic heuristic to its prompt based\non its findings, a multi-faceted adaptation that other methods are incapable of.\" Sharing examples of this behavior would strengthen this claim. For example, plotting the exploration temperature during learning would be interesting.\n\n   f. In Section 5.1, no explanation is provided to justify the choice of the LLM backbone.\n\n\nII. More precision is required to assess the work properly:\n\n   a. A more precise description of the Evolver action space would help clarify the contribution. Especially, the explanation of the deployment-time memory and tool-use routines, which remain high-level without disclosing the details.\n\n   b. Section 4.3 about the selection of configuration using UCB is unclear. Indeed, it is not clear if the agent interacts with the envorinment in a seperate training phase, in which case the additional interactions with the environment should be taken into account, or if a different configuration can be attempted at each step during learning, this would raise a problem of credit assignment because it would not be clear which configuration led to a fruitful outcome. \n\n\nIII. The presentation can be improved:\n\n   a. The appendix is rich, but most of its content is not referenced in the main text.\n\n   b. A bigger font size can be used for all figures. Additionally, a shared legend can be used for Figure 2 to avoid repetition.\n\n   c. Line 329, \"claude\" at the end of the line can be removed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)