Dual-Scale World Memory for LLM Agents towards Hard-Exploration Problems
Authors: Minsoo Kim
seung-won hwang
Venue: ICLR 2026 Poster (2026)
Keywords: hard-exploration problems
world memory
llm agents
text-based games
Primary area: applications to robotics, autonomy, planning
TL;DR: We introduce GLoW, a framework for LLM agents leveraging dual-scale world memory for hard-exploration problems.

Abstract: LLM-based agents have seen promising advances, yet are still limited in hard-exploration tasks which require agents to perform sustained exploration under sparse feedback. We present GLoW, a novel approach leveraging a dual-scale textual world memory, maintaining a trajectory frontier of high-value discoveries at the global scale, while learning from local trial-and-error in exploration through a Multi-path Advantage Reflection mechanism which infers advantage-based progress signals to guide exploration. To evaluate our framework for hard-exploration, we tackle the Jericho benchmark suite of text-based games, where GLoW achieves a new state-of-the-art performance for LLM-based approaches. Compared to state-of-the-art RL-based methods, our approach achieves comparable performance while requiring 100-800× fewer environment interactions. When scaled to stronger LLMs, GLoW surpasses all prior methods on 4 out of 6 difficult and extreme Jericho games.

Published as a conference paper at ICLR 2026
DUAL-SCALE WORLD MEMORY FOR LLM AGENTS TOWARDS HARD-EXPLORATION PROBLEMS
Minsoo Kim Seung-won Hwang"
Seoul National University
(mi nsoo seungwonh ennu. .ac.uu
ABSTRACT
LLM-based agents have seen promising advances, yet are still limited in hard- exploration tasks which require agents to perform sustained exploration under sparse feedback. We present GLoW, novel approach leveraging a dual-scale textual world memory, maintaining a trajectory frontier of high-value discover- ies at the global scale, while learning from local trial-and-ertor in exploration through a Multi-path Advantage Reflection mechanism which infers advantage- based progress signals to guide exploration. To evaluate framework for hard- exploration, we tackle the Jericho benchmark suite of text-based games, where GLoW achieves new state-of-the-art performance for L.LM-based approaches. Compared to state-of-the-art RL-based methods, our approach achieves compara- ble performance while requiring 100-800x fewer environment interactions. When scaled to stronger LLMs, GLoW surpasses all prior methods on 4 out of 6 difficult and extreme Jericho games]
INTRODUCTION
While LLM agents Yao et al. 2023: Sumers et al. 2024: Wang et al. 2024) excel at leveraging vast pre-trained knowledge in tasks such robotic planning, software engineering, and web tomation (Ahn et al. 2022 Yang et al. 2024 2025), they are reportedly limited in hand-exploration problems (Sutton Barto 2018 Ecoffet al.] 2021). Hard exploration problems are typically characterized by large sate-action spaces, deceptive local optima, and sparse rewards. These fac- tors often trap naive exploration in local optima, such that exploration fails to reach deeper states with rewards. For LLM agents, such problems pose two central challenges: (1) Global learning, for maintaining long-term knowledge of valuable discoveries during exploration, (2) Local trial-and- error, for quickly refining exploration policies from sparse environmental feedback. Current LLM agent approaches such as ReAct (Yao et al. 2023) or Reflexion (Shinn et al,, (003) support local trial-and-error, but lack mechanisms for long term knowledge accumulation. Consequently, LLM agents fall short on haddexxploration tasks that humans can often solve effectively (Cui et al. 2025 Phan et al. 2025).
In this work, we introduce Global-Local World Memory (GLoW), a framework enabling effective exploration in hard-exploration problems through dual-scale world memory for global and local learning. Inspired by recent views of LLM world models as building implicit representations of task-relevant world knowledge (Li al. 2024 Ding et al. 2025). our world memory modules capture experiential knowledge from exploration trajectories as structured textual representations, rather than modeling transition dynamics. To instantiate our dual-scale world memory, we build on the Go- Explore (Ecoffet et al. (222I) algorithm, which achieves breakthroughs on hard-exploration problems by enhancing the exploration capabilities of RL and L.LM-based agents (Lu et al. 2025). The key idea of Go-Explore is to store discovered states into a state archive. Then, based on this archive, Go-Explore decomposes hard-exploration into alternating between: (1) a selection phase, choosing promising state from the archive to return to, and (2) exploration phase, to continue discovering new states from the selected state. In its original implementation, Go-Explore used
"Corresponding author.
"ooee is available at httpp:::/githbb..coo///

---

Published as a conference paper at ICLR 2026
hand-crafted heuristics for selection, and random action sampling for exploration, while later work, such as IGE (Lu et al. 2025) improved selection to leverage LLM inference.
In this work, our insight is that both selection and exploration require structured learning from past exploration experiences, but at different scales: we first enrich beyond an archive of isolated states, by additionally maintaining trajectory frontier, which keeps the full temporal context of how high value states were reached and why progress stalled, into a global world memory for richer structured learning. This allows LLM-based analysis across the frontier to infer high-value regions as well as bottleneck states with high future potential, enabling principled state selection beyond heuristic or LLM-internalized notions of interestingness. At the local scale, to guide explo- ration from the chosen state, we draw insights that advantage- based rewards better capture progress signals than Q-values (Kazemnejad et al. 2025 /ettur al. [025]: Our Multi-patt Advantage Re- flection mechanism explores multiple trajectories from the same starting state and leverages LLM reasoning to infer intermediate advantages at key state-action pairs. Through these advantage sig- nals, the local world memory enables controlled exploration under sparse environmental feedback.
To evaluate the capability of LLM agents in hard-exploration problems, we study the Jericho bench- mark suite of text-based games (Hausknecht et al.) (2000), where state-of-the-art methods have been RL-based solutions with e-greedy or softmax exploration (Hausknecht et al. 2020: Ammanabrolu & Hausknecht] 2020 Guo al.) 2020), and MCTS- exploration (Jang et al. 2021 Shi et al.
2025). However, these methods suffer from poor sample efficiency, relying on extensive trial-and- which requires hundreds of thousands of environment interactions. Meanwhile, existing LLM agents have been insufficient to address the challenge of learning from exploration in Jericho games, showing limited performance compared humans (Cui et al. 2025 Phan al., 2025).
Through extensive experiments, we show that GLoW improves the performance of LLM-based agents while achieving orders of magnitude improvement in sample efficiency compared to RL baselines. Our contributions are summarized as follows:
We propose GLoW, a novel LLM agent framework for hard-cxploration problems through global-local world memory.
We conduct comprehensive comparisons with existing agent approaches (RL, MCTS, LLM) and ablation studies validate the components of our method.
We achieve a new state of the art for LLM- approaches on Jericho, achieving perfor- mance comparable to the best RL-based methods while reducing environment interactions required by 100-800x.
BACKGROUND
Jericho Benchmark The Jericho benchmark (Hausknecht et al. 2020) remains an unsolved hard- exploration problem, where the text-based game environments provide two fundamental chal- lenges (Ammanabrolu & (021]: (1) partial observability, requiring agents to construct models of the world from local textual descriptions, and (2) combinatorial state-action spaces. For ample in Zork1, the game vocabulary has 697 words and up to five-word commands, resulting in (6975)) 1.64 x 1024 possible actions per step, though only a tiny fraction are grammatically coherent and contextually relevant. As a result, RL approaches, with simple exploration strategies, incur hundreds of thousands interactions to offset sample inefficiencies in exploration. This makes Jericho an ideal testbed for evaluating whether agents learn by exploring, rather than brute-force discovery.
Methods for Explorrxpoortion Problems Go-Explore (Ecoffet et al. 2021 achieved break- throughs in exrr-axploration problems by maintaining an archive of discovered states as global knowledge to (1) select promising states and (2) explore from the state. Algorithm [ iillustrates this framework across Go-Explore variants, contrasting selection and exploration strategies. The origi- nal algorithm uses novelty-based heuristics for state selection and random actions for exploration. XTX (Tuyls et al. 2022) improves upon these with imitation learning for selection and DQN with curiosity rewards for exploration, while IGE (Lu al.) 2025) leverages LLM inference for both phases. Our approach introduces two key innovations: (1) a trajectory frontier FF with LLM-bbsed value decomposition for principled state selection, and (2) Multi-path Advantage Reflection (MAR)
2

---

Published as a conference paper at ICLR 2026
Algorithm Go-Explore-based Algorithms
1: procedure Go-EXPPLORE-FMMIXX(,, Thiter)
2: A (so, 0)}
D. Archive of (state, score) & Collected trajectories Trajectory Frontier
3: T
JF
5: for 1 to Riter do
Go Phase (State Selection)
6:
Go-Explore: 8nezt h(AA)
Hand-crafted heuristic (e.g., visit count, domain score) Imitation learning Ill-defined promising-nes > Principled value decomposition (Sec. 3.1
XTX: Sneet Imitationl Leaninng( (T)
IGE: Smont LL...elecdPrommsinng(A
9:
GLoW: Wolaha < (T)
10:
+- alignLLM (A, Wglobat)
11:
Explore Phase
12:
Go-Explore: + GandomActions(sneess
> No learning b DQN with curiosity reward Standard LLM agent
13:
XTX: T f- DQN( Smext)
14:
IGE: 7 <- ReAAt((Seen
15:
GLoW:
for j
do
LLM agent advantage drivenn exploration (Sec.
17:
t Texplore (Sueatt, Wincat {T...... 7 7 F)
18:
Wincal + MAR((TI......33)
19:
end for
20:
(T...,, Ta) --top-k(F ;
Collect trajectories
21:
Update trajectory frontier
22:
Archive Update for each state s' in do IssNotRedunaant then
23:
24:
Domain spenifc novelty
25:
A A {s' end if end for
26:
27:
28:
end for
29:
end procedure
for learning from local exploration. Beyond the Go-Explore family, MCTS-based methods like MC- LAVE (Jang al. 2021) and MC-DML (Shi et al. 2025) leverage tree search with language-driven exploration and LLM priors respectively, though requiring 400,000+ interactions.
METHOD
In this section, we describe the dual-scale learning paradigm of GLoW's world memory.
3.1 GLOBAL WORLD MEMORY FOR STATE SELECTION
The global world memory module extracts value signals from accumulated exploration trajectories. Unlike traditional state-hased archives, we maintain trajectories in value-ranked frontier. The global world memory module additionally maintains I.LM-generated trajectory analysis.
As the source of value information, the global world memory module maintains trajectory frontier = {TI, 72,... Th containing the k highest-value trajec- tories discovered during exploration, ranked by value function T R. Each trajectory T; = (st, at, ri. si, ..., at. r st.) represents a complete episode generated by the exploration pol- icy Texplore defined by the LLM agent, where E S are states, a₂ AA are actions, and IR are rewards. For the trajectory value function v, we use the maximum cumulative reward achieved dur- ing the episode, u(T) = max E/= ry- This is an effective choice for Jericho's 's sparse reward structure, where agents can encounter negative rewards or terminal failures. contrast to state-only representations, which lose the context of action and observation sequences, preserving complete trajectories enables accurate credit assignment and value estimation in sparse-reward environments where success depends precise action sequences. For instance, in Zork1, progressing past the
Value-Ranked Trajectory Frontier
3

---

Published as a conference paper at ICLR 2026
troll requires first acquiring both the lantern and the sword before descending into the cellar, but only entering the cellar yields a reward. Analyzing complete trajectories, which may each capture different portions of these sequential dependencies, enables inferring across them that both items are necessary despite the sparse feedback.
The frontier evolves progressively through iterative exploration. When exploration from selected states (detailed in Section 3.2) produces trajectory Tnew with value ((Tnew), the frontier is updated:
Fi+1 top-k(FF U (Tuew).))
(1)
This sliding window mechanism ensures the frontier maintains diverse high-value strategies, while allowing newly discovered superior trajectories to replace outdated ones. For any state 8,, we can derive the achieved value v(s;)) max Er v(T), representing the maximum value reached from state 9; across all frontier trajectories. By tracking complete trajectories, the frontier serves as both estimator of achieved values and repository of successful action sequences.
Motivation: Decomposing value for select and explore Inspired by UCB's s value decomposition which balances exploitation with exploration bonus
(s) + +oV log(NN
12g
where (s) is the empirical mean value and the second term is the exploration bonus based visit count nt we annotate two types of values e and v'. corresponding to each term, by analyzing pat- terns across all frontier trajectories F, to extract set of critical global states with value annotations, forming the global world memory:
W global (F) (1, (0))),(s))) (82, U2,
Uk v/)3
Here, each (s;, Uy et)) represents a critical global state, key semantic landmarks such as exploration frontiers, bottlenecks, and milestones, identified from frontier analysis by a prompted LLM JLLM, where denotes the achieved value from 8;, while v/ reflects LLM's estimate of future value po- tential. Importantly, this potential value vf cannot be derived from trajectory scores alone, requiring LLM's reasoning about why trajectories fail and what progress could be achieved by resolving cur- rent bottlenecks. For instance, state where multiple trajectories fail might have low achieved value, but have high potential value when: (I) multiple high-value trajectories converge but fail to progress further, suggesting unexplored regions beyond, (2) partial solution patterns indicate missing com- ponents, or (3) environmental hints suggest valuable areas remain undiscovered. This implements a semantic form of optimism under uncertainty (Auer) 2003; Brafman & Tennenholtz) (003) where UCB uses statistical bonuses while we derive optimistic values from LLM analysis of bottlenecks. See Appendix F.] for full example of W global generated for Zorkl.
Balancing Exploitation and Exploration in State Selection We maintain a state archive A {(s,, sc)re(s;))} containing discovered states with their achieved scores. Given Wgiobal, we select the next exploration state Snext by balaneing achieved and potential values via LLM as shown in state selection operation which evaluates how
Fig. (a).
We leverage alİgN,IM- an
Value Decomposltion
Algorithm Value-aligned State Selection
geees
State: Dollee Juz iwde death taat
paal rewand
Spare
Foss
Require: Frontier F, State archive A
Smeee 25
BLN
fiom
Ensure: Selected state Snent
Sahiing trail Aestiiennti
Xate 30
I: W Watabat +- gL.. (F) where W Woloba! [[>1] v1, v1) vf).... (Sk, Uks v/2)
Soree 5
allgo .M
0e: achieved, u/: potential
2: for each state E A do
Trajectory Frontier F
Trajectoy added
3: score(s] + aligNLLM(s, Watobal)
Togeetar Rove / (Max Scar:: Real Seare: 36p Freer Kittwo -n Open meprier (9 Thome gat Beath
4: end for
5: Snezt arg max.ed score[s] return Sweaet
Trajectary Rark (MacScons Real Score: Kort
window
Figure 1: (a) Select procedure in GLoW, (b) Illustration of selection with Global World Memory
well each archived state aligns with the high-value patterns identified in W Wytohal using prompted
4

---

Published as a conference paper at ICLR 2026
for the full prompt). Since Wolobal contains both achieved and potential values for key frontier states, this aligmment naturally balances exploitation (favoring states similar to proven high-reward regions), with exploration (prioritizing states near identified bottlenecks with high potential). Fig. (b) illustrates selection in GLoW with the Global World Memory Module where new trajectory (highlighted in gold) has been added the frontier. Once state is chosen, we replay the stored sequence of actions to return to the state) which becomes the starting point of the next exploration phase, described in the following section.
LLM (see Appendix E.2
3.2 LOCAL WORLD MEMORY FOR EXPLORATION
In addition to the selection of states which align with exploration goals with high potential value, exploration can be enhanced by learning which actions are likely to lead to further progress, which the objective of the local world memory module.
Motivation: From Q-values to Advantages for Exploration Existing LLM learning methods like self-reflection be viewed estimating state-action values (Q-values) from single trajectories. However, Q-value estimation from sparse rewards is notoriously high-variance (Sutton et al. 1999. Schulman al. 2017), and we observe the same challenge in LLM- M-seededd learning: inferences from entire trajectories with sparse feedback are prone to incorrect causal attribution.
Drawing from RL theory, advantage functions A(s,a) = Q(s, a) V(s) reduce variance by com- paring actions to baseline rather than estimating absolute values. Recent work on process reward models (PRMs) further demonstrates that advantage-based rewards are more suited for exploration, by better capturing progress signals than Q-values, which tend to exploit known strategies (Setlur ctt al. 2025 Kazemnejad et al. 2025).
Multi-pate putti-pathh&o Advantage Reflection (MAR) Inspired by TRPO (Schulman et al. 2015 which com- putes robust advantage in sparse-reward setting over multiple rollouts from the same state, we pro- pose Multi-path Advantage Reflection to compare multiple trajectories from the same starting state, to produce pseudo-denss advantage signals from sparse environmental feedback. This effectively densifies the reward signal by inferring intermediate advantages at key acteon pairs, providing rich guidance for exploration where environmental rewards are insufficient.
Given state selected based the global world memory, perform iterative exploration by sampling n trajectories sequentially: after each trajectory т;, we perform MAR to extract learnings that inform the next trajectory 7 in the form of local world memory Wiocal This creates sequence T, = (2,,, J where each trajectory benefits from insights gained from previous attempts.
Algorithm 3 Exploration with MAR
Advantage driven Exploration
Require: Selected state Smezt, Frontier F. Ex- ploration count
Entered the kitchen
Ensure: Trajectory set T.
Opentrapdoge
1: T. +
R:
2: Wlocal +
gariic
Inspdor
3: for to do
(MARF) Plus
4: Ti Texpiore(Sneatt, Wiocal, F)
titt meard
To < T,
6: Wieet +- M MAR(T,,FF
Throw: SHOrd
Morriyy Advantage (MAR) M_..
where Winent ((si (5), Acg))
-6
7:
Vourleed
You died
killed
8:
return To
Figure 2: (a) Explore procedure in GLoW, (b) Illustration of exploration with Local World Memory
Semantic Advantage Representation Concretely, MAR is an LLM operation taking the local ex- ploration trajectories T, and frontier trajectories F as inputs, and generating the structured textual output Wiocal {( A.; ), (sū, As)), where si.... S% are critical states (typically 2-4) and
Note that this assumes deterministic environment. We discuss this limitation and possible stochastic extensions Appendix ||
5

---

Published as a conference paper at ICLR 2026
each As; encodes semantic advantages. MAR features two design principles for enhancing the curacy of semantic advantage inference: First, multi- multi-trajectory comparison enables LLM reasoning to aggregate over divergent outcomes revealing good/bad actions, consistent patterns confirming reliable strategies, while focusing analysis on critical states where these signals are most informative. Second, the frontier trajectories (representing the best outcomes achieved far) provide stable reference point, grounding the LLM's evaluation of whether new trajectories constitute meaningful progress. This implements functional role analogous to a value baseline through context-based reasoning rather than numerical subtraction
Unlike scalar advantages A(s. a), these semantic advantages capture progress signals not expressed by sparse rewards, while serving an analogous functional role by guiding exploration policy through Wiocat- We provide full example of Wicent generated for Zork1 Appendix F22
Exploration Policy
The local world memory module enhances the exploration phase by guiding policy defined by LLM agent,
Texplore Fexplore(a|ss: h1) = Agent by, Wiecal, Tay F)
where he is the current trajectory history, T, contains previous trajectories in the same exploration phase, and the policy leverages both learned advantages from Wiecal and successful strategies from frontier F. Fig. 2 iillustrates exploration in GLoW with the local world memory module. Consider a trajectory (gold) that reached the cellar but failed at the troll bottleneck without the sword. Based on the global world memory (Fig. D. which identifies high o' at the cellar state, this state becomes Snezt (orange root, Fig. 2p. The local world memory guides multiple exploration attempts (purple paths), where MAR identifies advantages for "taking sword" despite immediate reward. This advantage learning guides successful exploration through the troll bottleneck (rightmost path). Fi- nally, to address Jericho's exponential action space, implement hybrid approach combining free generation with soft constraints given by valid actions from the Jericho environment (further
details are provided in Appendix E.4
4 RESULTS
We evaluate GLoW on the Jericho benchmark suite. We next present baselines (Sec. 4.1. exper- imental setup (Sec. [4.2). main results demonstrating the effectiveness of GLoW (Sec. [4.3] and ablation studies (Sec. 4.4 isolating each module contribution. Lastly, provide detailed analysis
of exploration dynamics in (Sec. 4.5)
4.1 BASELINES
We perform comprehensive comparison against baselines spanning RL-based, MCTS-based, and LLM-based approaches. Furthermore, we compare with specialized methods for hard-exploration problems each type of baseline. All methods assume access valid actions from Jericho.
RL-Based Methods: DRRN (He et al. 2016) is value-based RL approach for choice based games, learning Q-values for valid actions using GRU encoders and decoders trained via TD loss. KG-A2C (Ammanabrolu & Hausknecht] (020) is an on-policy RI agent that adapts Advantage Actor Critie (A2C) (Mnih et al., (2166), augmented by a dynamic knowledge graph as state rep- resentation that is learned during exploration. Similar to DRRN, RC-DQN (Guo et al. (020) is a DQN-based agent (Mnih et al., 2015), but leverages object-centric neural reading comprehen- sion architectures (Seo et al., (017) for computing Q-values from observations. eXploit- eXploit-Then- Then- eXplore (XTX) (Tuyls et al. 2022) is the current state-of-the- method in Jericho, implementing Go-Explore with imitation learning promising trajectories for state selection, and DQN with in- trinsic curiosity reward for exploration. RL-based methods rely on million-scale interaction data to learn, leveraging parallel environments for training, with the exception of RC-DQN which leverages 100,000 interactions.
'I Appendix A we provide theoretical motivation showing how multi cilli-ttaaectory comparison with stable reference can reduce variance in numerical advantage estimation. MAR applies these principles through L.M reasoning rather than explicit numerical computation.

---

Published as a conference paper at ICLR 2026
MCTS-Based Methods: Monte Carlo Tree Search is widely adopted for large sequential decision- making problems (Browne et al. 2012. Silver et al. 2016), which explores effectively by combining random sampling and tree search. MC- MC-LAVE (Jang et al. 2021) combines MCTS with language-
driven exploration, concentrating search effort on promising actions identified based on value es- timates from semantically similar past actions. MC-DMI. (Shi al., (025) enhances MCTS by incorporating LLMs as action priors in the PUCT algorithm (iilerr et al. 2016), which balances exploration and exploitation during tree search. The LLM equipped with a cross-trial memory mechanism, allowing it to learn from past experiences such as death in Zork1. Both methods require around 400,000 environment interactions build comprehensive search trees.
LLM-Based Methods: ReAct (ao et al., 2023) is the widely adopted standard LLM agent ap- proach interleaving reasoning and acting- Reflexion (Shinn et al., 2023)
multi-episode ap- proach building on ReAct, incorporating self-reflection on each episode to guide future episodes. In-context Reinforcement Learning (ICRL) (Song et al., 2026) is another multi-episode approach leveraging in-context reinforcement learning, using cumulative history of past trajectories and re- wards as context for future episodes. Intelligent Go-Explore (IGE) (Lu et al. 2025) implements Go-Explore with LLMs, leveraging LLM-based state selection from state archive, combined with ReAct-based exploration. As LLM-based baseline methods were not originally applied on Jeri- cho, we re-implement them for Jericho using the action generation approach with valid action soft- constraint described in Sec. 3.2 All I.-M-based approaches use 1,000 interactions to balance per- formance and API cost. We provide details of LLM API usage and cost Appendix D
4.2 EXPERIMENTAL. SETUP
Implementation Details Each method is evaluated over 3 runs, reporting mean and standard devi- ation of maximum achieved scores. ReAct performs 20 independent 50-step episodes. Reflexion performs 20 trials of 50-step episodes, incorporating Sliding-windod vious attempts. Likewise, ICRL includes a sliding window of 10 previous trajectories as in-context examples. IGE and GLoW adaptively alternate between state selection and 50-step exploration episodes within the total 1,000 step budget. We found 50 steps to be sufficient for baseline agents, as they typically plateau early on puzzles or repetitive action loops before reaching this limit. We use temperature 0.5 for all methods except IGE, which uses 0.3 following Lu al. (025). For GLoW hyperparameters, n=3 exploration trajectories and k=5 trajectory frontier size is used.
memories from up to 10 pre-
Evaluation We evaluate on 10 games from the Jericho benchmark (Hausknecht al. 2020), span- ning different difficulty levels. Jericho's games feature massive combinatorial state-action spaces (e.g., 110 rooms and 70 interactable objects in Zork1) with no explicit task instructions, requiring agents to discover intermediate goals and hidden causal dependencies through exploration. Further- more, as an established benchmark suite evaluated by several prior works, enables direct compar- ison with RI., MCTS, and L.Le-based approaches. Following the benchmark's 's categorization, we test on possible games (Pentari, Detective, Temple, Ztuu) featuring moderate puzzles and frequent rewards, difficult games (Zorkl, Zork3, Deephome, Ludicorp) requiring more complex inventory management, puzzle-solving and navigation, and extreme games (Enchanter) involving non-standard actions and spell mechanics. We use the standard Jericho interface providing textual observations and access to valid actions at each step. Unlike some prior work, we do not augment observations with explicit "look" or "inventory" commands, instead allowing agents to learn these through play.
4.3 MAIN RESULTS
We report our main results in Table GLoW achieves new sfate-of-the-a art performance among LLM approaches across 7 out of 10 games. On Zork1, a canonical game of the Jericho suite, our method reaches score of 73.0, a significant improvement over the next best L.MM method (ICRL at 51.7), and surpassing all compared approaches (with the exception of XTX), including RL and MCTS baselines that use orders of magnitude more interactions. We observe the same strong improvements over the closest LLM method in Ludicorp (73.7 vs. 32.0 for ICRL), Enchanter (61.7 vs. 50.0 for IGE), Ztuu (29.3 vs. 18.7 for ReAct), and Balances (16.7 vs. 11.7 for ICRL).
Notably, our implementation of baselines with hybrid action generation approach shows surprisingly strong performance, whereas prior works reported near-zero scores for LLM agents on Jericho (Shi
et al. 2025 Cui al. 2025 Phan al. 2025). Our implementation enables ReAct, Reflexion
7

---

Published as a conference paper at ICLR 2026
Games
Rl-haord
MCTS-hoodd
LJM-bourd
DRRN
KG-A2C
RC-DQN
XTX
MC-LAVE
MC-DML
RoAct
Redlesion
ICRL
IGE
GLoW (Oers)
Sieps
1,00,,000
1,600,000
100,000
800,000
--000000
~0000,000
1000
1000
1000
1000
1000
Enchanterr
12.1
52.0
20u03
46.7
48.3
43.3
50.0
Zorki
32.6
40.2
588
1634
45.2
48.66218
48.3
48.0g50
51.7
44.3
73.5
7ork3
0.5
0.0
2,83
4.2
3,3a
3.0mn
2.7
3.0g00
3.7
4.3
Deephome
20
77.7-21
35
678141
11.0aa2
22.0sta
24.0a57
71.3a4
75.0ss 0330
Ludicorp
13.8
19.8
17
78.8
22.8
19.67
19.7,09
21.7
32.0g71
28.3gm
23.7-110
Balances
10
10
10sto
10ate
11.7a24
10.0gaa
1677
Pentari
27.2
44
43.8
49.6
68
70..a
30.0.00
30,0aaa
26.7447
30.0.aa
30.0aco
Desoctive
197.8
33Resa
291.3
312.2
330
346.67:944
113.3
7anes
233.3
316.7p47
310.0a13
Temple
7.4
8
8:ep
Kahh
8.70m
8.7
8ano
13.7
13.0
Ztuu
21.6
Sa0g
23.67a12
18.7az-
18.3a2a
16.7a41
15.0
29.3:4
Table 1: Comparison of RL-based, MCTS-based, and LLM-based methods on Jericho benchmark games. We report mean i standard deviation over 3 runs following prior works (Tuyls et al.) [022] Shi et al. 2025). Bold indicates best overall performance, and underline indicates second- best. Steps shows total environment interactions. The color of game names indicates original game difficulty categories from Hausknecht et al. (2020): extreme, difficult, and possible. GLoW achieves state- of-the-art among LLM-based approaches in 7/10 games, and is overall best among all compared approaches in 3/10, second- in 5/10.
and ICRL reach 48.3, 48.0, 51.7 on Zork1, respectively, and similarly on par with RL baselines such as KG-A2C and RC-DQN across the board. While this reveals the sample efficiency of LLM agents, these baselines still fall far short of more advanced exploration methods such as XTX and MC-DML, demonstrating the necessity of effective exploration for L.M agents.
Next we compare GLoW against advanced exploration approaches. First, comparing with IGE which is the most directly comparable to ours as an LLM-based Go-Explore method, GLoW sub- stantially outperforms with better performance on out of 10 games. GLoW also achieves com- petitive performance with state-of-the-art RL and MCTS methods, XTX and MC-DML. We nearly match the overall state-of-the-art XTX, which uses 800x more interactions, on both Deephome (75.0 vs. 77.7) and Ludicorp (73.7 vs. 78.8), and notably surpass it Enchanter (61.7 vs. 52.0). It also outperforms MC-DML, which employs extensive MCTS- based exploration around 400x more interactions, on most games including Zork1 (73.0 vs. 48.66), Deephome (75.0 67.0), and Lu- dicorp (73.7 vs. 19.67). These results demonstrate that our dual-scale approach combining global world memory for value-based state selection, with local advantage learning for exploration, enables significant performance gains in LLM agents, competitive with sample-intensive RL approaches.
4.4 ABLATION STUDY
To validate the contribution of each component of GLoW, we perform systematic ablations and report the results in T
Effectiveness of Local World Memory We first analyze the efficacy of our local world memory by ablating MAR. We replace MAR by Reflexion, which performs the same multi-path exploration but does not leverage our proposed advantage learning, instead performing single-traeectory reflection on the latest trajectory. The results show that the performance drops significantly across most games, demonstrating that MAR's 's advantage-bass based formulation more effectively leverages multi-trajjecorry information than Reflexion, improving exploration under sparse rewards.
Effectiveness of Global World Memory Next, we analyze the effectiveness of the global world memory module, which consists of the frontier of high-value trajectories, and the LLM-based value analysis and alignment state selection. We first ablate the global world memory W Wotobat, leveraging the raw frontier trajectories for state selection. The negative performance impact shows that, using LLM to reason across the frontier trajectories to infer potential value is indeed effective. Next, we ablate the trajectory frontier F altogether, such that it is not used for state selection or leveraged by the exploration policy. This causes further decrease in performance, confirming the contribution of the trajectory frontier in both phases.
8

---

Published as a conference paper at ICLR 2026
Ablation Variants
Zorkl
Zork3
Enchanter
Deephome
Ludicorp
Balances
GLoW (Full)
73.0=45
4.3:99
61.7=3m1
75.0=a7
73.7=11
16.7=24
X [Local WMJ Multi-path Advantage Reflection (MAR)
70.0am1
4,3ans
51.7=94
56.7am1
54.7am4
11.7434
x (Global WMJ State selection with gatchal
62.0
4,3
60.0=108
61.3s360
63.3a147
13.3+24
(Global WM] Trajectory frontier F
61.7a19
4.0g00
13.3
$7.7
63.3
11.7#24
above
51,3a52
4,3
51.7=99
56.0am12
22.0aa8
10.0aa0
Standard IGB
44.3
3.7400
50.0=1.
71.3249
28.32m1
10.02a0
Table 2: Ablation study on GLoW components. We evaluate the contribution of: (1) Local world memory through MAR, (2) Global world memory for state selection, (3) trajectory frontier F.
Synergy of Local World Memory and Global World Memory Finally, we ablate all the above components together. The resultant model is similar to IGE, with multi-path Reflexion for explo- ration. The results show that simply adding multi-path reflection does not lead to a clear improve- ment over IGE, indicating that the overall performance of GLoW comes from the complementary synergy of its components.
4.5 ANALYSIS
Controlling global vs. local focus with n exploration parameter We study the tradeoff between local learning depth and global exploration coverage by varying n, the number of explorations per selected state. Larger n enables MAR to learn from more trajectories, while smaller n increases state selection frequency, helping escape local minima. With budget B=1000 and steps s=50, minimum state selections is m = BB/ (s n)] 1. With n=1, MAR is turned off. With n> 1, MAR analyzes n-1 local trajectories plus the global frontier trajectories.
Table 3: Controlling the focus on global (less explorations per state but more frequent state selection) vs local learning (more explorations per state). The results demonstrate n=3 explorations from promising states strikes a good balance between the two.
Explorations per State
Max. Steps per Exploration Phase
Min. State Selection
Zorkl
Zork3
Enchanter
Deephome
Ludicorp
Balances
(no MAR)
50
59.0:37
3.7
58.3
59.7
34.02156
13.3
2 (MAR
50
67.3=a7
3.7
55.0a7.
43.3
66.0a37
11.7
3 (MAR
50
73.0=44
4,3:e9
61.7=201
75.0s87
73.7=110
16.7:24
4 (MAR
50
63.02es
4,3
66.7
73.7144
62.0a124
16.7
5 (MAR
50
59.3
4.0
46.7262
76.3168
53.3270
15.0g00
Table 3 shows that extreme values of n generally yield suboptimal performance, When n=1, ef- fectively disabling MAR, performance drops significantly on certain games like Ludicorp (34.0 vs 73.7 with n=3). Conversely, Deephome shows consistent improvement with increasing suggest- ing particularly benefits from deeper local exploration. The results demonstrate that moderate increases in improve performance across several games, consistent with our theoretical motiva- tion (Appendix AA that MAR should benefit from multi-trajectory comparisons. However, setting n=5 begins degrade performance, excessive commitment to individual exploration phases duces minimum state selection frequency to just 3, increasing susceptibility to local optima. These findings indicate that balancing global and local learning is crucial. We select n=3 as our default parameter, as it achieves the best overall performance by providing sufficient trajectories for robust advantage estimation while maintaining adequate state selection frequency to escape local minima.
4.6 SCALING WITH STRONGER LLMS
To validate that GLoW generalizes across model capabilities, we further evaluate all LLM-bbsse methods using GPT-4.1 on the Extreme/Difficult games. Notably, we find that our method estab- lishes new state-of-the-art across all existing agents on 4 out of 6 games, even surpassing XTX on out of 6 games, while using 800x fewer interactions. These results demonstrate the robust- ness of our proposed approach across LLM capabilities. We provide further qualitative analysis of how model capability affects W Willball quality in Appendix and analyze failure modes of GLoW in Appendix H

---

Published as a conference paper at ICLR 2026
RL
LLM-based
Games
GPT-4.1 mins
GPT-4.
XTX
ReAct
Rfi
ICRL
IGE
GLoW
ReAct
Rn
ICRL
IGE
GLoW
Sieps
800K
IK
1K
IK
1K
1K
IK
1K
1K
1K
IK
Enchanter
52.0
46.7194
48.3194
43.3ax
50.011
61.7.m
38.3134
58.3124
45171
68.3
98.3
Zock1
103.4:1s
48.3647
48.0a50
51.7a44
44.3a00
73.0s45
45.0s00
54.3a45
48.0aaa
86.7a241
103.0sat
Zork3
4.2e01
3.0eno
2.7=05
3,0e00
3.7409
4.3-00
3.3e05
2.7.05
3.0500
5.0
Doephome
71.7221
11.0a43
22.0a16
24.0ax1
71.349
75.0sa7
32.32196
22.3
34.7=187
82.0c56
114.7=278
Ludicorp
78.8
19.7
21.7=11
32.0am1
28.3
73.7=110
31.0=24
29.0
31.7a05
89.0=1a
79.0+168
Balances
24
10_no
10100
11.7
10.0zon
16.7124
18.31z4
18.3
16.7134
16.7
26.7.24
Table 4: Results comparing GPT-4.1 mini and GPT-4. on Extreme/Difficult games. XTX, the strongest RL baseline, for reference.
We include
5 RELATED WORKS
Go-Explore-based Methods Go-Explore (Ecoffet et al., 2021) enables effective exploration in sparse-reward environments by decomposing exploration into state selection and exploration phases. IGE (Lu et al.] (025) adapts Go-Explore for LLMs, using LLMe-ased "promisingness" for state se- lection and ReAct for exploration. However, IGE's limited exploration and ill-defined selection criteria limit its effectiveness in complex environments like Jericho. Our work addresses these limi- tations through principled value decomposition for selection, and multi-path advantage learning for exploration.
Agents for Test-based Games RL approaches to Jericho include DRRN (He et al. 2016), KG- A2C (Ammanabrolu & Hausknecht] 2020), and RC-DQN (Guo et al. 2020), and the aforementioned XTX, where all are sample-intensing based methods like MC-LAVE (Jang al., 202T) and MC-DML (Shi al. 2025) leverage tree search but still rely on a similar scale of interactions. We show that LLM agents can achieve compa- rable performance RL methods, while requiring orders of magnitude fewer interactions through structured exploration and learning mechanisms.
relying on hundreds of thousands of interactions. MCTS-
Learning in LLM Agents Recent works have studied how LLMs can learn from experience, Reflex- ion (Shinn et al. 2023) enables learning through self-reflection on failed attempts, while in-context reinforcement learning (ICRL) (Song et al., (026) leverages previous trajectories' history as context. However, these approaches struggle with sparse rewards due to noisy learning signals. Our MAR mechanism addresses this challenge through multi-path advantage-based learning, providing more robust learning signals.
World Models for LIM Agents While traditional world models in model-based RL focus on tran- sition dynamics (Ha & Schmidhuber 2018 Halner et al., (2024), recent works adopt expanded paradigm of world models as mechanisms for implicit representations of task-relevant knowl- edge (Ding al.] 2025 Li et al. 2024). Li et al. (2024) formalize this notion through state ab- straction theory (Abell 2022). showing that effective LLM agents build goal-oriented abstractions without recovering full dynamics. GLoW 's dual-scale world memory draws from this view, where the global world memory extracts value decompositions across global discoveries, while the local world memory captures semantic advantage signals for exploration.
CONCLUSION
We introduce GLoW, a dual-scale world memory framework to tackle hard-exploration problems. GLoW leverages global world memory that enables principled decomposition of state values, and a local world memory that integrates trajectories from the same state as controlled exploration feedbacks. Our approach achieves state-of-the-art performance among LLM methods the chal- lenging Jericho benchmark, while matching RL-based methods that require 800x more environment interactions. By learning global value patterns across discoveries, and local progress signals from multi-path exploration, GLoW overcomes a key limitation of LLM agents in explorraton tasks, demonstrating sample-efficient yet high performance results.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure reproducibility of our results, we provide comprehensive implementation details in the paper. Algorithm provides the complete pseudocode for GLoW, and hyperparameters are detailed in Section 4.2 (n=3 exploration trajectories, temperature=0.5, k=5 frontier size, 1000 environment steps). All prompts used for the global world memory (Appendix E.T). LLM-based state selection (Appendix E.2). MAR (Appendix E.3 and exploration policy (Appendix E.44) are provided in full. Experiments use 1-mm...1m.ii.. 1-wp:ii--222-------1- to evaluate scaling behavior, reporting results averaged over
as the LLM backbone, with additional experi-
ments using gpt-4 1--025--0----- 3 runs with standard deviations. We implement all LLM baselines using the same action generation approach (Section (.2)) for fair comparison. The Jericho benchmark is publicly available, and the standard evaluation protocol from Hausknecht et al. (2020). Code implementation is publicly released.
ACKNOWLEDGMENTTS
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea (oV)) INO.RS-2021- II11134,, Artificial Intelligence Graduate School Program (Scoul National University)], and the National Research Foundation of Korea(NRF) grant funded by the Korea govemmenn(MSITT (No. 18-2024-004149811, and the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program 2025-202000-01126 super- vised by the IITtinse for Information & Communications Technology Planning & Evaluation).
REFERENCES
David Abel. A theory of abstraction in reinforcement learning. arXiv preprint arXiv:2203.00397, 2022.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huel Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can and not as say: Grounding language in robotic affordances. arXiv preprint urXiv:2204.01691, 2022.
Prithviraj Ammanabrolu and Matthew Hausknecht. Graph constrained reinforcement learning for natural language action spaces. In International Conference Learning Representations, 2020.
URL https: http:::://ppeeeevieeee net htte:/ffommmiid-lllllooo
Prithviraj Ammanabrolu and Mark Riedl. Modeling worlds in text. In Advances in Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2021. URL https:
Copeeneevieev.eet/ooruundddddd THHNEEEE
Peter Auer. Using confidence bounds for exploitation-exploraiiii trade-offs. J. Mach. Learn. Res., 3(mull):397-422 March 2003. ISSN 1432-4435.
Ronen Brafman and Moshe Tennenholtz. R-max a general polynomial time algorithm for near- optimal reinforcement learning. J. Mach. Learn. Res., 3(null):213 2311, March 2003. ISSN 4432. doi: 10.1162/15224440776600000
URL hctps: /tt::///doo..o wttp:/..........1//o////.1 1166/
Cameron B. Browne, Edward Powley, Daniel Whitehouse, Simon M. Lucas, Peter I. Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in Games, 4(1):1-43, 2012. doi: 10. 1011099/CIANG2222222881100

---

Published as a conference paper at ICLR 2026
Christopher Cui, Xingdi Yuan, Ziang Xiao, Prithviraj Ammanabrolu, and Mare-Alexandre Côté. Tales: Text- adventure learning environment suite. arXiv preprint arXiv:2504. 14128, 2025.
Jingtao Ding, Yunke Zhang. Yu Shang. Yuheng Zhang, Zefang Zong, Jie Feng. Yuan Yuan, Hongyuan Su, Nian Li, Nicholas Sukiennik, Fengli Xu, and Yong Li. Understanding world or predicting future? comprehensive survey of world models. ACM Compul. Surv., 58(3), Septem- ber 2025. ISSN 0360-0300. doi: 10.1145/3746449, URL https://dci wtt.:/..g//1.....100 3746449
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint 190.vv:9911119990 2021.
Xiaoxiao Guo, Mo Yu, Yupeng Gao, Chuang Gan, Murray Campbell, and Shiyu Chang. Interactive fiction game playing as multi-paragrappe reading comprehension with reinforcement learning. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP),, pp. 7755-7765, Online, November 2020. Association for Computational Linguistics. doi: (011863//112220.mmm// 624. URL httpps::/acaaanllllgrgto wrt.aaatoooooyooggg//00.. org/2020 emnlp main. 624/
David Ha and Jürgen Schmidhuber. lution. In
Recurrent world models facilitate policy Advances in Neural Information Processing Systems, 2018.
URL
httppeerr.lesssppprr//////////// 2de5d1 (beeca1.88.cc3330007744222222223 Paalzba-Pappe... pdf
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301. 10044, 2024.
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. Inter- active fiction games: A colossal adventure. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7903-7910, Apr. 2020.
doi: 10. 10.1609/aaii.va. h3tp://.ii..440...omm v34105.6297. URL https:
//ojs. aaai org//ndeg/////oo /ttp:////p/paaaaaaaeee////////// www.aceeeeeeeeeeeeteetete.eetttt
AAAI
6297
He, Jianshu Chen, Xiaodong Hc, Jianfeng Gao, Lihong Li, Deng, and Mari Ostendorf. Deep reinforcement learning with natural language action space. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1621-1630, Berlin, Germany, August 2016. Association for Com- putational Linguistics. doi: 10. 10.18663/vv 16653//1/P16-1135 URL https:://aaaantthhteee 916-1153/
org/
Youngsoo Jang, Scokin Seo, Jongmin Lee, and Kee-Eung Kim. Monte-carlo planning and learning with language action value estimates. In International Conference Learning Representations, 2021.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Refining credit assignment in RL train- ing of LLMs. In International Conference on Machine Learning, 2025. URL https:
Zichao Li, Yanshuai Cao, and Jackie CK Cheung. Do LLMs build world representations? probing through the lens of state abstraction. In Advances in Neural Information Processing Systems, 2024.
net wwtp:ooummmidlleeeeee. YYuWgY
Cong Lu, Shengran Hu, and Jeff Clune. Intelligent go-explore: Standing the shoulders of gi- ant foundation models. In International Conference on Learning Representations, 2025. URL
https::/openreviees http:/rwwwwwwweeeeeofoeetee htt.:eeeeee/ooooooeodaaaaaeeeaee
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, February 2015. ISSN 00280836. URL http://dx. doi wtt.:odo...ooo../. 100 1038 / 4236
12

---

Published as a conference paper at ICLR 2026
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep rein- forcement learning. In International Conference Machine Learning. 2016. https:
//proceed ngs... .og.... ml aal..a press/
v48 /ateess/v88/// hhal 6 html
Long Phan, Mantas Mazeika, Andy Zou, and Dan Hendrycks. Texiquests: How good are Ilms at text-based video games? arXiv preprint arXiv:2507.23701, 2025.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region
policy optimization. In International Conference on Muchine Learning, 2015. URL https:
//proceedingg igg ml r.l..... w.w...prsss...... mesass//33//chtt man 1 15. htm
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional atten- tion flow for machine comprehension. In International Conference on Learning Representations, 2017. htttp::
www.
9ge
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In International Conference on Learning Representations, 2025. URL htt.htttp::/ppeeeee httara/eeeeee...eeoo/oooooooaaaa
ew. net. www.nne//forrtt////// wtt
Zijing Shi, Meng Fang, and Ling Chen. Monte carlo planning with large language model for text-based game agents. In International Conference Learning Representations, 2025. URL
net/ http://ppweeeeeeeeeeeeeeereennaa
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learn- ing. In Advances in Neural Information Processing Systems, 2023.
URI
https: http:////prcceennnnsse.ee.ooooep cc/pppe files/ http:/pppr...leeee/ppp////////// Ceaneer-Coffeeencc.c pdf
David Silver, Aja Huang, Christopher Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schritwieser,, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lilli- crap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-503, 2016. URLhttp: html
7/ www. nature com nature http:////.aatetee///////////////
Kefan Song, Amir Moeini, Peng Wang. Lei Gong, Rohan Chandra, Yanjun Qi, and Shangtong Zhang. Reward is enough: LLMs are in-context reinforcement learners. In International Confer- ence on Learning Representations, 2026. URL
net forum? ?
keCXNHOe4W
Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. Cognitive architectures for language agents. Transactions on Machine Learning Research, 2024. URL https: openreview net wet./forummddiii022222
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http: http://cccootttteeeeeeeeeeeeeoos
html
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in Newral Information Processing Systems, 1999. URL UR//httpss///rrcceeiiiee
http:/gss..niii......e...cseeees wasc/paper_fte les/ http:///999.22ee222222222233e2e0
pdf.
Chen Feng Tsai, Xiaochen Zhou, Sierra S. Liu, Jing Li, Mo Yu, and Hongyuan Mei. Can large language models play text games well? current state-of-the. of-ttheartt and open questions. arXiv preprint arXiv:2304. 02868, 2025.
13

---

Published as a conference paper at ICLR 2026
Jens Tuyls, Shunyu Yao, Sham M. Kakade, and Karthik R Narasimhan. Multi- episodic control for strategic exploration in text games. In International Conference on Learning Representations, 2022.
Lei Wang. Chen Ma. Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Ji- akai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), March 2024. ISSN 2095-2236. doi: 10. 1007/s00 17000//11774-02-442231--- URL http://dxx..../// http:://dxdddddooooooooo
10.1007/$50 174 020111000----4--0 024-40231-
John Yang. Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, 2024. UtL/tttps://ppeeeviiie net/ fert/oorm?3 2id22idd-- md=mXpquut 8J3
Ke Yang. Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. Agentoccam: simple yet strong baseline for LLM-based web agents. In International Conference Learning Representations, 2025. URL httpp::
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. URL
vIcYUL-X X
A THEORETICAL MOTIVATION FOR MULTI-PATH ADVANTAGE REFLECTION
A.1 VARIANCE REDUCTION THROUGH TRAJICTRA COMPARISON
The design of MAR draws motivation from classical results variance reduction in advantage estimation. We present this theoretical background that inspired our approach, noting that MAR implements these principles through LLM reasoning rather than explicit numerical computation.
Proposition 1. Consider n trajectories {TI...., J starting from state s. For any state s* visited by m S. of these trajectories, an advantage estimate computed by averaging over m trajectory outcomes has variance reduced by factor 1/m. compared to single-trajectory estimate, assuming bounded variance across trajectories:
Var VarlAmatal (s [ss)] Var (ari/Aiiggee( [s
Proof. For trajectory passing through state s* and taking action aj. let Rz(s", az) denote the random variable representing the sum of future rewards from 8* onward. This provides an unbiased estimate of the true Q(s", a,).
The single-trajectory advantage estimate for action is:
Asingle (s", a) R,(s", .) V(s*)
where (s**) is estimate of the state value. This estimate has high variance because relies single sample: Var VaigAsi (s*,a)] = Var(R, (s", (s*,a)] when V(s*) held constant.
Now consider multi-trajectory approach. From the m trajectories passing through s", let me denote the number of trajectories taking action a. Averaging outcomes yields an improved Q-value estimate:
Quatta (s", = ma jiay ma
Σ R
Using basic properties of variance for independent random variables with equal variance of:
0- = σ2
Varl (s" .a] Var
Σ R
= m
Ma. j:ay=a =a
m.
14

---

Published as a conference paper at ICLR 2026
This shows variance reduction by factor me for the Q-estimate. For the baseline, incorporating stable reference (such as outcomes from previously successful trajectories) rather than a fluctuating estimate further reduces variance. Under the assumption that this stable baseline has low variance relative to the Q-estimate, the variance of the advantage estimate is dominated by the Q-component:
0a
oa S
Var[AAatu($ (s", a)] s Var[Qnutu .a]
Var Var(Asingee(** (s",, ,a]
me
For any action with ma samples, we achieve variance reduction factor of ma-
A.2 CONNECTION TO MAR
The proposition above motivates why multi-trajectory comparison with stable baseline can reduce variance advantage estimation. While not directly approximating the numerical quantity, MAR operationalizes these principles through LLM reasoning rather than explicit numerical computation:
Multi-trajectory aggregation. Rather than computing the average 1 Σj Ry, MAR prompts the LLM to reason across multiple trajectories from the same starting state, identifying consistent pat- terns and divergent outcomes. This achieves benefit analogous to variance reduction through averaging, as the LLM can implicitly weigh evidence from multiple outcomes when inferring which actions led to better progress.
ni
Stable baseline via frontier. The frontier FF serves analogous role the stable (s*)) in the proposition. Like target networks in DQN (Mnih et al. 2015) that update periodically to provide stable targets, J updates only when superior trajectories are discovered, providing consistent ref- erence point for the LLM's evaluation of whether new trajectories constitute meaningful progress.
B ALGORITHMS
We provide the full algorithm of GLoW in Alg.
C CONTAMINATION CHECK
Table 5: Data contamination analysis: LLM accuracy (%) on navigation questions without seeing gameplay.
Accuracy (%)
Game
Questions
GPT-4.I-nano
GPT-4.1-mini
GPT-4.1
Zorkl
230
11.7
10.9
13.5
Zork3
194
8.8
Enchanter
239
10.0
9.2
9.2
Detective
66
13.6
9.1
15.2
Balances
54
5.6
1.9
1.9
Pentari
70
1.4
Deephome
288
13.5
17.0
14.9
Temple
92
19.6
12.0
8.7
Ludicorp
320
22.5
19.7
20.0
Zruu
71
9.9
11.3
13.0
10.4
10.9
To assess whether large language models have prior knowledge of Jericho games, we conducted a data contamination analysis following the methodology of [sai et al.] (2025). We evaluate con- tamination by testing whether models can navigate between locations without being shown any gameplay. Specifically, we: (1) collect a walkthrough trajectory by executing up to 300 steps from each game's 's built-in Jericho walkthrough actions, (2) build graph of locations and transitions from this walkthrough. (3) generate navigation questions asking for paths between observed locations, and (4) query the model with these questions without providing any context. Navigation questions
15

---

Published as a conference paper at ICLR 2026
Algorithm 4 GLoW: Global-Local World Memory
1: procedure (so, liters 71, k)
2:
A F --
Initialize frontier Initialize archive
3:
[(S0, 0)}
1 to Bitte
5:
Snext + SELECTSTATE(FF.4L
6: T <- EXPLOEE F.
7:
UPDATEARCHIVEIT A,A,,
8: end for
9: return E U(T)
10: end procedure
11:
12: procedure SELECTSTATE(F, .A)
13: W globall + (FL
14: Spext + arg maKse alig W Vetaba)
D Select state based on decomposed value
15: return Snext
16:
end procedure
17:
procedure EXPLORE(S,
19:
T +- 0
Initialize trajectory set for current exploration phase
20: Weal t-
21:
for j n do
22:
TT + (T,
Teaplose (8, Whoaal F.F)
Rollout full trajectory from
23:
24:
Winal + MAR(T, F)
25: end for
26: return T
27: end procedure
28:
29: procedure MAR(T,
30:
Wooal &- fLLM(T,
Extract semantic advantages key states
31: return Wixal
32:
end procedure
33:
34: procedure UPDATEARCHIVESTT.FE F.A.A
35:
for T do
36:
Ft (op-k(FU
D. Update trajectory frontier
37:
s'
38:
A sc', score(s))) end for
Add states to state archive
39:
40:
end for
end procedure
take the form: "In [GAME], what steps would you take go [LOCATION B] from [LOCATION AJ?" We evaluate responses using strict pattern matching with word boundaries, requiring the exact sequence navigation commands to appear consecutively in the model's response.
Table 5 shows results of contamination checks for GPT-4. 1naano, GPT-4. 1...-mmii. and GPT-4.1 across 10 Jericho games. We observe minimal contamination
all three models, with all individual game accuracies remaining below 23%. The majority of games show less than 15% accuracy across all models, consistent with random guessing or generic text adventure knowl- edge. The slightly higher accuracies for Ludicorp (up to 22.5%) and Deephome (up to 17.0%) likely reflect the model providing common navigation commands (e.g.. "go south") that occasion- ally match by chance. Notably, scaling model capacity from nano to the full GPT-4.1 does not meaningfully increase contamination, suggesting that these results reflect generic text adventure heuristics rather than memorized game knowledge. Even famous games like Zorkl show accu- racy near chance level (10.9-13, 13.5%). while less-known games like Balances (1.9-5.6%) and Pentari prior knowledge. These low accuracy rates, combined with the model's
(1.4%) show essentially generic responses that lack game-specific details, indicate that our experimental results reflect gen- uine exploration and reasoning capabilities rather than memorized solutions.
16

---

Published as a conference paper at ICLR 2026
D LLM API COST
for all LLM components ($0.40/S1.00 per million in- put/output tokens). Per-run costs of all LLM-based approaches with 1,000 environment steps range from $1 to $7, maintaining practicality for research iteration. For experiments with stronger LLMs, use gpt 1--025-004 04-14 for all LLM components ($2.00/S8.00 per million input/output tokens). Per-run costs range from $7.5 to $45.00, demonstrating that while stronger models increase costs by 5-6x, the relative efficiency of our approach remains consistent with 440% fewer tokens than ICRL., while achieving superior performance.
We use gpt-4. 1pt--.1-minn 10-11minnn 111-1mmnn-222-04---
Table 6: Comparison of LLM API costs with GPT-4.1-mini. We report the average token consump- tion and costs across games (Zork1, Zork3, Deephome, Ludicorp, Detective, Temple).
Method
Input tokens
Output tokens
Total tokens
Avg. cost/run
ICRL
17.6M
43.1K
17.6M
$7.10
GLoW
10.9M
115.1K
11.0M
$4.54
Reflexion
7.2M
50.6K
7.3M
$2.98
IGE
3.9M
44.1K
$1.61
ReAct
2.6M
35.5K
2.6M
$1.08
Table 7: Comparison LLM API costs with GPT-4.1. and costs across the same 6 games.
We report the average token consumption
Method
Input tokens
Output tokens
Total tokens
Avg. cost/run
ICRL
22.4M
55.
22.5M
$45.24
GLoW
13.3M
126.2K
13.4M
$27.58
Reflexion
9.9M
62.5K
10.0M
$20.31
IGE
4.9M
64.2K
5.0M
$10.33
ReAct
3.5M
58.2K
3.5M
$7.45
E PROMPTS
We present the full prompts used in GLoW. Our prompts rely solely on simple instructions and structured output formats without requiring few-shot exemplars, enabling the method to generalize across diverse game scenarios.
E.1 FRONTIER TRAJECTORY ANALYSIS
Analysis (gLLM) Prompt
Analyze these successful game trajectories to identi fy patterns and strategy:
(For each trajectory Fi}
Trajectory N (Peak: x, Final: Y):
[score] action -> observation (reward: +/-N if non-zero)
[score] action -> observation
Based on these trajectories, provide a strategic analysis:
1. FRONTIER & EXPLORATION STATUS:
What areas/locations have been successfully reached?
What remains unexplored or inaccessible?
2. GAME CHECKPOINTS PROGRESS:
17

---

Published as a conference paper at ICLR 2026
What are the key mtiestonns//hhcckointss fdentified?
What items or abilities unlock new areas?
What phase the game are we
3. BOTTLENECKS & CHALLENGES:
Where do trajectories commonly get stuck? What obstacles block further progress? What resources or knowledge are we missing?
4. REWARD STRUCTURE:
When and how are points nts earned?
What actions yield the nighest rewards? Are there patterns to the scoring?
5. NEXT INVESTIGATION GOALS:
What specific objectives should we pursue? Which unexplored areas are most promising? What items or states do we need to reach?
Provide concise strategic summary focusing actionable insights.
E.2
STATE SELECTION
State Selection (alİgn Prompt
STRATEGIC GAME ANALYSIS ===
(Analysis of frentier trajectories W
Based on the above analysis, select the state from the archive that:
Best aligns with the identifiea investigation goals
Can help overcome identified bottlenecks
Explores promising frontiers
Has potential for high rewards based on patterns
Current state archive:
0: [Score: X, Steps: Y, Visits:
Observation: (state observation] Inventory: (state inventory)
[Score: Steps: Y, Visits: Observation: (state observation) Inventory: (state inventory)
Choose state index (0-N). Respond in JSON format:
"thought": "Your reasoning about which state best aligns with the strategic goals", "index": <number>
E.3 MULTI-PATH ADVANTAGE REFLECTION (MAR)
The MAR prompt generates Wieaa as described in Section 3.2 identifying critical decision points
and their associated advantages from multiple exploration trajectories. The prompt incorporates
18

---

Published as a conference paper at ICLR 2026
three inputs: (1) the global trajectory frontier containing highest- value trajectories that serve value baselines, (2) local exploration attempts from the current phase showing different outcomes from the same starting state, and (3) previous W Wical outputs when available, enabling cumulative learning within the exploration phase.
By comparing outcomes across these trajectory sources, MAR produces Wixal ( As; ) identifying where specific actions provide clear advantages. This semantic representation captures causal relationships (e.g., "taking the lamp enables combat in darkness") rather than strictly scalar values, enabling the exploration policy to leverage both statistical patterns from trajectory compari- son and LLM reasoning about game mechanics critical states.
W Wiocal Generation Prompt (MAR)
Review these exploration attempts and identify KEY STATE ADVANTAGES:
(Previous Wiocal from earlier iterations, any)
(Global frontier trajectories F}
(local exploration trajectories from state s}
Analyze all trajectories and identify ADVANTAGES at KEY STATES:
For each important location/state observed ALL attempts, list:
STATE: [dcscription of statc/location]
ADVANTAGES discovered:
[specific action] + sspecific benefit/outcome) (score impact if clear)
[what to avoid] [consequence] (score impact clear)
loptimal sequence] why it's better]
Example format:
STATE: At house entrance with lamp
ADVANTAGES:
"go east" + finds sword (enabled +15 points later) "open mailbox first" + gets crucial map (+5 immediate)
avoid "go upstairs" early overall)
in empty attic (-7
Focus on:
1. states that appear
multiple attempts (to different
outcomes)
2. Critical decision points where scores diverged si signi ficantly
3. Action sequences that consistentll led to success or failure
4. Items information that enabled later progress
Provide 2-4 KEY STATES with their discovered advantages. Bе apecific about actions, items, and locations from the actual game.
E.4 EXPLORATION POLICY IN GLoW
The prompt for the exploration policy in GLoW is shown below. We instruct the LLM to use JSON output format with reasoning and action fields.
Hybrid Action Space While previous works use either constrained selection from valid actions in RL agents (Hausknecht et al. 2020 Ammanabrolu & Hausknecht) 2020 Tuyls et al. 2022) or pure free-form generation in LLM agents such as ReAct. provide the valid actions to the LLM soft constraint, while still allowing frre-form generation. This avoids failure modes both approaches, where constrained selection harm action diversity, while pure generation produce many
19

---

Published as a conference paper at ICLR 2026
invalid actions. As shown in Section 4.2, this hybrid approach, which consistently across both GLoW as well as all LLM baselines, significantly improves the base LLM performance with only lightweight prompt and no few-shot examples.
System Prompt
You are exploring text adventure game. Your goal to make progress and increase your score.
Generate actions that explore possibilities and make progress.
Respond in JSON format:
"thought": "Your reasoning about what to try", "action": "the exact command execute"
User Prompt (at initial step)
(Context includes: Whoal, global frontier trajectories F, previous exploration trajectories, and the current trajectory.
Step 1:
Observation: fcurrent observation] Score: (current score)
Valid actions: (comma-separated list of valid actions]
What is your next move?
User Prompt (subsequent steps)
Step {N}:
Observation: (current observation]
Score: (current score]
Valid actions: (comna- separated list of valid actions]
what is your move?
F QUALITATIVE EXAMPLES
F.1 GLOBAL WORLD MEMORY
Global World Memory Example (Zork1)
Strategic Analysis of Game Trajectories
1. FRONTIER & EXPLORATION STATUS
Successfully Reached Areas:
Starting point: Mailbox and House exterior (north, south, east, west of house)
Forest Fath and Clearing with grating
Up a Tree (bird's nest with egg and leaflet)
Behind House (window entry)
Kitchen and Living Room insice the heuse
Attic (Trajectory 1)
20

---

Published as a conference paper at ICLR 2026
Cellar and Troll Room (Trajectory 1 2)
Unexplored or Inaccessible Areas:
Upstairs beyond the dark staircase (pitch black, dangerous)
West door with gothic lettering (nailed shut)
Crawlway south in collar and forbidding hole west in Trell Room
Steep metal ramp collar (unclimbable)
Chasm area (Irajectory 1 ends in death)
Any further passages beyond troll or trap door
2. GAME CHECKPOINTS & PROGRESS
Key Milestones:
Obtaining jewel-encrusted egg and leaflet from mailbox/tree Using egg and leaflet in grating to unlock cellar access Entering house via window and acquiring items (food, garlic, water, lantern, sword, rope, knife)
Opening trap door and descending into cellar
Defeating or engaging troll cellar
Items /eems//bilities Unlocking New Areas:
Rgg + leaflet Grating Callar entrance
lantern (turned on) Safe navigation
dark cellar
Sword (elvish, glowing) Combat with trell
Rope and knife (potential utility, not fully leveraged yet)
Food, garlic, water (likely survival or puzzle items)
Current Phase: Early-to-mid game exploration of house and callar, initial combat encounters
3. BOTTLENECKS CHALLENGES
Common Stuck Points:
Combat with troll (Trajectories Navigating dark areas without proper light (pitch black upstairs) Attempt ing risky actions (jumping over chasm leads to death) locked or barred doors (gothic door, trap door closing behind player)
end in death
death)
Obstacles Blocking Progress:
Physical barriers (nailed door, barred trap door)
Hostile NPC (troll)
Environmental hazards (darkness, chasm)
Limited inventory or missing key itens
Missing besources/Knooledgee:
Rffective combat strategy stronger weaponry to defeat troll
safely
Means to reopen or bypass barred trap door
Safe traversal methods for chasm or dark upstairs
Possible puzzle solutions involving rope, knife, or other items
4. REWARD STRUCTURE
when Points Are Earned:
for taking the egg
+10 for acquiring food and water itens in kitchen
21

---

Published as a conference paper at ICLR 2026
+25 for entering cellar ssignficcam mi milestone) for moving north from troll room to passage -10 on death and respawn (penalty)
Highest Reward Actions:
Descending into cellar (+25)
Collecting key items early (+5 +10)
Frogressing past major checkpoints
Scoring Patterns:
Early expl exploration yields small incremental rewards
Major area transitions yield large rewards
Death penalizes score, encouraging cautious play
5. NEXT INVESTIGATION GOALS
Specific Objectives:
Develop reliable strategy defeat bypass troll without dying Investigate use rope and knife for puzzles or combat
Explore crawlway south and forbidding hole west in troll room
Find safe method to cross or hypass chasm Attempt to open circumvent nai led gothis door
Most Promising Unexplored Areas:
Crawlway south and forbidding hole west in cellar/troll reom
Upstairs dark staircase (with lantern other light scurce)
Beyond barred trap door if it be reopened
Areas beyond chasm once safe crossing method found
F.2 LOCAL WORLD MEMORY
Local World Memory Example (Zorkl)
Based on detailed exploration attempts, here are KEY STATES with discovered advantages:
STATE:
The Troll Room
Small room with passages cast/south, hole west; troll blocks all exits; sword glows
ADVANTAGES:
"hit troll with sword" kills trell, clears all exits for exploration (score: 40)
"take after troll death obtains valuable weapon for later puzzles
avoid engaging without glowing sword + likely failure optimal: kill troll first, then take axe (mandatory to proceed)
STATE:
Maintenance Room (Flood Control Dam #3) Room with colored buttons, tool chests, wrench, screwdriver, toothpaste, leaking pipe
ADVANTAGES:
"take wrench, screwdriver, tube" essential tools for envi rovment interaction
22

---

Published as a conference paper at ICLR 2026
"push blue button" triggers leak, raises water level to access new areas
"push red button" toggles lights, affects water level
avoid throwing lantern breaks critical light source
optimal: collect tools manage buttons + control water without drowning
Temple Torch Room Dome Room / Altar Large temple with inceriptionss dome rai railing; rope for descent; ivory toren; brass bell; goid coffin
STATE:
ADVANTAGES:
"take ivory torch" stable light for deeper exploraticn
"take bell" key item for spirit/wraith interaction
"ring bell at Entrance to Hades" paralyzes wraiths, enables passage
"blow out candles" + enables safe descent passage optimal: acquire torch bell sceptre manipulate altar control spirits
STATE:
East-West Passage / Chasm Area
Narrow passage with stairs; chasm with paths; multiple routes
ADVANTAGES:
"east" then "north" leads to Reservoir South and further areas "tic rope to railing" enables safe descent into lower levels avoid getting stuck in loops wastes noves optimal: explore chasm edges use rope for vertical access Dome/Torch
Cross-Cutting Insights:
Inventory Management: Strategic dropping/picking essential for critical arti
Light Preservation: Maintaining lantern/torch crucial for dark exploration
Combat Readiness: Glowing sword indicates combat opportunity (essential for progress)
G IMPACT OF MODEL CAPABILITY ON WORLD MEMORY QUALITY
To assess the impact of model capability on global world memory quality and trajectory evalua- tion, we compare three capability scales (GPT-4. -nano, GPT-4.I-mini, and GPT-4.1) on two games (Zorkl and Deephome). To isolate the effect of model capability, use identical frontier trajec- tory set extracted from a GPT-4.1-min checkpoint, generating a new Wglobal with each LLM.
The results shown below. We present abbreviated trajectory contexts (the inputs for W Walobal generation), then summarize each Wglobal into content capturing achieved value (v) and potential value (v').
We first observe that the evaluation of u consistent across model capabilities, indicating that even smaller model (GPT-4.1-nanl --nnoo successfully evaluate trajectories identify achieved goals and checkpoints. Case Study in Zorkl (early game, trol puzzle) serves as the control case, where all three L.LMs largely align in both and v!.
23

---

Published as a conference paper at ICLR 2026
In contrast, in other cases observe clear qualitative differences in the generated o' between GPT- 4. I-nano and the stronger models. In Case Study 2 (Zorkl mid- gand), the trajectories present multi- ple simultaneous puzzles, and find that the specificity of reasoning about these objectives shows qualitative differences correlating with model capability. GPT-4.1 and 1-9T-4.1mmini target specific objectives, while generates generic targets. In Case Study 3 (Deephome early game), models must infer that Ember (a town mentioned in note) is the location containing necessary items to proceed. This information must be recalled from reading a note that provides this locational clue. While the larger models successfully make this inference, 1 1-maoo I-nao fails, resulting in generic "correct parts for elevators (mentioned in note)." missing the erucial Ember reference.
Since W global directly influences subsequent state selection, LLM capability gaps in global world memory generation propagate to exploration performance. The average performance across both games reflects these qualitative differences, shown in Table
Table 8: Impact of model capability on GLoW performance. Scores are mean + standard deviation over runs.
Model
Zork1
Deephome
GPT-4.1-na -nanoo
43.0 2.2
18.3 5.3
GPT-4.1-min
73.0 :: 4.5
75.0 8.7
GPT-4.1
103.0 + 6.8
114.7 27.8
Case Study 1: Zork1 Early Game
Trajectory Context:
Trajectories 1, (score 40): house - cellar - defeat troll -> maze
Trajectory 0 (score 15): house exploration, troll
Achieved value (v)) models consistently identify "Defeating the Troll" as the key milestone.
Potential value (v'):
GPT-4.1-nano: "Explore maze, find keys, unlock grating"
GPT-4.1-mini: "Navigate maze, find safe chasm crossing, unlock grating"
Case Study 2: Zork1 Mid Game
Trajectory Context:
Trajectorics 22, 23 (score 70): dam controls - reservoir drains take trunk +15 pts
Trajectorics 15, 16, (score 55); dam area, Loud Room (can' take platinum bar), no trunk Key observations: Dam puzzles, Loud Room/platinum bar, reservoir access
Achieved value (v): All models consistently identify key milestones: "dam controls," "reservoir access," "trunk acquisition."
Potential value (v'):
GPT--.1-nnao: "Secure tooly (wrench, screwdriver, key)". "Explore blocked passages", "Use col- lected artifacts" (generic objectives)
GPT-4.I-mini: "Find how to turn valve at Dam Base", "Explore grating under leaves in clearing". "Explore reservoir as water levels change" (specifiic locations and puzzles)
GPT-4.1: "Solve Loud Room/Platinun Bar puzzle". "Cross chasm dome room". "Unlock trophy (specific puzzles and objectives)
24

---

Published as a conference paper at ICLR 2026
Case Study 3: Deephome Early Game
Trajectory Context:
Trajectory 2 (score 20): take note - examine warning - explores to City Gates
examine warning say Trajectory (score 1): say manaz - immediate death (never reads note)
Trajectory (score 8): take
death
Key observation: Note text reads "I traveled south to human town called Ember to procure the
correct parts."
Achieved value (v): models consistently identify key milestones: "reaching City Gates," " "discov- ering control panel," and "reading the note."
Potential value (e');
GPT--.I-namo: "correct parts for elevators (mentioned note)" (no destination specified)
GPT-4.1-mini GPT-4.1: "Obtain elevator parts from Ember (human town)" (specific target loca- tion)
H QUALITATIVE ANALYSIS OF FAILURE MODES
We present qualitative analysis identifying primary failure modes observed in GLoW trajec- tories (-Pi-..l-mini).
Conservative State Selection: In Zorkl, we observe that GLoW can repeatedly select early-game states (e.g., State 37: Kitchen, score 15, empty inventory) despite frontier states with significantly higher scores (e.g., State 58: Chasm, score 45, with lantern and sword inventory) being avail- able. Analysis reveals the model over-prioritizes the acquisition of -aseciyassociated items (lantern, sword) from the environment, failing to recognize that frontier states already possess these items in inventory. We posit that this stems from two interacting factors: (a) logical reasoning failure to verify item possession in the frontier states, and (b) an overvaluation of safety-related semantics, which distorts the estimation of potential value (v' in Eq. 2). This biased selection creates a loop where the agent re-solves completed objectives rather than progressing from the frontier.
illustrate a case where exploration stalls despite unlocking a critical mechanism, due to failure to resolve subsequent multi-step de- pendencies. After activating the City Generator (step 827, +30 points), which powers the railway system, the agent fails to pursue the unlocked paths. The solution requires using the powered railway to access intermediate areas (Blacksmith, Waterfall) containing prerequisites for further objectives Water Works (+30) and City Gates (+40). However, the agent attempts the Water Works objective directly failed attempts) and reverts exploring familiar areas, never testing the railway visits in the remainder of the game), This reveals a difficulty in systematically resolving multi-step de- pendencies, particularly when many intermediate objectives are present. We present the two cases in detail below.
Multi-step Dependency Reasoning Failure: In Deephome,
Case 1: Zorkl
Game Context:
Reached deeper frontier states through Troll Room Maze (score 50) and Chasm (score 45),
State archive includes:
State 72: Score 50, Location: Maze, Inventory: lantern, sword,
State 58: Score 45, Location: Chasm, Inventory: lantern, sword
State 37: Score 15, Location: Kitchen, Inventory: empty
Observed Behavior:
Reperitive state selection:
Selection Iter. 2: State 37 (score 15)
Selection Iter. 3: State 37 (score 15)
Selection Iter. State 37 (score 15)
"State 37 is in the living room where the brass lantern and the elvish sword are present. Acquiring and activating these items is crucial for safe exploration... This state aligns well with investigation goals..."
Reasoning for state selection:
25

---

Published as a conference paper at ICLR 2026
Case 2: Deephome
Game Context:
Agent has activated the City Generator (+30 points), which powers the Railway Station controls.
Further progress requires using the Railway access intermediate (Blacksmith, Waterfall).
Observed Behavior:
Step 827: Activates generator (+30 points, score 65)
Failed subsequent multi-stage progression:
Steps 128-1000:
Railway Station visits:
Blacksmith visits: 0
Waterfall visits: 0
Water Works: 2 attempts wheel, hoth fail
I ASSUMPTION OF ENVIRONMENT DETERMINISM
Our implementation assumes deterministic environments, which is leveraged by the state restoration mechanism which replays actions to return to a selected state. This limits applicability in stochas- tic environments where action replay may not return to the intended state. Our trajectory- based design facilitates intuitive extensions to such settings, where a potential approach is framing state restoration goal-reaching problem guided by replay trajectories.
We note that the MAR component naturally handles stochasticity since samples multiple explo- ration paths from a selected state and performs inference over the paths set, enabling the LLM- based analysis to reflect on observed stochastic variations.
J
LLM USAGE
We utilized Claude for minor grammar and language edits in paper writing.
26


Reviews:
[{"review_id": "2zLCuZ1MVC", "reviewer": "Reviewer_Ru4g", "summary": "The paper introduces GLoW, a dual-scale framework designed to improve exploration efficiency of LLM-based agents in hard-exploration environments such as the Jericho benchmark. It combines a Global World Model, which maintains a trajectory frontier and uses LLM reasoning to estimate both achieved and potential state values for principled state selection, with a Local World Model, which applies Multi-path Advantage Reflection (MAR) to infer semantic advantage signals from multiple rollouts under sparse rewards. Experiments show that GLoW achieves new state-of-the-art performance among LLM agents.", "questions": "The paper mentions using GPT-4.1-mini for trajectory analysis. I wonder whether smaller models could achieve comparable effects. It would be insightful to include a few case studies exploring how different model influences the quality of world-model reasoning and trajectory evaluation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed framework achieves strong empirical performance and high sample efficiency on multiple games from benchmark.\n- The proposed semantic advantage representation enhances interpretability compared to scalar rewards.", "weaknesses": "Overall, I like this paper, I just have a few minor questions:\nIn the global world model, how exactly is a trajectory defined? If an episode does not reach a terminal state (e.g., success or failure), does it default to a fixed-length rollout of 50 steps?\nWould excessively long interactions degrade the reasoning quality of the LLM? Conversely, if the LLM is capable of longer reasoning chains, can the trajectory length be adaptively extended? Simply truncating trajectories might risk losing key contextual information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kV1qQR0h3D", "reviewer": "Reviewer_XV8D", "summary": "This paper introduces GLoW, an LLM agent framework for tackling hard-exploration problems through the combination of a global and local world model. The global world model keeps track of a trajectory frontier of states to return back to, while the local world model performs Multi-path Advantage Reflection (MAR). The authors evaluate their method on the Jericho benchmark, a well-established benchmark for text-based games, and find their method to be state-of-the-art among LLM-based approaches (but not among pure RL approaches). Finally, the authors also provide ablation studies to see how different components of their method contribute to the final performance.", "questions": "n/a", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The authors provide strong empirical results: they find their method to be state-of-the-art among LLM-based approaches. While they are still a bit behind the strongest RL-based approaches, the proposed method uses a lot less environment steps.\n2. Conceptually, I like the authors' proposal to think about *semantic* optimism to describe exploration potential, despite being only loosely implemented in prompts.\n3. I commend the authors for providing a data contamination analysis in Appendix C.\n4. I like Algorithm 3 in the appendix, which shows the differences and similarities between various Go-Explore-based algorithms.", "weaknesses": "1. Why is Algorithm 1 in the main text the Go-Explore Algorithm? This feels a bit strange as this is an already existing algorithm and is nothing new.\n2. Page 3: “…. preserving complete trajectories enables accurate credit assignment and value estimation in sparse-reward environments where success depends on precise action sequences” → Could the authors explain why this is the case? There seems to be no evidence or reasoning provided for this.\n3. [Minor] Page 3: this sentence needs rewriting: “This is an effective choice for Jericho’s sparse reward structure, and the possibility of encountering negative rewards or terminal failures.”\n4. Page 3: “to extract a set of critical global states with value annotations” → what are “critical” global states?\n5. Page 3: “Once a state is chosen, we replay the stored sequence of actions to return to the state” → does this mean the method only works in the deterministic setting? If so, I think the authors should mention this as a limitation since some of the prior works (e.g. DRRN, XTX) can operate in both settings.\n6. Page 5: how is the actual advantage identification that’s visualized in Figure 2 done? It doesn’t seem to be described as far as I could see. From the appendix it seems it's all done via prompting, but that's not made clear in the main text.\n7. Page 9: “While traditional world models in model-based RL focused on transition dynamics (Ha & Schmidhuber, 2018; Hafner et al., 2024), recent works show that in the context of LLMs, world models are usefully expanded as mechanisms for extracting task-sufficient state representations (Tang et al., 2024; Li et al., 2024).” → I disagree with the authors here that the current work can really be thought of as world models, which are usually grounded in transition models (even in the mentioned work of Tang et al., 2024). The current work does not build any such thing, and hence I would recommend the authors change the phrasing from world model to something else that more accurately depicts the proposed algorithm.\n8. The writing is somewhat weak in its presentation and contains quite a few places with spelling or grammatical errors.\n9. Figure 1: what does align_{LLM} do? It doesn’t seem to be explained.\n10. Method seems to require quite a bit of hand crafted prompt design based on the prompts in Appendix D.\n11. Since some of the standard errors are a bit large (e.g. 20.1 for Enchanter), it would be good to run a few more seeds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h7wkikFFyZ", "reviewer": "Reviewer_pUbB", "summary": "This paper addresses the limitation of LLM agents in \"hard-exploration\" tasks, which are characterized by sparse rewards, deceptive local optima and large state spaces. The authors propose GLOW, a novel framework that learns at two scales to solve this. GLOW addresses the global learning challenge by maintaining a \"trajectory frontier\" of high-value discoveries, allowing an LLM to analyze these trajectories and select the most promising states for future exploration. To tackle local trial-and-error, its Local World Model uses a Multi-path Advantage Reflection (MAR) mechanism. MAR explores multiple paths from the state selected by the global world model and uses an LLM to infer advantage-like signals. On the Jericho benchmark, GLOW achieves SOTA performance among LLM agents and shows performance comparable to SOTA RL-based methods, with 100-800x fewer environment interactions.", "questions": "- How does the framework's performance differ when using LLMs with different capabilities?\n- Can the authors provide a qualitative analysis of failure cases?\n- Can the authors justify the reason why the experimental environments (e.g., Game of 24, Put Next To (PN) from BabyAI-Text, and The Cooking Game (CG) from TextWorld), which are used in the closely related work IGE, are not used in this work?\n- Can the authors discuss the potential of the proposed method for LLM reasoning? As there are some existing works that also propose different methods to enhance the exploration ability of RLVR algorithms.\n\nI am willing to update my evaluation if my questions and concerns are well addressed during the discussion stage.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Clear and specific motivation\n- Novel framework, combining in-context learning with established RL theories.\n- Much higher sample efficiency without compromising performance, a critical advantage in environments where interaction is costly or slow.\n- Effective use of LLM reasoning, leveraging LLMs for high-level reasoning to provide dense, semantic feedback from sparse rewards.", "weaknesses": "- Heavy reliance on the LLM's reasoning abilities, especially the inferred \"semantic advantages\" are not as rigorous as traditional, mathematically-derived advantage functions.\n- The analysis of efficiency is one-sided, focusing on sample efficiency (environment steps) while obscuring the total LLM calls.\n- The choice of experimental environment is not well justified.\n\n#### Minor\n- The format of reference is inconsistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0VQB3EEGsu", "reviewer": "Reviewer_PRB8", "summary": "This paper introduces GLoW (Global-Local World Models), a new framework for Large Language Model (LLM) agents designed to tackle hard-exploration problems. The core idea is a dual-scale mechanism. At the global scale, GLoW maintains a \"trajectory frontier\" of the most successful past episodes. An LLM analyzes this frontier to decompose state values into achieved rewards and future potential, guiding the selection of promising states to explore from (similar to the \"select\" phase of Go-Explore). At the local scale, it employs a \"Multi-path Advantage Reflection\" (MAR) mechanism, where the agent makes several exploration attempts from a single state. An LLM then compares these attempts to infer \"semantic advantages\" for specific actions, guiding the subsequent exploration policy (the \"explore\" phase).", "questions": "Could the authors first clarify their choice to use the term 'world model', given that the mechanism is not a predictive dynamics model, as is standard in the RL literature, and this term was notably absent in its predecessor, Go-Explore?\n\nThe MAR method is interesting but I guess it would fail in a stochastic setting, do you have any intuition if there is a way to extend it to the stochastic setting?\n\nHave the authors considered the potential of adapting the Multi-path Advantage Reflection mechanism as a powerful reward shaping or exploration module for traditional, sample-based RL agents?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper's central concept of a dual-scale world model is interesting. At the global scale, it improve the Go-Explore simple state archive with a value-ranked trajectory frontier. This allows an LLM to reason across successful episodes to identify bottlenecks and infer a state's \"potential value,\" providing a more principled method for balancing exploration and exploitation than the vague heuristics of prior work. The local-scale Multi-path Advantage Reflection (MAR) uses the LLM to compare divergent outcomes from multiple exploratory paths starting from the same state. This generates dense, semantic advantage signals, creating an interesting solution for credit assignment in sparse-reward environments and a clear improvement over single-trajectory reflection methods.", "weaknesses": "In the established literature of model-based reinforcement learning and robotics, a world model is understood to be a learned, predictive model of an environment's dynamics, specifically its transition function p(s' | s, a) and reward function r(s, a) (Ha & Schmidhuber (2018)). The \"world models\" presented in this paper function as structured, retrospective representations of experiential knowledge: the global model is a curated repository of successful trajectories, and the local model is a synthesized summary of inferred action advantages. The paper's clarity and contribution would be significantly strengthened by either explicitly justifying this terminological choice or, preferably, by adopting a different terminology.\n\nThe experimental setup provides a clear comparison by standardizing on a single LLM (GPT-4.1-mini). While we recognize the practical and budgetary reasons for this choice, a targeted ablation study, even on a subset of the Jericho games with a different LLM, would greatly enhance the paper's contribution. This would provide valuable evidence that the dual-scale mechanism is a robust and generalizable principle, rather than one that relies on the specific idiosyncrasies of the chosen model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)