MIRA: Memory-Integrated Reinforcement Learning Agent  with Limited LLM Guidance
Authors: Narjes Nourzad
Carlee Joe-Wong
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement learning (RL)
Large language models (LLMs)
Memory Graph
LLM-derived priors
Sample Efficiency
Sparse-Reward Environments
Primary area: reinforcement learning
TL;DR: MIRA integrates LLM guidance into RL through a memory graph and utility-shaped advantages, achieving faster learning with far fewer queries.

Abstract: Reinforcement learning (RL) agents often face high sample complexity in sparse or delayed reward settings, due to limited prior knowledge. Conversely, large language models (LLMs) can provide subgoal structures, plausible trajectories, and abstract priors that support early learning. Yet heavy reliance on LLMs introduces scalability issues and risks dependence on unreliable signals, motivating ongoing efforts to integrate LLM guidance without compromising RL’s autonomy. We propose MIRA (Memory-Integrated Reinforcement Learning Agent), which incorporates a structured, evolving memory graph to guide early learning. This graph stores decision-relevant information, such as trajectory segments and subgoal decompositions, and is co-constructed from the agent’s high-return experiences and LLM outputs, amortizing LLM queries into a persistent memory instead of relying on continuous real-time supervision. From this structure, we derive a utility signal that softly adjusts advantage estimation to refine policy updates without altering the underlying reward function. As training progresses, the agent’s policy surpasses the initial LLM-derived priors, and the utility term decays, leaving long-term convergence guarantees intact. We show theoretically that this utility-based shaping improves early-stage learning in sparse-reward settings. Empirically, MIRA outperforms RL baselines and reaches returns comparable to methods that rely on frequent LLM supervision, while requiring substantially fewer online LLM queries.

Published as a conference paper at ICLR 2026
MIRA: MEMORY-INTEGRATEED REINFORCEMENT LEARNING AGENT WITH LIMITED LLM GUIDANCE
Narjes Nourzad
Carlee Joe- LoenWWng
University of Southern California Electrical and Computer Engineering Los Angeles, CA 90007, USA nour nourzad(uscceau
Carnegie Mellon University Electrical and Computer Engineering Pittsburhh, PA 15213, USA cj cjoewonge andrew. cmu edu
ABSTRACT
Reinforcement learning (RL) agents often face high sample complexity in sparse or delayed reward settings, due limited prior knowledge. Conversely, large language models (LLMs) can provide subgoal structures, plausible trajectories, and abstract priors that support early learning. Yet heavy reliance on LLMs in- troduces scalability issues and risks dependence unreliable signals. motivating ongoing efforts to integrate LLM guidance without compromising RL's auton- omy. We propose MIRA Memorry-Inegrrtee Reinforcement Learning Agent), which incorporates a structured, evolving memory graph to guide early learning. This graph stores decision-relevant information, such trajectory segments and subgoal decompositions, and is co-constructed from the agent's high-return ex- periences and LLM outputs, amortizing LLM queries into a persistent memory instead of relying on continuous real-time supervision. From this structure, we derive utility signal that softly adjusts advantage estimation to refine policy up- dates without altering the underlying reward function. As training progresses, the agent's policy surpasses the initial I..M-derived priors, and the utility term decays, leaving long-term convergence guarantees intact. We show theoretically that this utility-basee bbaee shaping improves carly-stage learning in sparse-reward set- tings. Empirically, MIRA outperforms RL baselines and reaches returns compa- rable to methods that rely on frequent LLM supervision, while requiring substan- tially fewer online LLM queries
INTRODUCTION
Reinforcement learning (RL) models sequential decision-making as interactions with an environ- ment and learns behavior from reward-driven feedback. RL has achieved strong results in domains including robotic manipulation, dynamic scheduling, and autonomous planning (Nourzad et al., 2024; Liu et al., 2024; Luo et al., 2024). However, these advances often rely on environments with dense, readily accessible rewards. In many tasks, rewards are sparse or delayed, appearing only when specific goals are reached or several steps after the action unfolds. These weak or infrequent rewards obscure which past actions influenced the outcome, making it difficult to credit the eventual reward appropriately (Velu et al., 2023). This uncertainty weakens the gradient signal, leaving pol- icy updates underinformed. Thus, agents become highly data-hungry and require large numbers of interactions to learn useful behaviors (Devidze et al., 2022). These challenges intensify under partial observability, as agents must generalize from limited state information and often struggle early in training (Hausknecht & Stone, 2015; Kurniawati, 2022). In such settings, random exploration rarely uncovers informative trajectories, leading to slow convergence and high variance in returns.
Large language models (LLMs) provide a complementary source of prior knowledge, especially in environments where rewards are sparse, feedback is delayed, and observations are partial. They have demonstrated capabilities in reasoning over abstract goals, interpreting high-level intent, and lever- aging broad prior knowledge (Jimenez et al., 2023; Xu et al., 2024). These properties make them
Project wehpage: https://narjesno.. thub. 1oobii///RR//

---

Published as a conference paper at ICLR 2026
Partial Observations
Environment
Action
Language Model
Screening Unit
Ubilrt Computer
R MIIRA
VS
Healthy Crafts
Trajectory
Languago Model
Utility Signal
Memory
Figure 1: Offline priors and online LLM suggestions are filtered by a screening unit before being incorporated into the memory graph as healthy grafts. MIRA agent acts under partial observations, interacting with the environment. A utility module evaluates trajectory rollouts against the evolving memory graph, producing utility signal that shapes advantage estimation and policy updates.
natural candidates for providing structured guidance for RL, agents (Schoepp al., 2025). A grow- ing body of work has explored how pretrained LLMs can support RL to improve sample efficiency. One line of research positions the LLM as an implicit or explicit reward model, either estimating reward signals from environment descriptions or generating code to define reward functions (Ma et al., 2025; Kwon et al., 2023; Fan et al., 2022; Rocamonde et al., 2023; Bhambri et al., 2024; Xie et al., 2024). Another line leverages LLMs to generate high-level plans, policy sketches, by-step guidance during training (Du et al., 2023; Hu & Sadigh, 2023; Dasgupta et al., 2023; Wang al., 2023; Zhou al., 2023). A third direction focuses task-level guidance such as subgoal decomposition, curriculum design, or goal interpretation from natural language (Wang et al., 2024a; Ma et al., 2023; Shinn et al., 2023), Additional related approaches appear in Appendix B.
step-
RESEARCH CHALLENGES. The existing approaches, although promising, typically (often per-step) LLM supervision, making the agent's performance heavily reliant on LLM infer- ence, which introduces several difficulties. First, it can interfere with the RL learning signal (Zhou al.. 2023), limiting autonomous decision-making and reducing the agent's ability to generalize or adapt if the L.M becomes unavailable. Second, since L.MSs cannot interact directly with the environment or gather real-time feedhack, full reliance on their instructions is suboptimal (Qu al., 2024; Gao et al., 2024; Cao et al., 2024) and dilutes environment-drivee feedback. Indeed, LLMs carry fundamental risks such as hallucinated outputs, prompt sensitivity, and limited grounding in physical environments (Ji et al., 2023b; Tonmoy et al., 2024; Bang al., 2025), making their out- puts potentially unreliable. Frequent queries also raise scalability concerns due to computational cost and latency (Zhou et al., 2024; Wan et al., 2023). Still, relying solely RL ignores the struc- tured knowledge encoded in LLMs that could accelerate learning or shape behavior in meaningful ways. Thus, the fundamental challenge lies in incorporating LLM guidance in a way that leverages its complementary benefits while preserving the optimization dynamics that make RL effective.
frequent
OUR CONTRIBUTIONS, In this work, we propose MIRA ing Agent), a method that integrates LLM-derived guidance into reinforcement learning through structured memory graph. The memory graph provides an evolving representation of task-relevant information, co-constructed from the agent's 's experience and LLM outputs. Offline priors initialize the structure, while infrequent online queries on batches of partial observations can further refine during training. Nodes represent decision-relevan contexr, such as trajectory segments, and edges encode the hierarchical decomposition linking goals to their subgoals. The graph is designed to re- main compact, adding minimal overhead relative standard replay buffers (Schaul et al., 2015). The memory graph allows the agent to organize and reuse information without repeated LLM queries, providing persistent source of structured knowledge.
Reinforcement Learn-
Over time, the agent validates, revises, and extends the structure based on its own experience, im- proving beyond what LLM guidance alone provide and filtering out mistaken online suggestions. The resulting graph limits dependence on real-time LLM access, alleviating concerns about latency, query cost, and scalability. To integrate the M-L--vevveed information into learning, we derive
2

---

Published as a conference paper at ICLR 2026
utility signal from the memory graph and use softly shape advantage estimates in each RL it- eration. This signal guides early rollouts by reinforeing reward-driven gradients when aligned and moderating updates that arise from inaccurate critic, helping the agent explore more effectively in sparse-reward settings without overriding the environment's feedback. Theoretically, we show that the utility term accelerates early learning. As the policy improves and surpasses the usefulness of LLM-derived guidance, the shaping influence fades, ensuring convergence in the long-horizon limit. We empirically evaluate the effectiveness, sample efficiency, and overhead of incorporating LLM guidance across multiple benchmark environments. Our contributions are summarized follows:
A MEMMORY-INTEGRATES FRAMEWORK FOR LLM GUIDANCE: We propose MIRA, a reinforcement learning agent that integrates LLM-derived guidance through a memory graph co-constructed from agent experience and offline infrequent online LLM outputs. The graph evolves throughout training, reducing reliance on real-time LLM queries.
We introduce utility-shaped advantage esti- mation, which incorporates graph-derived utility into advantage computation without ar- chitectural changes and is compatible with any advantage-ber
UTILITY-BASED ADVANTAGE SHAPING:
policy-gradien. method.
SHAPING: We provide theoretical guarantees showing that by decaying the shaping influence as the policy improves, we preserve long-horizon convergence properties of Proximal Policy Optimization (PPO) (Schulman et al., 2017a) and correct any inaccuracies L.M outputs.
CONVERGENCE-COMPATTE
EMPIRICAL VALIDATION ACROSS BENCHMARKS: We demonstrate empirically that MIRA improves sample efficiency over RI. and hierarchical baselines and reaches final performance comparable to methods requiring continuous LLM supervision (Zhou et al,, 2023; Bhambri et al., 2024), while using far fewer online queries.
The remainder of this paper is organized as follows. Section details MIRA's architecture; Sec- tions 3 and 4 present experimental setup and results across multiple benchmarks; and Section concludes with discussion our findings and possible directions for future work.
METHODOLOGY
We now describe the design of MIRA, whose development is guided by two desiderata: (1) improve early learning by incorporating task-relevant priors from an LLM, (II) minimize reliance on con- tinuous real-time LLM supervision to ensure scalability and maintain autonomous policy learning. MIRA is built on the standard policy-gradient formulation for reinforcement learning (Appendix A).
2.1 MEMORY GRAPH DESIGN
The agent maintains an evolving memory graph that organizes information drawn from both LLM suggestions and agent rollouts. Nodes of the graph represent decision-relevant context, and edges encode the hierarchical decomposition goals into subgoals provided by the LLM (Figure 2). This structure can be expressed as
g f
} U (RE) U (g)). (1)
Each trajectory node j consists of a partial observation Or, and an action ary. It is also associated with a goal term & & E Lgy,.g kiz indicating either final goal (g,) or an ab- stract subgoal (x²*) that the trajectory is intended to com- plete. In addition, the node stores an estimated (sub)gool re- ward fj for the action sequence, reflecting progress toward completing its associated subgoal, and confidence score e, derived from the LLM-derived statistics. The second set of nodes (KrH= represents subgoals к2 provided by the L.LM from the environment description. The final term (g,)
Figure 2: MIRA's evolving mem- ory graph. Trajectory segments тy are grouped under subgoal nodes Ky- Subgoals can be shared across multi- ple final goals, enabling reuse of com- mon behaviors.
denotes the agent's target goal(s).
3

---

Published as a conference paper at ICLR 2026
The graph is initialized with offline LLM priors and evolves training progresses. Nodes added or updated when the agent produces trajectory segments that either introduce novel segment for a known (sub)goal achieve higher estimated return than the existing entry for the same (sub)goal. Existing nodes can also be updated when the agent's experience strengthens entries that were initially derived from low-confidence LLM outputs. Online LLM suggestions may also be added as new nodes when available, provided they pass screening; an online query is triggered only when the rollout utility signal remains near zero for several consecutive episodes, indicating that the current graph offers no helpful guidance. In such cases, the current policy is either exploring regions unsupported by the memory graph or that the graph itself is missing useful segments. Nodes are pruned when they remain unused for fixed horizon of episodes. Each node maintains an access counter updated whenever it contributes to the utility; nodes whose counters do not change within this window are removed, reflecting reduced relevance to recent rollouts. Offline LLM nodes, though generally stable, may also be removed when rendered obsolete. Additional details about graph construction and environment-speciff examples are given in Appendix E. This process keeps the graph compact and adaptive, while ensuring that any low- low-quality or misleading prior segments naturally fade higher-return agent-generate trajectories replace them lead to their pruning.
2.2 OFFLINE AND ONLINE GUIDANCE
MIRA incorporates two complementary forms of LLM guidance, accessed either offline prior to training or online during training. Offline outputs are generated using full access to the task descrip- tion, providing trajectory segments and subgoal decompositions that initialize the memory graph with structured priors. Offline nodes accelerate early exploration and remain a persistent baseline source of guidance that complements the adaptive updates introduced by online LLM queries.
Online suggestions are incorporated during training when the agent fails to obtain useful guidance (i.e., when the rollout utility remains near zero) from its memory graph for several consecutive episodes. The LLM is constrained to the same partial observability as the agent and, when trig- gered, may return plans that correspond to short trajectories. Alternatively. it may provide control signals that bias the action preferences over an extended horizon until the current task segment is completed. All online outputs are passed through the Screening Unit, which discards low-confidence suggestions. Accepted plans are grafted into the memory graph as new trajectory segments, while accepted control signals bias the policy through soft logit injection, i.c., adding a bounded penalty to the logits of discouraged actions. This penalty induces only a soft preference before the softmax and cannot collapse the action distribution. PPO's clipped objective controls the update size, ensuring that the injected bias functions lightweight guidance that the critic can override when it strongly disagrees with the value estimates (Biza et al., 2021).
SCREENING UNIT. To ensure reliability, online outputs are passed through a lightweight Screening Unit designed to reduce hallucinations and reasoning failures (Ji et al., 2023a; Bubeck et al., 2023;
Wang et al., 2022; Zhao et al., 2021). Confidence is estimated in two complementary ways. When token-level likelihoods are available, we compute the geometric mean of the per-token probabilities of the completion as a confidence measure. When such likelihoods are unavailable or incomplete (e.g.. only top-k likelihoods are provided), we instead estimate confidence by sampling multiple in- dependent completions and measuring agreement across them. applied to retain only those outputs that appear reliably across the sampled completions. Sugges- tions that fail to meet a fixed threshold under either criterion are discarded. While this procedure does not eliminate all high-confidence errors, it serves as an effective filter that reduces the risk of hallucinated or low-quality outputs. The screened outputs, referred to as healthy grafts in Figure 1, incorporated into the memory graph as new nodes to further help the policy learning.
majority-consistenee test is then
2.3 UTILITY SIGNAL COMPUTATION
Utility is defined at the level of individual state state-action pairs and is computed from the same roll- outs used for advantage estimation under the current policy. Each statc-action in the trajectory = {(ot,,e) {)){E is matched against the corresponding ttae-acttion pairs (or, ag) in the stored trajectory Tin The appropriate memory node m is selected based on the environment instance (e.g., the seed-specific layout) in that training iteration. We then compute the utility for each pair t as:
U² - C. Pm p(go: Sm) I ((ot, at); (or, (00,0e))r )..))
4

---

Published as a conference paper at ICLR 2026
Steps that do not match any stored trajectory segment receive zero utility. The similarity function ((..)) measures how closely the agent's 's behavior aligns with the stored trajectory. It incorporates both action agreement and spatial consistency, such as overlap in grid positions or directional align- ment in tabular settings (A (Algorithm 3). To capture semantic structure, the raw similarity score is weighted by goal-alignment factor p(:;-). Each subgoal description specifies target object or region and a high-level action applied to it; simple rulc-based parsing yields a paired entity and action-phase token. The Jaccard similarity between the entity-phase token sets extracted from the agent's s target subgoal and each memory entry defines p. This weighting increases the influ- ence of memory entries that share underlying entities or action phases with the target subgoal and downweights matches corresponding to unrelated parts of the task (Algorithm 4). Thus, a transition contributes to utility only when both its behavioral similarity and its semantic alignment with the relevant subgoal are high. Finally, the score is modulated by the confidence Co and estimated reward fm attached to the memory node (Algorithm 5). This formulation also helps limit the influence of incorrect LLM guidance that may have passed screening. When an LLM suggestion is inaccurate, its influence naturally diminishes: such segments typically contribute little utility since their estimated reward is low or their similarity score remains small, reflecting weak positional alignment in- consistent action patterns between the agent's rollout and the stored LLM-derived segment. Further details on the utility computation are presented in Appendix F.
2.4 ADAPTIVE ADVANTAGE SHAPING
We incorporate memory memory-derived utility into the policy update by augmenting the standard advantage term. Algorithm outlines the shaped PPO update. At iteration k, trajectories D; {( a,, rt) are collected under the policy T0,- The rollout batch is split into minibatches B for multiple gradient steps. The likelihood ratio r. compares new and old policies, and the clip parameter 6 constrains Tr within (1 + Ex) as a soft trust region.
The advantage function in policy gradient methods, denoted by Ar at given time f, quantifies how favorable an action a₁ is relative to the average action at state st. It drives learning by returns and suppressing those that fall short.
reinforcing actions that have tigher-than-expecter However, during early training the critic is poorly calibrated due to limited exploration, of- ten producing nearly uniform value estimates across actions (Henderson et al., 2018). As are- sult, the estimated advantages A, provide weak learning signals, even when the agent is follow- ing behavior that is meaningfully directed to-
Algorithm 1 Shaped PPO actor (changes)
for k 0, 1.... do
Collect D3 = {(s,, Tt)} using Te,
Compute A, and Ug from rollouts
A. 7 A E.U,
for epoch = 1 to K do
ward the task. This issue is particularly pro- nounced in sparse-reward settings or tasks with delayed feedback, where the critic lacks suf- ficient signal to distinguish between promis- ing and unproductive behaviors. In such cases, the estimated advantage tends to be near-zero or highly noisy for most timesteps, especially early in training (Figure 15).
for minibatch B c D4 do No, (arls:) min(re, 1 (
(6) To ( (s))s
Chappe (To) = E
0 + 0 + as Vo chapped(To)
end for end for end for
To address this, we introduce shaped advantage
A, = A + U,U,, 0< n. S 1, & S 8n. de [0, 1), lim Te = 1, lim E. = O. 10000
(3)
(-30
This formulation preserves the fundamental role of the advantage function, while refining it with utility-based guidance. forms cooperative process between eritic predictions and the memory- derived utility. The critic provides an estimate based on learned reward prediction and bootstrapping, while the utility term injects an inductive bias derived from language-guided priors. Together, they form a joint estimator in which each component compensates for the other's limitations without dis- torting policy optimization. When the critic signal is weak due to insufficient value discrimination, the resulting gradients are uninformative and impair the agent's ability to bootstrap from sparse or delayed rewards. The utility term provides additional directional guidance aligned with task objec- tives, accelerating learning by compensating for weak or flat gradients (Figure 15, Appendix G).
5

---

Published as a conference paper at ICLR 2026
As training progresses and the policy becomes more accurate, the critic's advantage estimates A become more reliable. Accordingly, the shaping weight & is annealed to reduce the direct influence of the utility term and π) is ramped toward 1 over training. Since the LLM imperfect, annealing & prevents inaccuracies in those signals from being preserved in the asymptotic regime. This ensures that the final policy is optimized with respect to the true reward function R and remains consistent with PPO's stability guarantees. Early in training, however, & remains large enough for the utility to accelerate exploration using the LLM's possibly imperfect, but still useful, prior knowledge. As & decays, any suboptimalities in these priors are naturally learned away, yielding the benefits of shaping during the sparse-rewas reward phase, without biasing long-run behavior or altering the policy critic structure. See remark C.4 for more explanation.
signals can be
Before turning to experiments, we provide an interpretive perspective on how adaptive advantage shaping affects optimization in sparse-reward regimes and in Appendix establish that the proposed shaping mechanism preserves the policy improvement property PPO under standard boundedness and scaling assumptions, which we formally enumerate in Appendix C.I. More broadly, the method remains compatible with policy gradient algorithms that relies on advantage estimation, offering a general mechanism for integrating minguage-derived priors into RL.
Theorem 1 (Non-Vanishing Updates in Sparse- Sparse-Reward Regimes), Define the shaped surrogate Eshoped (0) = L Ve log (e(ee)s)) As} and the PPO surrogate (PP) = E[Va log Ne pe(ac(s)) Ar]. Consider a training iteration such that the expected magnitude of the PPO advantage small, i.., E[]AA]] & E for some A = O. Under Assumptions 1-3, the expected norm of the shaped PPO policy update satisfies
Cahaped
2
&
|| -((A))
where CV V log To (orls)) U.].
Proof. Deferred to Appendix C.
3 EXPERIMENTAL SETUP
We validate our method through extensive experiments im- plemented using the RLlib (Liang et al., 2018). Our eval- uation focuses performance gains, sample efficiency, and the computational overhead introduced by LLM inte- gration. The objective is characterize the benefits and trade-offs of incorporating LLM guidance in RL, including how different levels of LLM capabilities influence the pol- icy learning dynamics and final policy quality.
3.1 SIMULATION PLATFORM
Figure 3: Evaluation environments.
We consider six distinct environments, which are selected span discrete vs. visual inputs, short- vs. long-horizon dependencies, reversible vs. irreversible dynamics, and with vs. without perceptual distractors, forming a compact yet representative benchmark for sparse-reward RL.
Top: REDBALL (navigation to tar- get), LAVACROSSING (long-horizon navigation with irreversible hazards), DOORKEY (sparse reward with key-goal dependency).
GYMNASIUM TOYTEXT. Gymnasium (Amoldo et al., 2024) provides simple tabular environments for controlled analysis of learning dynamics in low-dimensional settings. Despite their simplicity, these environments feature sparse rewards and require strategic exploration, making them suitable for isolating the early-stage benefits of memory-guided utility shaping. We include FROZENLAKE as a minimal benchmark where PPO reliably converges to the optimal policy, en- abling to verify that MIRA preserves convergence while accelerating early learning.
Bottom: REDBLUEDOOR (sequence- sensitive toggling), DISTRACTED DOORKEY (distractor-rich variant with key-goal dependency).
MINIGRID AND BABYAL. MiniGrid (Chevalier-Boisver et al., 2023) and BabyAI (Chevalier- Boisvert al., 2019) suites of lightweight, procedurally generated environments designed to

---

Published as a conference paper at ICLR 2026
evaluate exploration and planning in partially observable, sparse-reward settings. We use these tasks to assess the effectiveness of advantage shaping in long-horizon decision- making environments that require reasoning under uncertainty and robustness to irrelevant stimuli. We include five tasks, lected to cover diverse challenges involving planning, credit assignment, and distraction resilience (Figure 3). We pixel-based observations (RGB images) rendered from the environment as the policy inputs, to introduce perceptual complexity and evaluate agent performance under a more realistic observation setting.
3.2 BASELINE METHODS
PPO (RL BASELINE). We train a tabula PPO agent (Schulman et al., 2017a) that learns purely from environment interaction and rewards. Network architecture, PPO hyperparameters, and rollout settings are held fixed across all methods for fair comparison.
HIERARCHICAL RL. We include hierarchical reinforcement learning (HRL) (Matthews et al., 2022) baseline that uses pre-trained LLM option policies for temporal abstraction.
LLM-RS. We consider the method of (Bhambri et al., 2024), which we refer to as LLM-RS. This approach queries the LLM in real time generate plans for potential-basee reward shaping, with verifier refining them for valid action sequences.
LLM4TEACH. We include LLM4Teach (Zhou al., 2023) representative teacher-based approach. It employs pre- trained LLM as policy teacher and guides the RL agent through policy distillation, and is among the state-of ootte--f-th-arr theheetrt methods in this category.
EXPERIMENTAL RESULTS
To assess how effectively MIRA addresses the challenges raised in the introduction, we structure our core experiments around three guiding questions. These questions examine whether utility-shaped advantages compensate for PPO's weak early gradients in sparse-reward, long-horizon, and par- tially observable settings, and whether amortizing limited LLM guidance into a persistent memory structure provides sufficient return while supporting scalability and avoiding latency. Formally:
Q1: How does MIRA improve early learning efficiency and convergence relative to PPO, even when PPO's final performance is competitive?
Q2: How well does MIRA perform in long-hond horizon environments that demand extended exploration and multi-step reasoning?
Q3: How effectively does MIRA translate a limited number of LLM queries into perfor- mance gains compared query-heavy methods?
To isolate the contribution of individual components in MIRA's design, we further conduct ablations organized around three additional questions. These probe how online queries complement offline- initialized memory, and how sensitive MIRA is degraded, incorrect, or stylistically varied LLM outputs. Formally:
Q4: How do online LLM queries improve learning, beyond what offline memory provides?
Q5: How does MIRA handle late-stage exposure to degraded LLM guidance once its mem- ory is well-formed?
Q6: How do variations in LLM reasoning affect memory and downstream results?
Appendix G provides additional results, including evaluations on unseen seeds to assess generaliza- tion, wall-clock analyses of LLM-query overhead (Figure 16), and measurements of memory growth (Figure 17). We also report ablations on the screening threshold (Figure 14) and prompt wording (Figure 13), each examining its effect on overall learning behavior and addressing aspects of Q5. In addition, supplementary plots from sweeps over shaping weights (Figure 15) are included to analyze their impact early-stage learning dynamics and reward progression.
7

---

Published as a conference paper at ICLR 2026
RedBall
Doorkey
LavaCrussing
RociBhoDoor
Men Retin
Zone
lecation
400
600
2010
Necation
496
600
2000
becaiin
200
hecuion
100
2001
Baseline
Hienechical RL
1005
RodBBll
DooeKey
LavaCrossing
RedBlueDoor
2300
Beralian
Figure 5: Mean return (top) and success rate (bottom) across four MiniGrid and BabyAI tasks. MIRA consistently outperforms both baselines, achieving faster learning, higher asymptotic return, and greater success rates. These results are obtained with small LLM budget, using fewer than ten offline prompts to build memory graphs plus infrequent online queries to guide exploration.
4.1 TABULAR BENCHMARK AND PARTIALLY OBSERVABLE TASKS
We evaluate two variants of MIRA on FROZENLAKE-8X8 an offline-only version and an online-only version, and compare them to the PPO baseline, averaging results over four seeds. In the offline variant, three zero-shot GPT-o4-mini queries generate an initial memory graph. The LLM observes the grid layout (matching the agent's 's full observability) but does not not receive the slipperiness probability, which is hidden from both. As shown in Figure 4, this initialization results in faster early learning, and the offline variant maintains a higher return than PPO throughout, and the online-only variant in the first 1K iterations. The online-only variant begins with empty memory graph, without any global information about the map and issues LLM queries during training.
Q1:
Since the layout is deterministic and the action set is small, these online queries can infer short segments from the agent's rollouts and popu- late the memory graph as training progresses. This provides faster improvement rate than the offline variant, though it requires more I.LM queries. PPO eventually catches up, and by convergence, the asymptotic returns of all three methods become comparable. During training, the shaping signal primarily affects the early it- erations. As the policy improves, 17 increases, &, decreases, i.e. the shaping ratio ò, εi/n decays, reducing reliance on memory. Un- der standard stochastic approximation condi- tions (Kushner & Yin, 2003), this decay keeps the critic error within an (6))) neighborhood of the true value, which vanishes as ó; + O.
Frozen Lake
of
Mon gonen
Shaping
MIKA oelyoe SIEK oullie
Bocliee
1000
heration
2000
100
Deratin
1000
Figure 4: Mean return on FROZENLAKE (left): Both MIRA variants improve early-stage learning relative PPO, while PPO eventually attains comparable asymptotic return. Evolution of shap- ing terms пe. & and ratio 5, (right): ó, decays during training, ensuring convergence as 5, > 0.
Q2: We next evaluate MIRA on five tasks designed to isolate distinct challenges in sparse and partially observable environments. Figure 5 shows mean return and success rate across the four tasks, with performance averaged over four different seeds. In simpler tasks such as REDBALL, PPO shows moderate early gains but plateaus well below optimal performance. Although hierar- chical RI. eventually catches up, MIRA reaches optimal returns in under half the training iterations. In LAVACROSSING, PPO fails to improve beyond near-zero success, indicating ineffective explo- ration. Hierarchical RL improves steadily but converges more slowly than MIRA. In more complex tasks such as DOORKEY and REDBLUEDOOR, MIRA achieves substantially higher success rates, approximately twice those of HRL, while also converging faster.
These gains are achieved with limited LLM query budget that combines offline and infrequent online access. Offline queries scale with task complexity. In REDBALL, four zero-shot prompts to GPT-o4-mini are sufficient to build useful memory graph, while DOOR KEY requires about seven
8

---

Published as a conference paper at ICLR 2026
queries that mix few-shot and zero-shot prompts. Online queries are budgeted separately and also vary with task complexity. In REDBALL, about seven online queries suffice to suppress irrelevant actions throughout training. In REDBLUEDOOR, queries are triggered more frequently early in training to help interpret partial observations and suggest short sequences, such as turning, that align the agent with the door. Once the red door is discovered and toggled, the offline memory becomes sufficient. In this task, rooms behind the doors serve only as distractors; baseline agents, including hierarchical RL, often waste steps exploring them. As shown in Figure (lower right), HRL achieves higher success rates than PPO but yields similar average return in the beginning due to suboptimal trajectory use. By contrast, MIRA avoids such inefficiencies by focusing on goal-relevant behavior earlier in training.
Q3: To further evaluate MIRA, we compare it to LLM4Teach and LLM-RS in the custom variant environ- ment DISTRACTED DOORKEY. We also include Sole LLM baseline, where GPT-o4-mini executes plans under full observability without learning. Figure shows mean return progression selected training checkpoints. For Sole LLM, we report average return over 100 seeds to demonstrate that the task is L.L--solvable and that its outputs provide useful structural guidance. The accom- panying bar chart reports amortized return per cumulative LLM query under two fixed budgets (10 and 20 queries, both below the usage of any method), quantifying how each method translates queries into performance gains.
Meon Retam Progrestion
Per Return
irs
1kit
Per 10
Skars
114
68
MIRA
ROOM
LLM
Solo 11M
Mcan return (left): LLM4Teach shows faster early gains, while MIRA steadily improves and matches its final return with fewer queries. LLM-RS benefits early from reward shaping but plateaus lower. Return per LLM query (right): Under two query budgets, MIRA achieves higher efficiency.
Figure
MIRA achieves higher query efficiency than both LLM4Teach and LLM-RS, using about 30 queries per run (seven offline and 20 + 3 online) to obtain higher return per query. In contrast, LLM4'eeach issues dense supervision, querying the LLM once on every state-action-reward triplet within training batches, which in our experimental setup corresponds to more than 500 queries before the policy stabilizes. LLM-RS, which uses
LLM-generated reward code, queries once per layout, totaling over 50 calls in our setup. While lighter than L.M4Teach, this still requires layout-level access throughout training. Despite its heav- ier budget, LLM4TTeac achieves comparable final performance to MIRA, while LLM-RS fails to match MIRA's return. Notably, LLM-RS outpaces MIRA early due reward shaping, but falls be- hind later. LLM4Teach shows an early advantage through front-loaded queries, but at a significantly higher cost. Tables Appendix G report results on unseen evaluation seeds to assess generalization.
4.2 ABLATION STUDIES
Q4 (ONLINE QUERY FREQUENCY): We vary the number of online LLM queries issued during training of DOOR assess how constrained usage affects learning efficiency and final perfor- mance. Each agent begins with the same memory graph, initialized from identical offline queries, isolating the contribution of dynamic LLM input from that of static memory initialization. We com- pare MIRA under three online budgets: zero, a mid budget of 10 queries, and a high budget of 20. As shown in Figure (left), more frequent online access accelerates learning, with the large-budget variant achieving optimal return in fewer steps (Table 1, Appendix G). Even with just 10 online calls, MIRA substantially outperforms the offline-only variant. Nevertheless, MIRA (offline) still yields notable boost over baseline PPO, indicating that static memory alone can provide meaning- ful guidance when dynamic access is unavailable. Given this noticeable improvement over PPO, the offline-only variant is practical choice when run-time LLM access is restricted or when latency makes online queries impractical, since it relies entirely on the memory graph constructed offline. When limited run-time access is available, the online variant becomes preferable because dynamic queries update the memory during exploration and accelerate learning, at the cost of additional wall- clock time associated with LLM calls (Figure 16, Appendix G).
Q5 (UNRELIABLE LLM OUTPUTS): We evaluate a scenario where the LLM is swapped at a later training stage and the screening unit is disabled only for this final online phase in DISTRACTED

---

Published as a conference paper at ICLR 2026
Query Prequency
Iu Unreliable LIM
LIM Modols
Neanegg
Miii llege lostget twsthodlpot Rucilli
Mar Beaim
Sep Coucs
Bronice
1000
Beratoo
Figure 7: Ablation Studies. Query frequency (left): Agents share the same offline memory but vary in online budgets. More queries accelerate learning, with high-budget agents achieving optimal re- turns more quickly. Unreliable LLM (middle): With identical offline memory, screening is disabled and queries are swapped from GPT-o4-mini to Gemini Pro only in the late phase. Performance remains stable in the late phase, indicating reduced dependence have matured. LLM models (right): Agents differ only in the L.LM used for memory. Performance differences reflect divergent reasoning styles: Gemma3 induces inefficient checking, Claude favors cautious exploration, while Gemini Pro and 04-mini enable faster learning.
online guidance once policy
DOORKEY environment. All agents share the same difline-initializer memory graph and use GPT- 04-mini with screening during earlier online queries. In the final stage, we replace the LLM with Gemini Pro and omit screening. By this point, MIRA has accumulated sufficient experience and memory, allowing it to tolerate low-confidence or incorrect suggestions without collapsing perfor- mance. We prompted both LLMs with a scenario where the agent has already explored thoroughly and confirmed no key is present (implying it was already collected, since inventory is unobserved). When asked which action to down-weigs GPT-o4-min -oi-io gave consistent suppression, whereas Gemini returned a misaligned alternative. As shown in Figure 7 (middle), MIRA remains stable under degraded guidance, though convergence slows and final return drops slightly. Details of the L.M responses are shown in Figure 12, Appendix D).
Q6 (REASONING AND PERFORMANCE): We assess MIRA's sensitivity to the choice of language model by replacing GPT-o4-mini with alternatives such as GPT-4o (OpenAI, 2024), Claude Son- al., 2024), Gemini 2.5 Flash and Pro (Chen et al., 2024). All models go through the same process to ensure comparability. Unlike the ablation done before, the model swap is applied from the beginning of training. As shown in Figure 7 (right), the reasoning style shaping the memory graph strongly impacts downstream RL performance. For instance, Gemma3 performs poorly because it recommends checking the door after every pickup, leading to wasteful steps. Claude adopts an exploratory policy that yields slow but eventual progress, showing early improvement followed by plateauing, but it eventually re- covers as the memory is dynamic. GeminiPro and GPT-o4-mini both enable fast early learning, but o4-mini's memory includes detours that prove beneficial later, ultimately reaching the highest asymptotic return. These differences highlight how the reasoning processes behind LLM outputs directly influence MIRA's long-term policy quality. Figure 11 in Appendix D presents the reasoning traces produced by different L.LM models used.
net 4 (Anthropic, 2024), Gemma 3 27B (Ananthanarayanan
CONCLUSION AND FUTURE WORK
We propose MIRA, a reinforcement learning (RL) framework that integrates large language model (LLM) guidance via a memory graph built from high-return trajectories and LLM--iffrred infor- mation. By shaping advantage estimates with a utility signal derived from this memory, MIRA accelerates early learning without requiring continuous LLM supervision. Theoretical and empiri- cal results on sparse-reward tasks confirm improved sample efficiency and preserved convergence. Limitations of the current design are discussed in Appendix H. Extending MIRA to continuous- control and vision-based domains is a natural next step. Prior LLM-guided robotics work shows that semantic subgoals can be grounded in continuous spaces through learned affordances or visual embeddings (Brohan et al., 2023; Huang et al., 2022). In such settings, MIRA's discrete similar- ity function can be replaced with comparisons in a latent representation space, e.g., using encoder features or R3M-style embeddings (Nair et al., 2022), while memory scalability can be maintained through clustered or hierarchical organization. These adaptations would allow MIRA's memory graph and tility-based shaping to apply beyond grid worlds and support Longer-horizen tasks with richer state spaces. Extending MIRA to multi multi-goal domains such as Crafter (Hafner et al., 2023), where reusable subgoal structure is prominent, another promising direction.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported part by the National Science Foundation (NSF) under grants CNS- 2533813 and CNS-2312761 and the Office of Naval Research under grant N000142412073.
REPRODUCIBILITY STATEMENT
We have taken several steps to ensure reproducibility of our results. All theoretical assumptions and complete proofs are included in Appendix C. Appendix D details the environment specifications and the exact LLM prompts used for both offline and online queries. Appendix I lists the full set of hyperparameters for MIRA across every evaluated environment. We also provide pseudocodes for all proposed algorithms in Algorithms through 5, ensuring clarity and transparency despite their straightforward implementation. The code available at paper's webpage tttps: : /nar jarjesno github. i //MIII /MIRA/. Together, these materials supply all information necessary reproduce experiments and verify the claims of the paper.
REFERENCES
Ademi Adeniji, Amber Xie, Carmelo Sferrazza, Younggyo Seo, Stephen James, and Pieter Abbeel. Language reward modulation for pretraining reinforcement learning. arXiv preprint arXiv:2308. 12270, 2023.
Rajat Ananthanarayanan. Jaya Bhosale, Aakanksha Chowdhery, Danny Driess, and al. Gemma and 3: Open models based on gemini research and technology. arXiv preprint arXiv:2503.1978 2024. URL hwtps: / a/////rre arxiv. htt.://rrgaa//////aeeee /2503/b///25..1197
Anthoopic. Introducing the claude 4 model family. https: http::/wwwwwww////eeeee www.aaait.ttacooooooooo cotp:p/.....ccc/.........e claude-4, 2024. Claude 3.5 Sonnet released June 2024.
Arjun KG Arooldo et al. standard interface for reinforcement learning environments. arXiv preprint 1rXXi:2407.170333 2024. URL ht https://axxixv orw./rrg//o.... /ttp:/ab/224077003333
Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. Hallulens: Llm hallucination benchmark. arXiv preprint arXiv:2504. 17550, 2025.
Shreyas Basavatia, Keerthiram Murugesan, and Shivam Ratnakar. Starling: Self-supervised train- ing of text-based reinforcement learning agent with large language models. arXiv preprint arXiv:2406.0583 2024.
Edward Beeching, Jilles Dibangoye, Olivier Simonin, and Christian Wolf. Egomap: Projective mapping and structured egocentric memory for deep rl. Joint European conference on machine learning and knowledge discovery in databases, pp. 525-540. Springer, 2020.
Siddhant Bhambri, Amrita Bhattacharjee, Durgesh Kalwar, Lin Guan, Huan Liu, and Subbarao Kambhampati. Extracting heuristics from large language models for reward shaping in reinforce- ment learning. arXiv preprint 2xXh:22405.15744 2024.
Ondrej Biza, Dian Wang, Robert Platt, Jan-Wille Willem van de Meent, and Lawson LS Wong. Action priors for large action spaces in robotics. arXiv preprint arXiv:2101 04178, 2021.
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint a60iv:1606.0 04460. 2016.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. Conference on robot learning, pp. 287-318. PMLR, 2023.
Sébastien Bubeck, Varum Chandrasekaran, Ronen Eldan, Bruno Huster, Ananth Raghunathan, Jonathan Ray, and Yang Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. In arXiv preprint arXiv:2303. 127 2023.

---

Published as a conference paper at ICLR 2026
Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, and Yun Li. Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods. IEEE Transactions Neural Networks and Learning Systems, 2024.
Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Transdreamer: Reinforcement learning with transformer world models. arXiv preprint arXiv:2202.0948)) 2022.
Zhenzhong Chen, Siddharth Purohit, William Wang, Rohan Anil, and et al. Gemini 1.5 and 2.5: Unlocking multimodal understanding and reasoning in single model. arXiv preprint arXiv:2507.06261 2024. URL https: /ttpxaaxiivvv.///////a./eaaaa wttpv///rrggbb///////// 157.0000.2 06261.
Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. BabyAl: First steps towards grounded language learning with human in the loop. In International Conference Learning Representations (ICLR), 2019.
URL httph::pp///cc https: /tap:ss///ppeeeviewe net forum? id=:JexCoocc..
Maxime Cheveeier-Boisvert,, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR,
(bs/2300.13831, 2023. URL https: http://xxvvvv../ htt.gg/ab//336... 1331..
Kun Chu, Xufeng Zhao, Cornelius Weber, Mengdi Li, and Stefan Wermter. Accelerating reinforce- ment learning of robotic manipulations via feedback from large language models. arXiv preprint arXiv:2311.02379, 2023.
Murtaza Dalal, Tarun Chiruvolu, Devendra Chaplot, and Ruslan Salakhutdinov. Plan-seq- learn: Language model guided rl for solving long horizon robotics tasks. arXiv preprint arXiv:2405.00533, 2024.
Ishita Dasgupta, Christine Kaeser-Chen, Kenneth Marino, Arun Ahuja, Sheila Babayan, Felix Hill, and Rob Fergus. Collaborating with language models for embodied reasoning. arXiv preprint arXiv:2302. 00763, 2023.
Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Exploration-guiddr reward shaping for reinforcement learning under sparse rewards. Advances in Neural Information Processing Systems, 35:5829-5842, 2022.
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, pp. 8657-8677. PMLR, 2023.
Rasool Fakoor, Pratik Chaudhari, and Alexander J Smola. P3o: Policy-on policy-off policy opti- mization. In Uncertainty in artificial intelligence, pp. 1017-1027. PMLR, 2020.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35: 18343-18362, 2022.
Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arXiv preprint arXiv:2410.15115, 2024.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. Proceedings of the IEEE/CVE conference on computer vision and patlern recognition, pp. 18995-19012, 2022.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint S.ti::2301.041000 2023.
12

---

Published as a conference paper at ICLR 2026
Matthew Hauskmecht, Prithviraj Ammanabrolu, Mare-Alexandre Côté, and Xingdi Yuan. Interac- tive fiction games: colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7903-7910, 2020.
Matthew Hausknecht and Peter Stone. Deep recurrent Q-learning for partially observable MDPs. In AAAI symposia, volume 45, pp. 141, 2015.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Hengyuan Hu and Dorsa Sadigh. Language instructed reinforcement learning for human-ai coordi- nation. In International Conference on Machine Learning, pp. 13584-13598. PMLR, 2023.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint 1iXiv:2207700500. 2022.
Ziwei Ji, Nayeon Lee, Richard Frieske, Tiezheng Yu, Dan Su, Yanlin Xu, Etsuko Ishii, Yeon Seon Bang. Andrea Madotto, and Pascale Fung. survey of hallucination in natural language genera- tion. ACM Computing Surveys (CSUR), 55(12): 2023a.
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating Ilm hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1827-1843, 2023b.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve arXiv:2310.06770, 2023.
github issues? arXiv preprint
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-33, 1998.
Woo Kyung Kim, SeungHyun Kim, Honguk Woo, et al. Efficient policy adaptation with contrastive prompt ensemble for embodied agents. Advances Neural Information Processing Systems, 36: 55442-55455 2023.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:21 2021.
Hanna Kurniawati. Partially observable markov decision processes and robotics. Annual Review of Control, Robotics, and Autonomous Systems, 5(1)2253-277, 2022.
Harold J Kushner and G George Yin. Stochastic approximation and recursive algorithms and appli- cations. Springer, 2003.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint 20X00:2303.00001, 2023.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feecback-efficient interactive rein- forcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091, 2021.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gon- zalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning- In International conference on machine learning, pp. 3053-3062, PMLR, 2018.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint 22X10:2009.07753. 2022.
Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning model the world with language. arXiv preprint arXiv:2308.013399 2023.
13

---

Published as a conference paper at ICLR 2026
Long-Ji Lin. Self-improving reactive agents based reinforcement learning, planning and teaching. Machine learning, ((3):293-321, 1992.
Jung Chun Liu, Chi-Hsien Chang, Shao-Hur Sun, and Tian-Li Yu. Integrating planning and deep reinforcement learning via automatic induction of task substructures, In The Twelfth International Conference on Learning Representations, 2024.
Jung-Chun
Ruishan Liu and James Zou. The effects of memory replay in reinforcement learning. In 2018 56th annual allerton conference on communication, control, and computing (Allerton), pp. 478-485. IEEE, 2018.
Wenhao Lu, Xufeng Zhao, Sven Magg, Martin Gromniak, Mengdi Li, and Stefan Wermter. A closer look reward decomposition for high-level robotic explanations. In 2023 IEEE International Conference on Development and Learning (ICDL), pp. 429-436. IEEE, 2023.
Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via coman-in-thh-lowe reinforcement learning. arXiv preprint arXiv:2410.21844 2024.
Haozhe Ma. Fangling Li, Jing Yu Lim. Zhengding Luo, Thanh Vinh Vo, and Tze- Yun Leong. Catch- ing two birds with one stone: Reward shaping with dual random networks for balancing explo- ration and exploitation. In forty-second International Conference on Machine Learning. 2025.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 12931, 2023.
James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. In International conference on machine learning, pp. 2285-225-2222 22944. PMLR, 2017.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances Neural Information Processing Systems, 36:46534-46594, 2023.
Michael Matthews, Mikayel Samvelyan, Jack Parker-Holder, Edward Grefenstette, and Tim Rocktäschel. Hierarchical kickstarting for skill transfer in reinforcement learning, arXiv preprint arXiv:2207. 1554, 2022.
Lina Mezghani, Piotr Bojanowski, Karteck Alahari, and Sainbayar Sukhbaatar. Think before you act: Unified policy for interleaving language reasoning with actions. arXiv:2304. 1063, 2023.
arXiv preprint
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world mod- els. arXiv preprint urXiv:2209.00582 2022.
Volodymyr Mnih Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PmLR, 2016.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: univer- sal visual representation for robot manipulation. arXiv preprint arXiv:2203. 2022.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone.
Curriculum learning for reinforcement learning domains: A framework and survey, Journal of Machine Learning Research, 21 2020.
Narjes Nourzad, Jared Coleman, Zhongyuan Zhao, Bhaskar Krishnamachari, Gunjan Verma, and Santiago Segarra. Actor-twin framework for task graph scheduling. In The Seventeenth Workshop on Adaptive and Learning Agents, 2024.
OpenAl. Gpt-4o. https: /ttp:///pp/aaa/aaaeeee www. com/index/ggtt-aeeee 2024. Accessed July 2025.
14

---

Published as a conference paper at ICLR 2026
Fabian Paischer, Thomas Adler, Vihang Patil, Angela Bitto-Nemling, Markus Holzleimeer Sebastian Lehner, Hamid Eghbal-Zadeh, and Sepp Hochreiter. History compression via language models in reinforcement learning. In International Conference on Machine Learning, pp. 17156-17185. PMLR, 2022.
Fabian Paischer, Thomas Adler, Markus Hofmarcher, and Sepp Hochreiter. Semantic HELM: A human-readable memory for reinforcement learning. Advances Neural Information Processing Systems, 36:9837-9865, 2023.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightted regression: Simple and scalable off- ooficy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
Rudra PK Poudel, Harit Pandya, Stephan Liwicki, and Roberto Cipolla. Recore: Regularized con- trastive representation learning of world model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22904 22913, 2024.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International conference on machine learning, pp. 2827-2836. PMLR, 2017.
Qian Qiu, Fanyu Zeng, Haigen Yang, Guanyu Xing, and Shuzhi Sam Ge. Memory-augmented deep deterministic policy gradient. In International Conference Social Roboties, pp. 41-52. Springer, 2024.
Yun Qu, Boyuan Wang, Yuhang Jiang, Jianzhun Shao, Yixiu Mao, Cheems Wang, Chang Liu, and Xiangyang Ji. Choices are more important than efforts: Llm enables efficient multi-agent explo- ration. arXiv preprint arXiv:2410.025: 10.02111, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task plan- ning. arXiv preprini urXiv:2307.066 135, 2023.
Jan Robine, Mare Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based world mod- els are happy with 100k interactions. arXiv preprint arXiv:2303.07009 2023.
Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310. /29211, 2023.
Graeme D Ruxton. The unequal variance t-test underused alternative student's t-test and the mann-whitney u test. Behavioral Ecology, 17(4):688 6600, 2006.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.00 wtt.//11..05952... 2015.
Sheila Schoepp. Masoud Jafaripour, Yingyue Cao, Tianpei Yang. Fatemeh Abdollahi, Shadan Golestan, Zahin Suflyan, Osmar R Zaiane, and Matthew E Taylor. The evolving landscape of Ilm-and -lm-integraatd reinforcement learning. arXiv preprint arXiv:2502.15214 2025.
John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient estimation using stochastic computation graphs. Advances in neural information processing systems, 28, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint (tl.:15066024388 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. In arXiv preprint arXiv:1707.06344 20174 7a. URL https: arxii.. htgg/bbs/7701 06347.
15

---

Published as a conference paper at ICLR 2026
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017b.
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference Robot Learning, pp. 1332- 1344. PMLR, 2023,
Ruizhe Shi, Yuyao Liu, Yanjie Ze, Simon S Du, and Huazhe Xu. Unleashing the power of pre- trained language models for offline reinforcement learning. arXiv preprint arXiv:2310.20587, 2023.
Kyriacos Shiarlis, Markus Wulfmeier, Sasha Salter. Shimon Whiteson, and Ingmar Posner. Taco: Learning task decomposition via temporal alignment for control. In International Conference on Machine Learning, pp. 4654-4663. PMLR, 2018.
Noah Shinn et al. Reflexion: Language agents with verbal reinforcement learning. In NeurlPS, 2023.
David Silver. Lectures on reinforcement learning. URL: teaching/ 2015..
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. L.LM-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2998-3009, 2023a.
Jiayang Song, Zhehua Zhou, Jiawei Liu, Chunrong Fang, Zhan Shu, and Lei Ma. Self-refined large language model automated reward function designer for deep reinforcement learning in robotics. arXiv preprint wrXiv:2309.06689 2023b.
Martin Stolle and Doina Precup. Learning options reinforcement learning. In International Symposium abstraction, reformulation, and approximation, pp. 212-223. Springer, 2002.
Theodore R Sumers, Mark K Ho, Robert D Hawkins, Karthik Narasimhan, and Thomas L Griffiths. Learning rewards from linguistic feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6002-6010, 2021.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Richard Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999.
SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint wwXiv:2401.01333, 6, 2024.
Akash Velu, Skanda Vaidyanath, and Dilip Arumugam. Hindsight-dice: Stable credit assignment for deep reinforcement learning. arXiv preprint arXiv:2307.1 1897, 2023.
Xu Wan, Wenyue Xu, Chao Yang, and Mingyang Sun. Think twice, act once: A co-evolution framework of Ilm and rl for large-scale decision making, arXiv preprint arXiv:2506.02522 2025.
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, et al. Efficient large language models: A survey. arXiv preprint 13X2::3312.03633, 2023.
Guiding Wang, Yugi Xie, Kun Jiang. Xiangyu Lu, Weihong Zhang, Haowei Lu, Yitao Xiong, Qian Li, Chuyuan Xu, Minggang Huang, and et al. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 2..305./621.. 16291, 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- cry, and Denny Zhou, coffisoonis improves chain of thought reasoning in language models. arXiv preprint arXiv:2203./117/ 2022.
16

---

Published as a conference paper at ICLR 2026
Yongdong Wang, Runze Xiao, Jun Younes Louhi Kasahara, Ryosuke Yajima, Keiji Nagatani, At- sushi Yamashita, and Hajime Asama. Dart-llm: Dependency-award multi-robot task decomposi- tion and execution using large language models. arXiv preprint arXiv:2411.09002 2024a.
Yufei Wang, Zhanyi Sun, Jesse Zhang. Zhou Xian, Erdem Biyik, David Held, and Zackory Erick- son. RI-vIm-f: Reinforcement learning from vision language foundation model feedback. arXiv preprint arXiv:2402.0366/ 2024b.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229 256, 1992.
Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell. Read and reap the rewards: Learning to play atari with the help of instruction manuals. Advances in Neural Information Processing Systems, 36:1009-1023, 2023.
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Tex(2reward: Automated dense reward function generation for reinforcement learning. In International Conference on Learning Representations (ICLR), 2024 (0770552024-1110520004 Vienna, Austria), 2024.
Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking Ilm agents on consequential real world tasks. arXiv preprint 1rX1v:2412.14161, 2024.
Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. Keep calm and explore: Language models for action generation in text-based games. arXiv preprint arXiv:2010.02903, 2020.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang Huei Lee, Montse Gonzalez Are- nas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, al. Language to rewards for robotic skill synthesis. arXiv preprint 1xXi:2300008641 2023.
Eric Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.096600 2021.
Zihao Zhou, Bin Hu, Chenyang Zhao, Pu Zhang, and Bin Liu. Large language model as policy teacher for training reinforcement learning agents. arXiv preprint arXiv:2311.13372 2023.
Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, al. A survey on efficient inference for large language models. arXiv preprint 4rXiv:2404.14294 2024.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-actto models transfer web knowledge to robotic control, In Conference on Robot Learning, pp. 2165-2183, PMLR, 2023.
17

---

Published as a conference paper at ICLR 2026
APPENDIX
The supplemental material is organized as follows:
SECTION A reviews background reinforcement learning definitions and policy gradient algorithms to make the paper self-contained.
SECTION discusses related work relevant to our approach more depth.
SECTION presents the theoretical results and supporting proofs for our method.
SECTION D describes the LLM prompting procedures in MIRA and presents the corre- sponding reasoning traces.
SECTION E provides more detailed explanation of the construction of memory graph, expanding on the description in the main paper.
SEcTION F provides more detailed explanation of the utility calculation and includes the complete pseudocode, expanding on the description in the main paper.
SECTION G presents extended experiments, including analyses of runtime and detailed numerical results that were not covered the main text.
SectIOn H outlines limitations of the current design and identifies open challenges.
Section provides details support reproducibility of results.
A BACKGROUND
A.1 STANDARD REINFORCEMENT LEARNING
Reinforcement learning (RL) is typically modeled as a Markov decision process (MDP), defined by a tuple (S, A, r),, where S is the state space, AA the action space, / the transition function, r the reward function, and 7 E [0. 1) the discount factor. The agent's behavior determined by policy which defines probability distribution over actions given the current state: x <a]s)) Learning proceeds through interaction with the environment, producing trajectories, sequences of states, actions, and rewards of the form r (so, do, To, 81, (1,7,,..... and using them to improve policy.
The objective is to learn policy that maximizes the expected return, defined as the discounted sum of rewards along trajectory:
2'r(si, at)
E JT
The environment's reward function implicitly defines the final goal (g,) by assigning reward to behaviors that accomplish the task (Sutton et al., 1998; Silver, 2015). To estimate this objective, RL algorithms often make use of value functions, which quantify the long-term utility of states or state-action pairs. The state-value function (s) denotes the expected return when starting from state and following policy m:
(ss) = E,
rr(s 01) so =
(5)
The action-value function Q(s, further conditions the first action taken and is defined as:
(s, a) = E
rr(s, oz) ) so = 00
(6)
18

---

Published as a conference paper at ICLR 2026
A.1.1 PARTIAL OBSERVABILITY AND CREDIT ASSIGNMENT CHALLENGES
In many real-world scenarios, the environment only partially observable. In such cases, the MDP generalizes to a partially observable MDP (POMDP), defined by the tuple (S,A,P, 7, 7,0)) where O the observation space and the observation function. The agent does not directly ob- serve the true state , E S; instead, it receives observations from an observation space O, sampled via (o ss), and must rely on its history of observations and actions to make decisions (Kaelbling et al., 1998).
These difficulties are further amplified in environments where the agents face sparse and delayed rewards. Sparse rewards refer to the limited presence of nonzero rewards since this feedback is only provided upon reaching specific goals (i.e., rst: de) is typically zero until the agent reaches the final goal state (g,)) defined by the task). On the other hand, delayed rewards refer to settings where the consequences of an action are not reflected in the reward until several steps later. In both cases, the agent must reason over long horizons to determine which actions contributed to the eventual outcome, a challenge known as the credit assignment problem (Schulman et al., 2015a).
Credit assignment is closely tied to the broader challenge of exploration. Inefficient exploration occurs when the agent fails to sufficiently cover the state space, limiting its ability to discover high-return trajectories and improve its policy. This problem is exacerbated in high-dimensional environments, where the number of possible state-action sequences grows exponentially and ran- dom exploration becomes increasingly unlikely to encounter informative transitions with sparse or delayed rewards. In such cases, the combination of large search spaces and limited reward signals often leads slow convergence, poor sample efficiency, and high variance in learning outcomes.
A.1.2 SUBGOALS AND ABSTRACTIONS
In long-horizon tasks, reinforcement learning agents often benefit from structuring behavior around subgoals, intermediate objectives that facilitate progress toward the overall task. The concept of subgoals in reinforcement learning originated in hierarchical reinforcement learning (HRL), where it was formalized through the use of temporally extended actions. In particular, the options framework introduced by Sutton et al. (1999) defines options high-level actions composed of an initiation set, a policy, and a termination condition, often interpreted as achieving a subgoal (Stolle & Precup, 2002). These subgoals correspond intermediate states or conditions that decompose long-horizon tasks into smaller. temporally coherent segments that make the final goal more attainable when reached. More broadly, subgoals provide structure for reasoning over extended time horizons and facilitate learning in sparse-reward settings.
While early approaches focused on explicit or learned state-based subgoals, recent work has ex- plored abstract subgoals that capture semantic or latent-level progress. These abstractions may not correspond to a specific state but instead reflect high-level intentions and meaningful progress (e.g., opening a door, entering a room, or collecting ing at a higher level of granularity and are especially useful in environments with sparse rewards or delayed feedback. Subgoal discovery and abstraction have also been explored in curriculum learning, imitation learning, and human-in in-the-loop frameworks to improve exploration and sample efficiency (MacGlashan et al., 2017; Shiarlis et al., 2018; Narvekar al., 2020).
object). Such abstractions enable reason-
A.2 POLICY GRADIENT METHODS
Policy gradient methods directly optimize a parameterized policy Th (a(aas) by ascending the gradient of expected return. The objective is to find parameters 0 that maximize:
(0) = Er Σ
7r'((s, at)
where denotes trajectory generated by following the current policy. The gradient of this objective can be estimated via the likelihood ratio trick, yielding the REINFORCE estimator (Williams, 1992):
(m)( E, (Vo log (a, R),
where R2 is the return from time onward. While theoretically sound and unbiased, this estimator suffers from high variance, making it challenging to apply in practice without further refinement.
19

---

Published as a conference paper at ICLR 2026
A.2.1 ADVANTAGE- BASED POLICY OPTIMIZATION
To reduce variance and improve sample efficiency, modern policy gradient algorithms often use ad- vantage functions, which quantify the relative quality of an action compared to the policy's s baseline behavior. The advantage function defined as:
A(s,
V(s),
where Q(s, a) is the expected return from taking action in state s, and (s)) is the expected return from 8 under policy æ. Using this formulation, the policy gradient becomes:
(T)) = Ex Vo To ss)
(10)
which improves stability while preserving unbiasedness.
This idea underpins family of actor-critic algorithms, where the actor updates the policy using the advantage-weighh gradient, and the critic estimates value functions used to compute A(s, a)- Representative algorithms this class include A2C and A3C (Mnih et al., 2016), which leverage parallel actors accelerate training and stabilize updates, and PPO (Schulman et al., 2017b), which constrains policy updates by clipping the policy ratio in the surrogate objective:
PPO (T) = E [min(re-A,, clip(r,, 1 - E, 1 + +)))
(11)
where E >> 0 is small trust region parameter that limits how much the policy is allowed to change at each update.
These methods are widely used in modern deep reinforcement learning due to their scalability and consistent empirical performance across range of tasks. Since MIRA operates by shaping the advantage function, it is compatible with any policy optimization method that relies on advantage- weighted updates.
B RELATED WORKS
B.1 LANGUAGE MODEL GUIDANCE IN RI.
growing line of work explores how large language models (LLMs) can be integrated into rein- forcement learning by framing them as auxiliary components within the agent- envernenvironment loop. A recent taxonomy by Cao et al. (2024) outlines the roles of LLMs in RL along four key dimensions: information processors, reward designers, decision-makers, and generators.
As information processors, LLMs extract and organize task-relevant knowledge from natural lan- guage, environment descriptions, prior experience. This includes synthesizing high-level goals, parsing instructions, and transforming language input into actionable constraints or representa- tions (Wang et al., 2024a; Shinn et al., 2023). A common approach is to frozen pre-trained models to encode task-relevant features without fine-tuning, though they may perform poorly on out-of-distribution data due limited adaptability (Radford al., 2021; Paischer al., 2022; 2023). Alternatively, fine-tuned models can better align with task-specific distributions, leading to more ro- bust RL performance and improved generalization in unseen environments (Kim et al., 2023; Poudel et al., 2024). In addition, LLMs can convert human instructions or task prompts into formal repre- sentations or structured goals, and interpret descriptions of the environment, such as objects, layouts, or dynamics, into usable priors for downstream RI. modules. This reduces the burden of language comprehension for RI. agents and improves sample efficiency (Basavatia et al., 2024; Sumers al., 2021; Song al., 2023a; Liang et al., 2022). These models decouple information processing from control, with the LLM handling language grounding and feature extraction while the policy module focuses on Decision-making Such capabilities reduce learning complexity and acceler- ate policy acquisition by shaping the agent's representation space early in training.
As reward designers, LLMs provide auxiliary supervision by scoring agent behavior generat- ing rewards. This can take the form of natural language critiques, programmatic reward code, or goal-conditioned evaluations. In the implicit reward setting, L.LMs serve proxy reward models
20

---

Published as a conference paper at ICLR 2026
by either being directly prompted to evaluate agent behavior (Chu al., 2023; Wu al., 2023) or by computing alignment between visual observations and language goals using pretrained vision- language models (Wang et al., 2024b; Adeniji et al., 2023; Seo et al., 2023; Grauman et al., 2022). These methods enable reward shaping via natural language instructions or preference feedback and have been shown to improve learning efficiency and generalization. In the explicit reward setting, LLMs are used to generate executable code that defines reward functions programmatically. This includes frameworks that iteratively refine reward code using self-reflection and feedback from train- ing outcomes (Yu et al., 2023; Madaan et al., 2023; Song et al., 2023b). Compared to manually en- gineered rewards, these LLM-gennraan functions offer transparency and adaptability, and in some cases match or exceed human performance, especially in complex manipulation tasks.
As drcision-makers LLMs output action plans, policy sketches, or even direct action sequences based on current observations. These methods embed L.MSs tightly into the decision loop, either guiding exploration or dictating behavior in few-shot or zero-shot settings. One approach leverages pre-trained LLMs for direct action generation, often adapting transformer-based models like Deci- sion Transformers to treat offline RL as a sequence modeling problem. These LLM-backed policies show improved generalization, particularly in sparse-reward and Long-horizon tasks, by transferring latent structure learned from large-scale language data. Some methods further fine-tune LLMs using task-specific trajectories append small task-specific modules facilitate adaptation, achieving no- table gains in sample efficiency and task transfer (Zitkovich et al., 2023; Shi et al., 2023; Mezghani al., 2023).
Other works integrate LLMs more loosely as action guides, generating action candidates or ex- pert priors to support exploration and training. For example, LLMs can prune the action space by proposing high-probability candidates or decompose complex goals into sequential subtasks, improving exploration in environments with large or unstructured action spaces (Yao et al., 2020; Hausknecht et al., 2020; Dalal et al., 2024; Wan et al., 2025). They have also been used to regularize policy updates, align agent behavior with human intent, or inject expert-level motion plans. Across both low-level and strategic roles, LLM-based pecision-making enables agents to learn from rich, structured priors and execute more informed behaviors in complex settings.
As generators, LLMs contribute to reinforcement learning by either simulating environmental dy- namics or providing policy-level explanations to enhance transparency. In the simulation role, LLMs function world model simulators that generate trajectories or learn latent dynamics represen- tations from multimodal data, thereby improving sample efficiency in model-based RL. Recent work has leveraged Transformer-based architectures model complex visual or sequential envi- ronments, demonstrating gains in generalization and long-horizon reasoning. These models either ruto-regressively generate rollouts from pre-trained dynamics or use representation learning to pre- dict future states and rewards, often incorporating language as an additional modality for grounding and abstraction (Micheli et al., 2022; Chen et al., 2022; Robine et al., 2023). Separately, LLMs have been used as policy interpreters to generate human-readable explanations of agent behavior from state-action histories or decision trees. This facilitates interpretability, improves human trust, and can inform reward design or debugging, though current work has focused mainly policy-level summaries (Lin et al., 2023; Lu et al., 2023).
While MIRA incorporates elements of information processing and L.MSs as generators, its overall orientation remains distinct and more RL-centric from prior LLM- approaches. Rather than positioning the LLM mecision-maker a supporting role that gradually fades over time. LLM outputs are used intermittently to enrich a structured memory graph that informs, but does not dictate, learning. The primary learning signal remains grounded in environment interaction, with utility shaping softly modulating advantage esti- mates rather than overriding the reward function. This design prioritizes policy optimization through reinforcement learning rather than imitation or prompting
continuous feedback provider, MIRA relegates it to
B.2 MEMORY AND BUFFERS IN RL.
Augmenting RL agents with structured memory has been proposed as a means of supporting general- ization, planning, and long-hou horizon credit assignment. Early works such as Neural Episodic Control (NEC) and other episodic value-based methods enabled agents to recall high-value past experiences for more sample-efficient decision- decision-making via memory buffers (Pritzel et al., 2017; Blundell et al., 2016; Lin, 1992). Subsequent approaches extended this idea by integrating differentiable memory
21

---

Published as a conference paper at ICLR 2026
into policy networks (Qiu al., 2024). Other methods introduce structured representations, such as subgoal graphs or navigation maps, to facilitate hierarchical planning, exploration, or navigation in partially observable environments (Beeching al., 2020; Rama et al., 2023). Across these directions, the common pattern is to directly query stored structures, either through replay, imitation, or graph traversal, guide behavior.
MIRA aligns with this direction by maintaining a structured memory graph populated with high- return trajectory segments but departs from this pattern in several key ways. First, its memory graph is co-constructed from high-return agent trajectories and LLM-inferred subgoals, enabling abstraction and structure difficult to obtain early through interaction alone. Second, rather than querying memory for action selection or value estimation, MIRA distills the stored information into a utility signal that modulates advantage estimates during training. This indirect shaping avoids disrupting the optimization loop or overfitting to specific stored transitions. Finally, MIRA maintains a compact memory via pruning and infrequent updates, which avoids the inefficiencies of excessive memory the brittleness of sparse guidance (Liu & Zou, 2018). This makes MIRA more scalable and better suited for tasks where long-term structure must complement autonomous learning.
B.3 ADVANTAGE MODIFICATIONS RL
Modifying the advantage function has been studied way stabilize learning and improve sam- ple efficiency in policy optimization. A common approach adjusts the estimation process to better balance bias and variance. Generalized Advantage Estimation (GAE) (Schulman et al., 2015b) introduces a tunable parameter that interpolates between high-bias low-variance and low-bias high- variance estimators, and is widely adopted in actor-critic algorithms. Other methods reformulate policy updates in terms of advantages. avoids policy gradients and instead performs weighted regression over actions. P3O (Fakoor al., importance sam-
-Wenggge Regression (AWR) (Peng et al., 2019)
2020) combines on on-policy and off- off-policy learning by applying dedvantage-weightee pling to stabilize updates. the offline RL setting, advantage estimates are often used to filter ex- perience and address distributional shift. Advantage-based data selection (Kostrikov et al., 2021) discards transitions with low advantage, helping to focus learning on high-quality samples. Addi- tional work incorporates auxiliary signals into the advantage estimate. Preference-based RL (Lee et al., 2021) derives implicit advantage signals from human comparisons, while other approaches integrate value correction from ensemble critics or confidence measures to adjust learning.
MIRA builds on these ideas but takes a different path. Instead of replacing the estimator or introduc- ing new objectives, it shapes the advantage using utility term derived from structured memory graph. This utility reflects agent experience and LLLM-derived subgoals, allowing guidance with- out overriding reward feedback. The resulting signal is integrated into PPO's update rule without disrupting its optimization dynamics, enabling structured shaping while maintaining scalability and convergence guarantees.
C THEORETICAL RESULTS
Since the utility term does not alter the policy or critic structure, and enters additively, MIRA preserves standard stability and asymptotic properties of policy gradient methods such PPO under standard assumptions:
C.I ASSUMPTIONS
Assumption (Boundedness).
a. For all updates k and all (s,a) [s, S Amax. 0 Un(s, a) S Umax
(12)
b. Define the scale- adjusted shaping term as: Un(s, a) = As Un(s,)), where As = (lAul)
(13)
and set
U Unax Umax sup Av
(14)
22

---

Published as a conference paper at ICLR 2026
Assumption (Scale control).
a. For all k, the scaling parameters satisfy:
< 1) S 1, &x S
denn for some ö, E [0,
(15)
b. Asymptotically, the schedule satisfies:
lim Mh 1, k
lim & =
(16)
k-000
Assumption (Trust region).
11-9) 7)
K KL Tb+1)
2
(implied by PPO clip ratio Tx [1 Sk: Ex))
C.2 STABILITY AND SAFETY RESULTS
Lemma (Bounded Shaped Policy Updates). Under Assumptions I (Boundedness), (Scale Con- trol), and 3 (Trust Region), the magnitude of each shaped PPO policy update is uniformly bounded.
Proof. By definition, the shaped advantage satisfies
|A,| S 7//A,1 + E.UUAl.
Using Assumption 1 and Assumption 2(a), we obtain
| S Amax E.nmax S MmAmax mmxx S + Amax-
Under the PPO trust-region constraint in Assumption 3, the likelihood ratio is clipped and the score function Ve log Taa (a))s;) has bounded second moment. Consequently, the policy gradient update
Ca
sshoped
E
V log To (arls)) Ae]
uniformly bounded in norm.
Theorem 2 Non-Divergence under Trust Region). Under Assumptions (Boundedness)) (Scale Control), and 3 (Trust Region). the PPO updates computed using the shaped advantage Aş A E.U, remain within the prescribed trust region. In particular, the KL divergence between successive policies is uniformly bounded, and the optimization does not diverge.
Proof. At iteration k. PPO computes the policy update by maximizing clipped surrogate objective of the form
(0) IE min (r:(0) A., clip(r: sk. 1 Ex) =x) A:)
+x))
where r0) = ne (a)lss))) Ta, (ar (s)) The clipping operation enforces
(0) [1 Eks 1 1+ Ex],
which implies the trust-region bound in Assumption 3 (see Schulman al. (2017b)).
This constraint depends only on the policy parameterization and the clipping threshold Ex. and independent of the specific form of the advantage estimator. Therefore, optimizing the shaped surrogate induces the same likelihood-ratio constraint as standard PPO.
By Lemma (Bounded Shaped Policy Updates), the resulting policy update has uniformly bounded magnitude. Combining bounded update magnitude with the likelihond-ratio constraint yields
KL(T& || [T+1)) < (1=y)e
2
Hence, the sequence of shaped PPO updates remains within the prescribed trust region, and the optimization does not diverge.
23

---

Published as a conference paper at ICLR 2026
C.3 ASYMPTOTIC BEHAVIOR
Theorem 3 (Asymptotic Equivalence to PPO). Let { be the sequence of policies produced by PPO using the shaped advantage
(17)
where U. is computed on-policy from the same rollouts At- Under Assumptions I (Boundedness), 2 (Scale Control), and 3 (Trust Region), the shaped update is asymptotically equivalent to the stan- dard PPO update. In particular, any stationary point of PPO is also stationary point of the shaped objective.
Proof. The PPO policy gradient update with shaped advantage is given by
Lk
Eshaped Be
Vo log No As]
(18)
Substituting the definition of A, yields
Cohaped 176 CPPO Ek Ck
(19)
where
c. = C = E| log (a Ui]
Ad]
(20)
(21)
By Assumption both A, and U, are uniformly bounded. Under the trust-region constraint in Assumption 3, the score function V log (a(a)())) has bounded second moment, ensuring that both gradient components are finite.
By Assumption 2(b).
lim mh 1, k0000
lim En 100
(22)
which implics
lim Lg L
(23)
k-+00
Therefore, the shaped update converges to the standard PPO update. Any policy n² satisfying CPPO (T*) O also stationary point of the shaped objective asymptotically.
C.4 EARLY-TRAINING ADVANTAGE
Theorem[reinstate] (Non-Vanishine Vanishing Updates in Sparse-Reward Regimes).
Proof. Recall that the shaped advantage is defined
A Mr + EnUe, and the corresponding shaped policy gradient is
Lx Cahaped
E
V log To (arls)) A.
By linearity of expectation, the shaped gradient decomposes as
C eshaped = -mcCPP + C.
(24)
where
CPP E. V log (o (s)) Ar).
c = E log (acls)) U.)
By assumption, the expected magnitude of the PPO advantage is small, E[|4||] < A Under As- sumption |Ar] < Amax, and under the trust-region constraint (Assumption 3), the score function Va log (Ta (se(se) has bounded second moment. Consequently, there exists a constant C > 0 such that
| IPP S CE|]A.]] Ce
24

---

FROZENLAKE a tabular RL environment where the agent starts in the top-left and must reach the bottom- right goal while avoiding holes. For FROZENLAKE, we provide the LLM with the complete map of the envi- Published as a conference paper at ICLR 2026
Frozeni dise involwis crossing a frozen lake from the start die to the goal lile (6) without fallng into holes (H), by valking only on frozen Blls (F). The agent begins state and the goal located at the last state of the gld Any move that would take the agent off the board results staying in place. which sate and sometimes proterable to risky moves, Given the gie layout and state representasions, provide a brajestory from the start to the goal that avoids all holes and resches the goal along the shortest safe poth. Assume determiniclic transiionss Return the best and second-best best safe trajecioreee from atari (8) goal (G) the that state (D-lef), I-down, 2wright, Smup).
Python polccy dictionaries. for goal
key should be a state, and each value
one safe shartest path avoiding a11 holes is: [0 1 6
Another safe path that's still short (but one step longer): [0
14
15]
best 0: 2, 2, h. 1. 10: 3, 2, 15: kone)
second_best [0:
1. 6: 1. 10: 0,
13: 2. 14: 15: None)
0:264 win
Figure 9: Prompt and response from GPT-4o-mini for the FROZENLAKE environment. The LLM receives serialized map textual description and is asked to provide the best and second-best safe trajectories from start to goal, avoiding all holes under deterministic dynamics.
Taking norms both sides of Eq. equation 24 and applying the triangle inequality yields
Eshaped
> Ex|l
me
||
Substituting the above bound on | gives
[thaped
2 Ex|
cx||
||
7)6 CEA-
Since 0 < mh S by Assumption 2, the second term is (EEA), and obtain
shaped
Ex |
O(s)).
which completes the proof.
Remark (Bias-Variance Trade-off). The shaping mechanism can also be interpreted through the lens of the bias trade-off in policy gradient estimation. Standard PPO relies advantage estimates (e.g., GAE or Monte Carlo returns) that depend on long-horizon rollouts and typically exhibit high variance, particularly in early training. In contrast, the utility signal U, depends only on the current state- acteon pair and its similarity to stored high-return trajectories, yielding a lower- variance but biased learning signal. Initializing &n 0 stabilizes early optimization by injecting this bias, while the decay schedule & -> 0 gradually restores reliance on reward-based advantage estimates the policy and critic become more reliable.
D LLM PROMPTING AND REASONING
D.1 GYMNASIUM TOY TEXT
ronment, either as an image (Figure 8) or as serialized array repre- sentation such as ('F,, ' Though the environment is typically stochastic due to slipperiness, the LLM is instructed to assume deterministic transitions.
'H', '
'G') ].
Although much of the prompt is directly from the official environment description, for clarity and reproducibility, we include the full version. Figure Frozen Lake The prompt and the LLM's response shown in Figure 9.
(Gymnasium)
D.2 STANDARD AND CUSTOM MINIGRID AND BABYAI ENVIRONMENTS
Each environment chosen for specific purpose: REDBALL involves short-horizon navigation and fast spatial goal acquisition. LAVACROSSING introduces irreversible transitions that require long-horizon planning avoid dead ends. DOY requires the agent acquire key, unlock
25

---

Published as a conference paper at ICLR 2026
We in MiniGrid environment where the agent must pick yellow key, use it to toggle yellow door, and then reach the green goal action space
standard MiniGrid spedfiiation.
This environment challenging for classical RL algorithms due to its sparse rewards. There only one key-door pair (yellow), Do you understand this environment? Answer yes no.
objects like purple and distractors.
Figure 10: Prompt to Offline LLM for custom MiniGrid variant DISTRACTED DOORKEY. The prompt describes the task setting, object roles, and challenges, and asks the LLM to confirm under- standing before suggestions.
door, and reach the goal, forming a delayed dependency chain that challenges temporal credit assign- ment. REDBLUEDDOO tests the agent's s ability to commit to a correct action sequence, as opening the blue door prematurely ends the episode. At last, DISTRACTED DOORKEY introduces BabyAI- AI- style distractors (e.g., irrelevant balls and boxes) alongside the original multi-step dependencies of DOORKEY, allowing to test whether the LLM can generalize across known task elements and maintain coherent subgoal proposals under added visual distraction. For standard MiniGrid and BabyAI environments, used the environment descriptions provided the MiniGrid website. For our custom environment (DISTRACTED DOOR KYY), we mimicked the phrasing and structure of the official MiniGrid descriptions (Figure 10). Unlike in FROZEN LAKE, obtaining useful tra- jectories here was not as straightforward. MiniGrid-style environments often required multi-round prompting to obtain meaningful and desired outputs. Moreover, instead of providing an image of the environment, we found it more effective to use a textual description. This helped reduce confu- sion and encouraged the LLM to understand that object locations (e.g., the key, door, and agent in DOORKEY) can vary across episodes.
D.3 LLM REASONING PATTERNS ACROSS MODELS
We observed that different LLMS produced very different memory graphs. To better understand how different models reason about these environments, we recorded not only their output trajecto- ries but also their internal reasoning processes. For model that include system-level thinking (c.g., GPT-o4-mini), this was extracted directly from the response. For models that do not expose inter- mediate reasoning (e.g., Claude 3), we followed up with an auxiliary prompt such as: "Give me your reasoning as why you chose this sequence of actions."
These responses were not used in the MIRA framework, but we found them surprisingly reveal- ing. Despite receiving identical prompts, the models relied starkly different reasoning strategies. This divergence gave us unexpected insight into how various L.MSs process spatial structure, inter- pret decision sequences, and reason about reinforcement learning dynamics and learning objectives. Differences that, in turn, shape the quality of their output trajectories. In Figure 11, we present reasoning snippets from the LLMs' outputs. We omit the initial sections where models repeat the prompt or restate the environment description, and instead highlight the specific reasoning steps that led each model to select a particular trajectory. The influence these differing reasoning strategies on RL performance is reflected in the return curves shown in Figure 7.
D.4 CASE STUDY: DISTRACTED DOORKEY
In the ablation study presented in Subsections 4.2, GPT-o4-mini and Gemini return different outputs when presented with the same situation. Here, we provide the exact prompt and reasoning traces. As shown in Figure 12, both responses appear plausible at a surface level, but only one is consistent with the task dynamics: given that sufficient exploration has already occurred, the key is likely collected, making suppression of the corresponding action the correct response. In this case, the divergence leads to drop in performance under the misaligned output.
26

---

calls, Published as a conference paper at ICLR 2026
forward, until key Found. Teft, right, fornard. found: pickup continue for sall. Toggle door
until hit
back
accessible
claad Sprees
After attempting to pick up sonething (even if it wasn't the key), the agent should actively try to approach the door to see if it opens. Choose actions (forwrd, trrn left, turn right) randorly with equal probability btt.
The most effective way to de this is a "wall-following" a1 algoriths: pick a wall (e-g., the one on your right) and keep st to your right as you move (Tonwad/tumm right pattern). This guarantees you will evantually scan every reachable tile in the roon wi thouh getting lost in circles. then go to door.
Trajectories only seep segrents: fron
anything seeing key (xf pickup held. callinng before point))...
step
devert de
Envi ronsent is procedurally ly generated, Therefore, fixed path will fat door's "togple position" (same, row, one step may in the correct facing). don't condition advice on the agent true state because know don't 't have access to when key found: prckup wall followng to find door. Toggle door navigate goal
Remded
To avoid ovarfitting, randorly insert seall "detours" so the agent sees both variabilitty and reward.: e.g. after picking up the key, add 1-2 extra fornard/turn actions toward a randon distractor cell,, then re-alige to the optiral path. keep only clean. successful runs of reasonable Tength, The randon "deetor" insertions should also respect the current map pick distractor
that actually exist), so they" adapt each layout.
Reasoning traces produced by different LLMs in response to our custom enviromment prompt as part of "Offline LLM" prompting. After confirming they understood the environment, each model was asked: "If you were give RL agent useful trajectories to help solve this, what would you do?" For models that do not output internal reasoning (e.g., Claude), we issued follow- up prompt requesting their thought process. We omit repeated environment restatements and show only the key parts where the model explains how it decided on the action sequence.
Figure 11:
agent toggle probability agent forward progress (noving, navigating, reaching goal) maxfrize coverage rather wasting actions
toggle-it
I would lower the prybability of the prickup action.
The reasening is that since the key is not the agent's observation, the pickup action is not useful for raking progress towards the goal this time. Lowering its probabr lity wouldd prevent the agent from wasting time trying to pickup the purple ball, which has benefit, and would instead encourage it to continue exploring with the other available actions ( turn left, turn right, forward) to find the key.
015997 d6
Reasoning traces produced by Gemini and ChatGPT under "Online LLM" prompting. The prompt emphasizes that sufficient exploration has already been performed and, from the partial observation, no key is visible. A (flawed but plausible) line of reasoning is that the agent must still be the phase of searching for the key, so reducing the probability of toggle appears reasonable to prioritize movement actions for exploration.
Figure 12:
E MEMORY GRAPH CONSTRUCTION DETAILS
In this section, we further explain the procedure for initializing, updating, and pruning MIRA's memory graph. discussed in Section 2, the initial memory graph constructed from offline LLM-generated suggestions. Once built for specific environment, this graph can be reused across training episodes or even across agents within the same task. Since MIRA is designed to generalize across diverse settings, figure illustrates how the framework accommodates environments with single terminal objective as well as tasks with multiple independent objectives
Given that each task differs slightly, we largely focus detailed explanation on DOORKEY from the MiniGrid suite for the rest of the subsections, as it contains multiple subgoals and is sufficiently complex show the dynamics of the graph clearly.
27

---

Published as a conference paper at ICLR 2026
E. INITIALIZATION
As shown in figure 11, GPT-o4-mini tends to generate trajectory segments that begin after the key is picked up, with the subgoal "toggle the door". In contrast, models like Claude tend to produce longer, full trajectories from the beginning. Interestingly, segmented trajectories are often more useful in this environment. Since the environment is partially observable and reinforcement learning relies heavily on exploration, allowing the agent figure out how reach the key on its own helps it understand the overall layout of the environment better. Once the key is acquired, there is a higher chance that the door has already entered the agent's observation window, making memory-guided navigation toward the door more effective.
In addition to segments, the LLM also infers subgoals (Ke). While the obvious ones are "Pick up key," "Open door," and "Reach goal," o4-mini returns more detailed versions like:
K1 Go to key K2? Pick up key + K3 Go to door - K4 Toggle door - g.; Go to goal.
This fine- fine-grained subgoal sequence reflects the environment's control logic: the "open door" action is valid only if the agent is positioned one step away, properly aligned, and facing the door.
Moreover, for each memory segment, an estimated subgoal reward fm is stored in the node that reflects geometric progress toward completing its associated subgoal. In discrete environments (e.g., MiniGrid), progress is computed using the normalized shortest-path distance between the states in the segment and the subgoal's 's target location.
E.2 AGENT-INDUCED UPDATES
During training, new nodes are added to the memory graph or existing ones are updated whenever the agent produces trajectory segments that improve upon what the graph already stores, either by providing new segment for a (sub)goal or by achieving higher estimated return than the current entry for that (sub)goal. For example, if the agent independently discovers a shorter path to the key and then follows a memory- guiddd trajectory to reach the door or goal, the resulting sequence added as a new node in the graph. Likewise, the agent successfully executes a trajectory that was initially stored with low confidence from the offline LLM, we treat this behavior as implicit validation and increase the confidence of the corresponding memory node.
The memory graph remains lightweight throughout training. Each node stores a trajectory segment and metadata, and the total graph size stays compact. Compared to experience replay buffers in standard off-policy RL methods, which retain large volumes of data, the memory graph introduces negligible computational and memory overhead. To maintain compactness, unused nodes are peri- odically pruned based access frequency. Each memory node tracks access counter, which is reset every time the node is used. Nodes that are not accessed for 100 episodes are pruned, except for those corresponding final goal trajectories (g,). which retained since the agent might have reached them early on, but they are essential for guiding successful completion later in training.
Algorithm summarizes the add, update, and prune operations that govern how the memory graph is maintained during training.
E.3 ONLINE GRAFTING AND TRIGGERS
Since the agent has a limited number of steps per episode, it may fail to reach any subgoal (e.g. "Open Door") with a matching trajectory in the memory graph early on, preventing utility shaping from activating. To address this, MIRA includes a fallback mechanism: if the computed utility U is entirely zero for N consecutive episodes, the agent triggers an online LLM query. These online queries return short plans (e.g., "turn left, move forward, toggle")) based on the agent's partial observations to help the agent reorient. Once screened for quality, the new suggestion is grafted into MIRA. Another way online LLM queries contribute is by influencing the agent's policy preferences directly through soft logit injection. Importantly, the online LLM is constrained by the same partial observability the agent. It does not receive access the full environment state and therefore cannot, for example, determine the presence of a key elsewhere in the grid. Furthermore, since inventory status is not part of the agent's observation space, the LLM is unaware of whether the
28

---

Published as a conference paper at ICLR 2026
agent has picked up the key. Instead, the LLM receives a batch of recent partial observations and must infer from them whether any meaningful guidance can be offered.
F UTILITY COMPUTATION
In this section, provide detailed explanation of the utility computation introduced in the main text, clarifying how each component contributes to the shaping term. The utility measures how closely the agent's trajectory aligns with high-return segments stored in the memory graph. When reference trajectory matched, utility values are assigned based on reverse-aligned similarity with reference trajectories; unmatched steps receive zero utility. Below, we deseribe the role of each factor in the computation (Equation 2) and provide pseudocode for the full procedure.
F.1 SIMILARITY SCORE
The similarity function assigns score based on the information extracted from the agent's 's and the reference transition's 's observations. Depending on the environment, these observations may include position, orientation, action. For example, in FROZENLAKE, observations discrete and include only position, so direction is omitted the similarity check. High similarity indicates that the agent reproducing a locally meaningful portion of successful stored trajectory, whereas low zero similarity reflects small or no meaningful match.
Algorithm 2 Evolving Memory Graph During Training
Require: Memory graph 9, new trajectory and metadata (T', (',', prune window W M₁. {m G Sm }
if source = ONLINELMM then
if -CRREENNIN then
return G Discard online LLM suggestion end if' end if if M
No existing segment for this subgoal then
Create node mn' + (F', Initialize access,' Insert m into G
else
arg maxmEM; fm
if R'''S fm then (Tm; + (T',
Cm x- end if
if source AGENT then
C. + min(1, (i,(1,,, Cm + Ac)
Agent validation increases confidence
end if end if
for each node E G do
if access,, = for W episodes then
Remove m from > Prune nodes unused within inactivity window
end if end for return G
F.2 GOAL ALIGNMENT
The goal-alignment factor scales utility based on how closely the subgoal associated with matched memory segment relates to the subgoal that the current transition corresponds the RL agent never observes subgoal labels, the environment state uniquely identifies which subgoal
to. Although
29

---

Published as a conference paper at ICLR 2026
Algorithm 3 Similarity Score
Require: Agent 2a and Reference I' annotated transition (metadata)
if (pos., dir.) E (ou. 0m) match & aa = Qm then return high sim (1)
else if pos. E (0a. Oml) match & aa = am then align direction (0.7)
return mod sim
else if (da E 0a) = mod 4 = dm then return = low_sim
action aligned direction (0.4)
else
return f no_sim
(0)
end if
phase the transition belongs to, for example, in DOORKEY environment, whether the agent still approaching the key or has already picked up and is proceeding toward the door. Fach memory node carries subgoal label generated by the LLM. These subgoal descriptions consistently identify (i) the object or region involved and (ii) the high-level action applied to it. We extract these two components through simple rule-based parsing over the LLM-generated subgoals, yielding an entity token (e.g., key, door, ball) and an action- action-phass phase tag (e.g., navigate). The alignment score p is then calculated as the Jaccard similarity between the token pair of the memory node's subgoal and the token pair associated with the transition under evaluation. As result, locally similar transitions only contribute utility when they align semantically with the relevant subgoal, preventing behaviorally similar but semantically unrelated memory segments from influencing the utility signal.
Algorithm 4 Goal Alignment
Require: Agent Ça and Reference Sm subgoal
to + TOKENS(Sa) >> Agent entity-phase token
fm < TOKENS(Gm)
Memory entity-phase token
In im Ole
Tu Im Ula
return p TT/TTol
With the similarity score / and alignment factor p established, we combine them with the memory- stored quantities fm (estimated subgoal reward) and em (LLM confidence) to construct the utility used in advantage shaping.
Algorithm Compute Utility Score
Require: Agent Tagent and Reference Tm trajectory
Denote transition with metadata (e.g. subgoals) Initialize U < [0
= (o, r, meta)
Align the tail of Tagent length of Tm
for each (Iay F'm) (r/jiil Tm) do
I + I((0a, aa); (Om, am)) P - P(Sa, 5m) & Compute goal aligment factor
Compute similarity
Gm f' pp.
Assign u to corresponding index in U end for return
30

---

Published as a conference paper at ICLR 2026
G EXTENDED EXPERIMENTAL STUDIES
G.1 SENSITIVITY AND ROBUSTNESS STUDIES
1.0
Frozen Lake
G.1.1 PROMPT ROBUSTNESS
Mean Return
1.55
To evaluate robustness to reasonable variations prompt wording, repeated the FROZENLAKE ex- periment using an alternative prompt with simpler phrasing but identical task information. In this vari- ant, the prompt stated that "FrozenLake is a grid where S is the start, G is the goal, F are safe tiles, and H are holes. The agent moves from state O using actions O=left, ==down, 2-right, 3=up, and moves that go off the grid keep the agent in place and are safe. Using the grid and assuming deterministic tran- sitions, provide a shortest safe path from S to G that avoids all holes, and return the best and second-best safe paths as Python dictionaries mapping each visited state to its action, using None for the goal holes." Figure 13 compares MIRA under the original and alternative prompts. The learning curves and final returns are closely aligned, showing that MIRA's performance is stable under natural variations in how the environment description is presented.
MIBKA (beatine prorpp)
0.0
1000
Iteration 2000
3000
Figure 13: FrozenLake robustness to prompt wording. MIRA achieves similar performance under the original and alter- native prompts, showing stability natural variations in task description.
G.1.2 THRESHOLD SENSITIVITY
To assess the sensitivity of the screening rule, we conduct study that varies the acceptance threshold while keeping the overall method fixed. The thresh- old, therefore, controls when and how densely the graph is populated, not whether shaping exists at all. The main results of the paper uses k 3 LLM com- pletions, which was chosen for efficiency across all experiments. For this diagnostic test only, we set k 4 to obtain cleaner fractional thresholds cor- responding meaningful agreement levels on the DOORKEY environment. We evaluate three settings: a lenient majority rule (r 2 2/4), a stricter majority rule (T 2 3/4), and unanimous agreement (r 1).
Threshold Sensitivity
075
Mon Momitamm
0,50
0.2
100
beerice
Figure 14: Sensitivity to the screening threshold. Lenient thresholds graft more candidate nodes into memory early, pro- ducing broader shaping during exploration but a slower overall improvement rate. Stricter thresholds delay graph growth yet yield sharper mid-training gains once high- confidence nodes appear. All settings verge to a narrow performance band.
Figure 14 shows that the threshold primarily affects the early phase of learning. With a lenient thresh- old, more candidate suggestions are grafted as healthy graph nodes and added to the memory. This leads the agent to receive utility bonuses on more states while it is still exploring. As result, even if some early nodes may be slightly misaligned, the overall trend still pushes the agent toward regions that have higher returns. It is then up to the agent to cor- rect those nodes through experience or increase their confidence if they turn out to be helpful. which explains why the overall improvement can be slower in this setting. In the stricter settings, fewer suggestions pass the screening, so graph growth is delayed and the shaping signal remains sparse. Early learning progresses more slowly, but once these high-confidence nodes enter the graph, they produce larger and more coherent utility bonuses along trajectories that already correlate with high return, which creates the steeper rise visible in the mid-training region. The near-final performance still lies in a narrow band, since the shaping term is bounded and eventually dominated by the learned value function. This behavior is consistent with the design of our utility-based shaping, using the LLM's possibly imperfect but often still useful knowledge to accelerate the initial learning phase while ensuring stable progress more reliable evidence accumulates.
31

---

Published as a conference paper at ICLR 2026
G.2 EARLY ADVANTAGE DYNAMICS
Figure provides empirical support for the central intuition behind our shaping formulation. We plot return curves for each Ç group (color), across dif- ferent 7 values (line style). Early in training, return curves within each E group remain tightly clustered, indicating that At, the critic's estimate, provides lit- tle useful signal, regardless of how is weighted. Divergence points, marked the figure, denote the first iteration where the return spread across 7) values exceeds a certain threshold, signaling that A, has be- gun contributing meaningfully to the shaped advan- tage A, = A, + E.U.
Early Return
Mean Rocam
64
82
lientiion
Figure 15: Return curves for different 1j val- ues under fixed settings. Markers indicate the first iteration where performance begins to diverge, signaling when A, starts to mean- ingfully affect learning. Early on, the critic signal is weak, and A, is driven mostly by the utility term. When & is large enough, shaping accelcrates the critic's contribution by up to 50 iterations and leads to around 2.5x higher return compared to the unshaped case.
In the absence of shaping (E 0. gray lines), this occurs relatively late (iteration 131), whereas with shaping < > 0). happens substantially earlier (iterations 81-113, depending on Q). This shows that the utility term not only supports early learn- ing but also accelerates the emergence of reliable critic. These results validate our choice to softly shape advantages, and emphasize the importance of carefully tuning & and ry: insufficient shaping slows critic learning, which in turn leads to substantially lower mean returns.
Sparse-Reward Regimes). In sparse-reward environ- ments, standard policy gradient methods such as PPO may exhibit near-zero expected gradients in early training, as reward-based advantage estimates A, are often uninformative until a success- ful trajectory is observed. The non non-vanisee vanissiin update result (Theorem 1) implies that the proposed shaping objective induces a consistent gradient signal & Vo even when reward-based advan- tages are weak. This additional structure modifies the local optimization landscape by providing an informative descent direction derived from trajectory similarity, therehy facilitating earlier and more stable optimization.
Remark (Optimization Landscape
G.3 RELATIVE WALL TIME
We measure relative wall-clock time as the end-to-end runtime per iteration to assess each method's computational burden. Environments with a more complex step logic, such as DTS- TRACTED DOORKEY, which involves door toggling, key collection, and distractor dy- namics, incur higher per-step simulation costs. Tasks like REDBLUEDOOR and LAVACROSS- ING further increase runtime through frequent failures that trigger repeated episode resets and buffer re-initializations. In contrast, FROZEN LAKE'S tabular, low-dimensional transitions execute very quickly, so all methods complete rapidly (we do not run the online variant here since the offline approach suffices). Occasional LLM queries introduce network latency that further raises wall time in the slower domains. As a result, relative wall time grows with both the intrinsic simulation complexity of the en- vironment and any additional algorithmic over- head (e.g., L.M calls).
Time 0.5 Return
Time to 2k Step
Fronea ako
Time (ali))
Lava Consing
Besha
Dookkr
Inurada Doarkey
Lonerrsnnng
Battter
RuBall
D. Lreake
Time (min)
Baselne
(octline)
contine)
Figure 16: Wall-clock runtimes across environ- ments. Time required to reach 0.5 return (left): PPO reaches 0.5 only on FROZEN LAKE, while both MIRA variants converge across tasks. Run- time for 2k training steps (right): Online MIRA incurs extra overhead from initial LLM queries, but this cost reduces wasted exploration and leads faster convergence in terms of overall wall time.
32

---

Published as a conference paper at ICLR 2026
Figure 16 reports wall-clock times for two measures: reaching 0.5 return (left) and completing sk-step run (right). In the left panel, PPO reaches 0.5 only on FROZEN LAKE, while both MIRA variants converge across all environments. In the right panel, PPO shows the lowest per-step runtime because online MIRA incurs some additional cost from its initial LLM queries. However, early queries reduce wasted exploration, allowing online MIRA reach 0.5 return much faster overall, yielding net gain in efficiency despite the upfront overhead.
A complete view of these trade-off's comes from considering both Figure 16 and the right-hand panel of Figure 7. Together, they show how higher return and wall-clock time interact when online LLM latency is present. Although online MIRA incurs additional latency from occasional queries, a substantial portion of this cost is offset by faster policy improvement: the agent spends less time in unproductive exploration and reaches competent behavior sooner. This can be seen directly by comparing the left and right panels of Figure 16: online MIRA has slightly higher per-step runtime, yet reaches the 0.5 return threshold earlier in wall-clock time. The right-hand panel of Figure reinforces this result, showing that the higher cost of online queries is compensated by more rapid performance gains.
G.4 MEMORY GROWTH ANALYSIS
135
Memory Growth
We examine how the memory graph evolves dur- ing training. MIRA updates the graph only when computing the utility term during advantage estima- tion, so memory expansion occurs once per training batch rather than at every environment step. Because batches contain full trajectories, their lengths vary across environments. To obtain a consistent summary across tasks, we record the memory size every 100 training iterations.
Menory -3
RedBall
Dankay
5
Buratin
Figure 17: Memory size growth over train-
ing. Early growth is followed by conver- gence once trajectories become consistent, with final sizes increasing from REDBALL to DOORKEY DISTRACTED DOORKEY in line with task complexity.
Figure 17 reports memory growth for REDBALL, DOORKEY, and DISTRACTED DOORKEY. In all three environments, memory grows quickly during early training, reflecting the period where the agent encounters diverse high-return segments and, when applicable, issues LLM queries that generate candidate memory additions. As trajectories converge to consistent solution, memory expansion slows and eventually stabilizes. The plateau levels flect the structural demands of each task: REDBALL retains only a small set of nodes due to its simple subgoal structure, whereas DOORKEY and DISTRACTED DOORKE: KEY require richer collec- tion of region-anchored segments. DISTRACTED DOORKEY ends up with the largest memory, as the distractor objects create additional path variants that help guide the agent, while the total size remains bounded and does not grow throughout later training.
G.5 QUERY FREQUENCY PERFORMANCE SUMMARY
Table 1 expands on Figure 7 in Subsection 4.2. It shows how different online query budgets impact learning progress (SR90Retum, indicating the mean return when success rate first exceeds 90%). )), final return, and convergence speed (total steps to termination). The results reinforce that while all MIRA variants outperform PPO, higher online budgets further accelerate training and improve asymptotic performance.
Table 1: Performance on DOORKEY. SR90Return is the mean return when success rate first exceeds 90%; Final Return is the return the end of training; Final Step is the total environment steps. MIRA variants outperform the baseline in both early and final return, with MIRA (large) achieving the highest values while converging fastest.
Method
SR90Retum
Final Return
Final Step..
Baseline
0 = 0.002
0.009 + 0.001
10362
MIRA (offline)
0.233 + 0.087
0.295 + 0.123
10351
MIRA (mid)
0.284 0.065
0.902 + 0.012
10257
MIRA (large)
0.851 0.060
0.91 + 0.013
9961
33

---

Published as a conference paper at ICLR 2026
G.6 MINIGRID PERFORMANCE SUMMARY
Tables and report detailed numerical results for all four MiniGrid tasks, including mean returns and success rates averaged over unseen seeds. MIRA consistently outperforms both PPO and the hierarchical baseline across all environments, including the more complex ones such as DOORKEY and REDBLUEDOOR. Welch's t-tests (Ruxton, 2006) show statistically significant difference between MIRA and LLM4Teeah at the 0.05 level across metrics and environments (Table 4). These results support the aggregate performance trends in the main text (Figure 6), demonstrating that MIRA improves both final return and task completion.
Table 2:
Mean return on unseen seeds across MiniGrid environments. MIRA achieves high and stable success, comparable to LLM4Teach, despite requiring substantially fewer L.M queries.
Method
DOORKEY
LAVACROSSING
REDBLUEDOOR
REDBALL
Baseline RL 0,018 = 0.016
+ 0.012 0.027
0.044 + 0.042
0.329 = 0.205
HRL
0.852 + 0.017
0.798 + 0.090
0,830 = 0.021
0.939 + 0,046
LLM4Teach
0.912 1 0.075
0.884 1 0.100
0.901 +: 0.082
0.946 + 0.051
MIRA
0.898 + 0.093
0.855 + 0.132
0.911 0.077
0.942 + 0.054
G.6.1 T-TEST: MIRA vs. LLM4TEACH
To assess whether the performance differences between LLM4Teach and MIRA are statistically significant, we conduct Welch's t-tests on the evaluation metries across environments and seeds. Welch's t-test is two-sample statistical test that does not assume equal variance. As shown in Table none of the differences reach significance at the a = 0.05 level. This suggests that MIRA performs comparably to LLM4Teach across all reported metrics, despite MIRA having small lower final reward.
H LIMITATIONS
While MIRA improves sample efficiency and reduces reliance frequent LLM queries, it also comes with natural trade-offs. The method relies on offline LLM outputs to initialize its memory graph, which, if they include misleading information dynamics, can slow convergence or increase the need for online queries. Our screening and pruning mechanisms reduce this risk, and in practice it largely limitation of current LLMs that is expected to diminish as models improve. MIRA also introduces shaping terms that require hyperparameter tuning avoid instability between the actor and critic. We find, however, that they can be adjusted with standard procedures. Finally, our current study focuses on discrete action spaces; extending MIRA to continuous domains without discretization a natural next step.
not well aligned with the environment
1 REPRODUCIBILITY
Experiments were run on both Linux server with Intel Xeon E5-2630 v4 CPUs (40 threads) and an Apple M2 (8-core CPU, 10-core GPU, 16GB unified memory). All LLM models used in our experiments correspond to the publicly available versions released in the first week of August 2025.
I.1 SIMULATION PLATFORMS
I.1.1 GYMNASIUM TOY TEXT
ENVIRONMENT DETAILS. Horizon indicates the maximum number of steps per episode before automatic termination (i.e., maxsteps in the environment configuration).
34

---

Published as a conference paper at ICLR 2026
Table 3: Success unseen seeds across MiniGrid environments. MIRA achieves consistently high success rates, matching LLM4Teach while requiring fewer queries, and outperforming baseline and HRL methods.
Method
DOORKEY
LAVACROSSING
REDBLUEDOOR
REDBALL
Baseline RL
0.023 = 0.017
0.017 = 0.020
0.036 + 0.043
0.539 + 0.064
HRL
0.897 0.013
0.841 0.085
0.892 + 0.012
0.956 0.025
L.M4Teach
0.970 0.004
0.911
0.011
0.956 0.003
0.958 0.021
MIRA
0.953 0.043
0.913 = 0.077
0.944 + 0.020
0.956 + 0.036
Table Welch's I-test comparing LLM4Teach and MIRA (MR: Mean Return SR: Success Rate). None of the differences are statistically significant at a 0.05.
Metric
LLM4Teach
MIRA
p
95% CI
DOORKEY (MR)
0.912 = 0.075
0.898 = 0.093
0.203
0,8495
[-0.181, 0.209]
DOORKEY (SR)
0.970 = 0.004
0.953 0.043
0.682
0.5647
[-0.0885, 0.1225]
LAVACROSSING (MR)
0.884 = 0.100
1.85=00133
0.303
0.7778
[-0.2443, 0.3023]
LAVACROSSING (SR)
0.931 + 0.011
+
0.401
0.7260
[-0.1681, 0.2041]
REDBLUEEDOO (MR)
0.901 + 0.082
0.911
0.077
-0.154
0.8851
[-0.1906, 0.1706]
REDBLUEDOOR (SR)
0.956 = 0.003
0.020
1.028
0,4081
[-0.0362, 0.0602]
REDBALL (MR)
0.946 = 0.051
0.054
0,093
0,9302
[-0.1152, O. 0232]
REDBALL (SR)
0.958 0.021
0.956 0.036
0.083
0.9387
[-0.0717, 0.0757]
Table 5: FrozenLake environment details.
Property
Value
Observation Type
Discrete
Horizon
200
Reward Sparsity
Sparse 4 (tabular)
Action Space
Dynamics
Slippery, irreversible
HYPERPARAMETER.
Table 6 provides the main specifications of FrozenLake for PPOConfig in
RLlib.
Table 6: Hyperparameters of FROZENLAKE
Parameter
Value
Learning rate
x 10-1
Batch size
512
Mini-batch size
64
Number of epochs
4
Entropy coefficient
0.01
Discount factor (7)
0.99
GAE lambda (A)
0.95
Utility (E)
[0.9]
Batch mode
"complete episodes"
I.1.2 MINIGRID AND BABYAI
ENVIRONMENT DETAILS. Horizon indicates the maximum number of steps per episode before automatic termination (i.e., maxsteps in the environment configuration).
35

---

Published as a conference paper at ICLR 2026
Table 9: Hyperparameters of DOORKEY
Table 10: Hyperparameters of LAVACROSSING
Parameter
Value
Parameter
Value
Learning rate
2.5 x 10
Learning rate
2.5 x 10
Batch size
1024
Batch size
1024
Mini-batch size
64
Mini-batch size
64
Number epochs
Number of epochs
Entropy coefficient
0.01
Entropy coefficient
0.01
Discount factor (?)
0,99
Discount factor (r)
0.99
GAE lambda (A)
0.95
GAE lambda (A)
0.95
Utility (E)
[0.25, 0.15]
Utility (E)
[0.3]
Batch mode
"complete episodes"
Batch mode
"complete episodes"
Table 11: Hyperparameters of REDBLUEDDOR
Table 12: Hyperparameters of REDBALL
Parameter
Value
Parameter
Value
Learning rate
5 10 5
Learning Batch size
10
Batch size
1024
512
Mini-batch size
64
Mini-batch size
64
Number of epochs
4
Number of epochs
4
Entropy coefficient
0.01
Entropy coefficient
0.01
Discount factor (?)
0.99
Discount factor (?)
0.99
GAE lambda (A)
0.9
GAE lambda (A)
0.95
Utility (E)
[0.25]
Utility (E)
[0.2]
Batch mode
"complete episodes"
Batch mode
"complete episodes"
Table 7: MiniGrid suite details.
Property
Value
Observation Type
RGB
Reward Sparsity
Sparse and delayed 7 (tabular)
Action Space
View Size
Horizon
300
Table 8: MiniGrid environments and their dynamics.
Environment
Dynamics
REDBALL REDBLUEDOOR
Reversible
Inreversible
LAVACROSSING
Inreversible
DOY DISTRACTED DOORKEY
Subgoal seq.
+Visual distractors
HYPERPARAMETER. Tables 10- 12 provides the main specifications of all the MiniGrid environ- ments for PPCConfig in RLlib.
OBSERVATION SPACE In MiniGrid environments. the agent receives an RGB image of the grid, which is passed through convolutional encoder 18 to extract spatial features relevant for navigation and interaction.
36

---

Published as a conference paper at ICLR 2026
26
2 2
RGlingbbsrragee
Acy_branch
56
uR__branc
Conv2d
Figure 18: Convolutional encoder architecture used to process the agent's 56 x 56 x RGB observa- tion in MiniGrid environments. The input passes through series of Conv2D layers, reducing spatial dimensions while increasing channel depth. The final activation is flattened and fed to both policy and value heads. This encoder captures spatial layout, object presence, and agent-centric context for decision-making
This CNN processes the visual input into compact feature vector, capturing object positions, colors, and layout structure. The resulting embedding is concatenated with a learned directional encoding and passed to the policy and value heads for action selection and value estimation.
1.2 LLM CONFIDENCE SETTINGS
For completions where token-level likelihoods are available, confidence is computed using an expo- nential of the geometric-meat log- pee-babbbbbtyyy (exp E (i)) with fixed likelihood threshold of 2 0.65. When likelihoods are unavailable, we obtain k independent completions (k = 3) and retain only outputs that pass a majority-consisteng
with fixed agreement threshold 2/3.
USE OF LARGE LANGUAGE MODELS (LLMS)
During the preparation of this manuscript, the authors used OpenAl's ChatGPT assist with gram- mar and readability. No research ideas, technical content, or analysis were generated by the tool. All content was reviewed and verified by the authors, who take full responsibility for the final version.
37


Reviews:
[{"review_id": "hKLcVa8a2G", "reviewer": "Reviewer_e51T", "summary": "This paper proposes to use LLM's capability of generating trajectory-based plans to guide the online training of an RL agent. Since LLMs are expensive to query, the goal is to query it only a few times while training a performant RL policy. The paper proposes to integrate LLM guidance by maintaining a graph of goals, subgoals, and trajectories, and use such a graph to compute auxiliary utility signals (similar to reward shaping) to accelerate policy learning with PPO. Experiments show that the proposed algorithm can outperform RL without LLM guidance and some other baselines.", "questions": "(See questions above)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Using LLM to provide guidance on learning for decision making is a problem that has gained a lot of recent attention..\n\n- The idea of using a graph to represent the current knowledge learned, including subgoals, and final goals is interesting\n\n- Experiments in FrozenLake and MiniGrid are helpful in illustrating the utility of the proposed approach", "weaknesses": "- Despite having good performance in gridworld environments, the paper did not discuss the applicability of the proposed method in other RL environments, e.g. continuous control (Pendulum, Mountain Car), and stochastic transition (e.g. Pacman). I assume that in continuous control, defining subgoals can be a challenge? When one has stochastic transition, then having a trajectory-based plan may not be feasible?\n\nAlso, looking at Figure 8, it looks like significant prompt engineering is needed to allow the LLM to output useful trajectories. \n\n- For the experiments, I see that the proposed method can improve over LLM4Teach in Distracted DoorKey, but it looks like from Table 5 that LLM4Teach slightly outperforms the proposed method. Am I missing something? Is the benefit of MIRA more on the computational side, in that it makes fewer LLM queries than LLM4Teach? A comparison between the query costs between these algorithms seems important.\n\n- (Clarity) it would be nice if the paper can provide a full pseudocode that incorporates screening unit, the maintaining of the memory graph, the calculation of the utility signal. Some parts, e.g. the similarity function s, the \\rho function, and the \\hat{r}_m, and c_m in (2) are not clear to me. \n\n- I am not sure if Theorem 1 reflects the utility of the shaped advantage. Is U_k^bonus - U_max <= 0? Then the improvement rate of the proposed algorithm can be slower than that of the PPO baseline?\n\n- After reading the paper, I don't have a good idea about when to use MIRA(offline) versus MIRA(online). From Figure 14, it looks like MIRA (offline) does quite well despite being simpler. But from Figure 6 it looks like MIRA(online) can significantly improve MIRA(offline). Can the authors comment on this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2zP8MACTVP", "reviewer": "Reviewer_tfVi", "summary": "The authors propose MIRA (Memory-Integrated Reinforcement Learning Agent), a novel framework that integrates LLM guidance into RL using adaptive advantage shaping along with a structured, evolving memory graph. Empirically, MIRA outperforms standard RL and hierarchical RL baselines on a suite of sparse-reward tasks (MiniGrid, BabyAI). It achieves final performance comparable to heavily-supervised, query-intensive LLM-RL methods while requiring substantially fewer LLM queries.", "questions": "[Q1] How about the performance on longer runs? Will HRL gradually surpass MIRA?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "[S1] The core idea of ​​“adaptive advantage shaping” is well-motivated and has a solid theoretical grouding. Compared with heavy supervision using LLM (e.g., modifying rewards, which will change the structure of the MDP and affect asymptotic convergence), advantage shaping allows for initial training under LLM guidance and gradual evolution without being limited by the imperfections of the initial LLM guidance.", "weaknesses": "[W1] The adaptive advantage shaping mechanism requires annealing to zero to guarantee asymptotic convergence. This makes MIRA only effective for the initial learning phase, such as accelerating exploration. It cannot sustain an advantage using imperfect LLM signals.\n\nBesides, the annealing strategy introduces new hyperparameters ($\\eta_t$ and $\\xi_t$), which are crucial to performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kLlMpeQ7Lc", "reviewer": "Reviewer_9USL", "summary": "This paper presents MIRA, a reinforcement learning agent that integrates LLM-generated subgoals into a dynamic memory graph to accelerate learning in sparse-reward environments. By computing utility signals from this structured memory to guide early training while gradually reducing LLM dependence, MIRA achieves superior sample efficiency and matches LLM-teacher performance with significantly fewer queries, supported by theoretical convergence guarantees and empirical validation.", "questions": "1. Handling Conflicting or Erroneous LLM Guidance in Memory. Could the authors clarify the exact mechanism for dynamic graph updates—specifically, how conflicting agent experience and LLM-derived subgoals/trajectory recommendations are resolved in the presence of incorrect LLM priors? What is the recourse if LLM hallucinations are initially \"locked in\" to memory?\n\n2. Scalability Beyond Grid Worlds. All experiments are conducted in discrete, grid-based environments. Have the authors attempted to apply MIRA to more complex domains—such as continuous control, vision-based robotic tasks, or high-dimensional state spaces? If not, what are the anticipated bottlenecks (e.g., graph scalability, LLM prompting overhead, or similarity computation in pixel space)? Addressing this would clarify the method's potential for real-world deployment.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Strong methodological integration: MIRA's use of a structured memory graph, which is co-populated by both agent experience and LLM-derived subgoal decompositions (see Section 2.1), is a compelling hybridization of model-based memory with language-derived task priors. This approach is well-justified, especially for environments where exploration is bottlenecked by sparse feedback.\n\n2. Empirical rigor and benchmark variety: MIRA is evaluated in breadth across Gymnasium ToyText, MiniGrid, and BabyAI environments, occupying both tabular and partially observable/visual input regimes (see Section 3.1, Figure 2). The baselines (PPO, hierarchical RL, LLM-based reward shaping, and teacher models) are appropriate and state-of-the-art.\n\n3. Efficient and transparent ablation studies: The experiments systematically examine online vs. offline LLM guidance, varying query budgets, effects of unreliable LLM outputs, and different LLM models (Figure 6), elucidating the value and robustness of the proposed approach.", "weaknesses": "1. Insufficient Detail on Memory Graph Mechanics: While the memory graph is central to MIRA's design, the main text lacks operational clarity on key aspects—such as criteria for adding or pruning subgoal nodes, triggers for new LLM queries, and mechanisms for resolving conflicts between LLM suggestions and agent experience in dynamic environments. Critical implementation specifics are deferred to the appendix, and Figure 1 (the purported schematic of the graph) is absent from the main paper, impeding reproducibility and obscuring the precise novelty of the graph construction process.\n\n2. Narrow Empirical Scope: All experiments are confined to low-dimensional, grid-world environments. The absence of evaluations in high-dimensional, continuous, or real-world settings (e.g., robotics, vision-based control, or multimodal tasks) limits confidence in the method’s claimed generality, despite assertions of broad applicability and memory efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "47h7B1OJCi", "reviewer": "Reviewer_VZaE", "summary": "The paper introduces MIRA, a policy gradient method that adds a memory graph built from high-return rollouts and occasional LLM outputs. The memory induces a per-step utility signal that is added to the advantage to form a shaped advantage. The shaping weight decays while the standard advantage weight rises, so the influence of the utility fades during training. The claim is that this improves early learning in sparse reward tasks without hurting final convergence, and reduces the number of online LLM calls relative to teacher or reward shaping approaches. The paper evaluates on FrozenLake and several MiniGrid and BabyAI tasks, compares with PPO and two LLM-based baselines, and presents a PPO-style improvement bound under boundedness and scheduling assumptions.", "questions": "- Can the authors discuss what global information the offline LLM can see and whether baselines are allowed the same view. Provide runs where offline priors are restricted to the same partial view as the agent to show robustness. The FrozenLake discussion already notes that slipperiness is hidden from both, but the global grid is seen by the LLM. Please add a matched setting. \n\n- It is better to state the exact thresholds, number of samples in the agreement test, and the mapping from token log probabilities to confidence. Include a sensitivity study. If token log probabilities are not available, explain the substitute and its effect.\n\n- Can the aughoors give full formulas for the similarity function, the goal alignment term, the confidence mapping, and the way predicted reward and confidence combine? Report the run time cost of matching and the memory size growth as a function of steps.\n\n-Since the method relies on screening and logit penalties, the lack of precise settings may hide brittle behavior or conflicts with PPO clipping. This is a correctness risk rather than a style only, because poor settings can cause learning to diverge.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The shaped advantage $\\tilde{A}_t=\\eta_tA_t + \\zeta_tU_t$  drops in with no change to policy or critic structure. The paper repeatedly stresses that the utility is additive and scheduled to zero, which keeps the core PPO update intact. This design choice makes the method easy to implement and likely to work across actor-critic variants. \n\n- On several MiniGrid and BabyAI tasks, the method reaches higher success faster than PPO and a hierarchical baseline, and does so with modest LLM usage. The narrative connects each gain to either offline memory, occasional online advice, or both.", "weaknesses": "- The paper describes utility as a similarity-weighted score over stored trajectory segments that also uses a goal alignment factor and LLM confidence, and a predicted reward for the memory node. However, the exact form of the similarity function and the cost of matching are not specified in the main text. Without a precise definition, it is hard to reason about bias and computational overhead, and to reproduce the effect. \n\n- The improvement relation includes a term with a utility bonus minus a uniform cap. If the utility is badly calibrated early, the cap term may dominate and make the bound weak. The paper does not give conditions under which the utility contribution is reliably positive beyond the boundedness itself.\n\n-The screening unit filters online suggestions using sequence likelihoods or agreement across samples. In many LLM APIs, calibrated token log probabilities are limited or absent, and agreement can be brittle. The paper does not specify thresholds, how they are tuned, nor show sensitivity. Since confidence and a predicted reward weigh the utility, miscalibration can bias the shaped advantage.\n\n-For accepted online guidance, the method injects penalties into logits to suppress actions. The bounds on the penalty and its interaction with PPO clipping are not made precise in the main text. This matters for stability, since even bounded penalties can alter action selection in a way that conflicts with the critic and with the clipping rule if the scaling is not set with care.\n\n- Using offline priors with full environment context can give extra information not available to the agent or to baselines. This can inflate gains attributed to MIRA rather than to the prior. The paper should make this explicit, and either restrict priors or give fair matched baselines. The FrozenLake description makes this a real concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)