Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization
Authors: Zeyuan Liu
Jeonghye Kim
Xufang Luo
Dongsheng Li
Yuqing Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
LLM Agent
Exploration
Primary area: reinforcement learning

Abstract: Exploration remains the key bottleneck for large language model agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel states. We propose EMPO$^2$, a hybrid RL framework that leverages memory for exploration and combines on- and off-policy updates to make LLMs perform well with memory while also ensuring robustness without it. On ScienceWorld and WebShop, EMPO$^2$ achieves 128.6% and 11.3% improvements over GRPO, respectively. Moreover, in out-of-distribution tests, EMPO$^2$ demonstrates superior adaptability to new tasks, requiring only a few trials with memory and no parameter updates. These results highlight EMPO$^2$ as a promising framework for building more exploratory and generalizable LLM-based agents.

Published as a conference paper at ICLR 2026
EXPLORATORY MEMORY-AUGMENTEI LLM AGENT VIA HYBRID ON- AND OFF-POLICY OPTIMIZATION
Zeyuan L Jeonghye Kim 12 Xufang Luo't, Dongsheng Li', Yuqing Yang'
Miicosooft Research "KAIST gri tmaybe@gmail com, jeonghye. kimekaist.ac.
(xufluc, dongsli, yugyangg/mmccrosoftto
project page
aggett
ABSTRACT
Exploration remains the key bottleneck for large language model agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel states. We propose Exploratory Memory-Augmented On- and Off-Policy Optimization (EMPO*). a hybrid RL framework that leverages memory for exploration and combines and off-policy updates to make LLMs perform well with memory while also en- suring robustness without it. On ScienceWorld and WebShop, EMPO² achieves 128.6% and 11.3% improvements over GRPO, respectively. Moreover, in out- of-distribution tests, EMPO² demonstrates superior adaptability new tasks, re- quiring only a few trials with memory and no parameter updates. These results highlight EMPO² a promising framework for building more exploratory and generalizable LLM- agents.
INTRODUCTION
Large Language Models (LLMs) have recently emerged as powerful agents capable of reasoning. planning, and interacting with external environments (Achiam et al.) 2023 Park et al.) 2023 Yao cct al. 2023 Kim et al., 2025). When combined with reinforcement learning (RL), such agents can adapt their behavior based on experience and feedback, enabling them go beyond static prompting
or supervised fine-tuning (Guo et al.) 2025
Tan et al. 2024). This paradigm has driven recent
SclenceWoria spower components
In-Distribution
DustottDittibutiin
+11.2%
16.1
10
73.5
75
60
13.5
13.2
-15
Maa
Wc1bhop
Chenetry
Figure 1: (a) Comparison of the learning curves of GRPO and 2MPO² (ours) on the Science- World power- component task. While GRPO converges to suboptimal performance. EMPO² continues to improve and accomplish the task. (b) Comparison of EMPO² and other baselines in in-distribution (ID) and out-of-distributtin (OOD) settings on and WebShop. In ID experiments, it adapts well to familiar environments, achieving 128.6% shop improvements over GRPO. In OOD experiments, it also shows strong performance with few trials and weight updates, indicating effective use of memory to explore unfamiliar environments. Full results are in Tables I and Figure
FrienceWorle and 11.3% % Web-
Equal contribution: work done during an internship at Microsoft Research. Corresponding author.

---

Published as a conference paper at ICLR 2026
progress in areas such interactive decision-making, tool use, and embodied Al (Feng et al. 20256
Lu al. 2025b Feng et al. 2025a Dong et al. 2025 Luo et al. 2025).
However, key limitation of current LLM-based agents lies in their reliance exploiting prior knowledge rather than engaging in systematic exploration. While RL frameworks emphasize bal- ancing exploration and exploitation, many LLM-agent systems primarily leverage pretrained knowl- edge and conduct only limited search within familiar distributions. As a result, these agents often struggle in environments where progress depends on discovering novel states or actively acquiring new information, rather than reusing what is already known.
To address this challenge, recent research has incorporated external memory modules into LLMS as a form of long-term memory. This enables models to leverage past experiences to correct failed at- tempts, thereby improving decision-
in subsequent trials without requiring parameter updates (Shinn et al.) 2023 Zhang et al., 2023). However, as noted in Zhang et al. (2023), the performance of such methods tends to saturate quickly, since collecting experiences with static parameters cannot fully capture the diversity needed for continuous improvement.
In this work, we present a unified framework that enables LLM agents to learn more effectively through broader exploration by jointly updating their parametric policy parameters with RI. and their non-parametric memory module through interaction. Crucially, the non non-maraettric up- dates not only complement but also enhance the efficiency of paramet- ric learning, thereby enabling more effective exploration and adaptation. This dual-update paradigm serves as a bridge between parameter-leve optimization and memory-augmentee reasoning. While memory is uti- lized during learning, moving toward more generalizable intelligence re- quires reducing dependence on external memory and instead embedding its benefits directly into the model's 's parameters. To this end. we pro- pose Exploratory Memory-Augnen Auggeneed On- and Off-Policy Optimization (EMPO?), a new hybrid RL algorithm that incorporates two modes in the rollout phase- depending on whether memory is used and two modes in the update phase-on-polee on-policy and off off-policy learning- teaniing-thereby thereby enabling agents to leverage memory when available while remaining robust in its absence.
Non- Non-parametric
Parametric
Reflesion REMEMBERER
EMPO'
PPO GRPO GIGPO
Bclesting
Figure 2: Non-parametric updates can encourage exploration, bootstrapping parametric updates.
In our experiments, we evaluate EMPO² on two widely used multi-step embodied reasoning envi- ronments that require exploration to solve complex tasks: ScienceWorke (Wang et al.) 2022) and WebShop (Yao et al.] 2022). We compare its performance against a range of non- paramarraeee and parametric (offline and online) RL approaches. As summarized in Figure I EMPO² substantially outperforms prior algorithms,, achieving a 128.6% improvement on Science World and an 11.3% improvement WebShop over the strong online RL baseline GRPO. The training curve in Fig- ure [ (a) further shows that, unlike GRPO, which converges prematurely to a suboptimal solution, EMPO² leverages continuous exploration and successfully solves the task. Moreover, for the OOD experiments (Figure I rightmost), the model also achieves good scores with only a few trials and no weight updates, indicating that the updated model has acquired the ability to use memory to explore unseen or unfamiliar environments. These results highlight EMPO² as a promising direction for building more adaptive and generalizable embodied agents.
PRELIMINARIES
Online RL consists of alternating between rollout phase, in which trajectories are generated using the current policy parameterized by 0, and an update phase, in which the policy is optimized based on those rollouts.
Policy Rollout. We consider setting where, given sampled task u ~ p((f), an LLM agent solves the task through multi-step interactions with the environment. Starting from task u, the LLM Ty generates the first natural-language action ai Te(- u) E A. Executing this action, the envi- ronment returns a reward r1 and the next state $1. At a general timestep f, conditioned on the current state st and the task the policy produces the next action a++1 T- st, u). This interaction loop continues until the task is completed or a maximum number of steps is reached. A rollout trajectory is thus defined as the sequence of states, actions, and rewards, (u, 01 , T1, 81, 02 (2....,
2

---

Published as a conference paper at ICLR 2026
Group Relative Policy Optimization. Group Relative Policy Optimization (GRPO) (Shao al. 2024) updates the policy by comparing multiple rollouts of the same task u, removing the need for the value function in PPO (Schulman et al. 2017). Given a task u, the policy To generates N rollout trajectories {-(1), ..... ,....
r(M], Each trajectory receives a return {R(() ...., R(()) defined as the sum of rewards along the trajectory: R( D r= E
(6) For each action a; taken trajectory 7(),
define its relative advantage A(a,") (6) R >> R(3)
where actions from trajectories with higher-than-average reward obtain positive advantage, while those from lower-performing ones obtain negative advantage. The GRPO loss is then:
a(R)
N
T
Σ
Σ
min (po(af)) )((a))) -
unipll)) Fimm MRR
NT
==1
3 DKL (-(uu) Ter(-)u))
no To(a with B 2 controlling the regularization strength toward refer-
where po(at
Toas (az [s(")
ence policy Tief.
THE EXPLORATION PROBLEM OF LLM AGENTS
LLMs encode rich prior knowledge, but such priors often fail reflect the actual rules or dynamics of a given environment. Blind reliance on these priors can lead to erroneous behaviors, making it necessary for agents to adapt through direct interaction and trial-and-erros. such adaptation is exploration, which involves seeking information beyond pre-training, sometimes by taking atypical or counterintuitive actions. However, current Lased agents struggle with this (Qiao et al. 2024 Zhou et al.] 2024). as it demands stepping outside the distribution of behaviors where the model feels most confident.
key requirement for
Consequently, many prior studies have sought to align agents with new environments through warm- start supervised fine-tuning (SFT) using numerous golden trajectories (Song et al. 2024; Qiao et al.
2024 Xiang al.) 2024). leveraging large-scale models such as GPT-4 (Tang al. [ Lin al,
2023 or employing human engineering or well-established simulation information (Choudhury &
Sodhi 2025). While these methods achieve strong results in constrained settings, their effectiveness
is limited to cases where such external support is available, and they generalize poorly to unseen scenarios without it.
Your furn on light focus on light
Teetk Your is turn red light facus
Environmen:: This roum is poclled the aot tway. it. see the ageeti, sortale celedd set, pecture...
Environment: This roum cxiled the for twayy ln you soe agent, sutacamoo called wi, picture
Action: focus
Fall
Actioni focus bulb
Fall
T220
100
Episode Return of "Turn the Red Light Bulb" during GRPO Training
-100
100
150
200
Step
Figure 3: When training LLM with GRPO in ScienceWorkks the agent struggles because of insufficient exploration. For instance, in the task "turn on the red light bulb," the agent must first find the red light bulb before activating it. However, the agent fails to locate it and, as result, cannot complete the task. Rather than analyzing the cause of failure and exploring alternative actions, the agent proceeds unchanged, its score stagnates even as additional training steps are taken.
Therefore, we focus on how to efficiently train agents in online RL through trial and error, with- out any prior embedding of the environment's rules. The key challenge is that, without intrinsic exploration capability, online RL struggles to optimize effectively. As illustrated in Figure 3 in Science (Wang et al. 2022) environment the agent is given the mission "tur the red Tight
3

---

Published as a conference paper at ICLR 2026
bulb." The instructions specify that the agent should first focus the light bulb and then build circuit to activate it, based on the current room observation. However, since no red light bulb is present in the observation, the agent must search the environment locate it. Instead, the agent follows the instruction literally, attempts to focus on the red light bulb, and fails because it does not exist in the room. Ideally, when agent fails to reach its goal, it should analyze the reasons for failure and broaden its action space to discover successful strategies. Yet in representative online RL algorithms GRPO (Shao et al. 2024). prior trajectory rollouts provide no continuity beyond scalar reward signal, thereby restricting exploration and ultimately limiting learning.
METHOD
In this section, we present Exploratory Memory-augmented On- and Off-Policy Optimization (EMPO2), novel algorithm aimed tackling the exploration challenges in online RL. EMPOa operates in two modes for both rollout phase and update phase. During rollout, actions can be gen- erated either through (1) prompting without memory, where no retrieved information is used, or (2) meeorryaaugment phase, rollouts with memory-augmented prompting are used in two ways: (a) on-policy, where tips retained and the update is performed with the original prompt, and (b) off-policy, where tips removed during update. Notably, tips are generated not by separate model but by the policy ma itself, which is continually updated during training. The full algorithm is provided in Appendix A
prompting, conditioned on [tips] retrieved from memory. In the update
4.1 ADVANCING EXPLORATION WITH SELF-GENERATED MEMORY
A key component of EMPO2 is its use of memory to maintain continuity across rollouts. Information obtained from an agent's interactions can be encoded into parameters through policy optimization, but it can also be recorded in an external memory that the agent continuously consults. Since our policy is initialized from a pretrained LLM with inherent summarization and reflection abilities, these abilities can be leveraged as auxiliary signals in addition to scalar rewards, thereby guiding exploration more effectively. To realize this, EMPO² integrates both parametric (parameter updates within the LLM) and con-arrametric (external memory) updates, strengthening the linkage between rollouts and promoting exploration, with all data and guidance generated autonomously by the agent.
Thakk Your laakia un the light buto Pirat, focua rud light bulk. cruale edeciiical ciout powsra Loc"
Tealc jac tarn not tulti first, Aeauu light burt, Then, cranes
chcuil thad ponets and
Envhonmant This more called the wallwiyy in see:
Eewitomment This room called the tailway, Init Kotiaii go warkshhpp
agant,
Astioni focas on rediight
Enwinonsent
Suocess
Parametrio policy update
tips
Thco
The red light bulb ion't here s0 you
fips
Exploration guided by non-parametric memory updates and off off-policy internalization drives agent training.
Non paramatric memory update
Figure 4: In EMPO?, the current policy parameters me are used to review past rollouts, with the resulting insights added to memory. This updated memory conditions subsequent rollouts and pro- motes exploration.
In the non aanpaaraettrc updates, similar to Reflexion (Shinn et al., 2023), the agent reviews past rollouts, generates self-guidance tips], and stores them in memory. These tips help the agent avoid repeated mistakes and explore new strategies. Unlike Reflexion, focuses on iterative verbal guidance to achieve higher rewards in the next trial, our approach aims for these tips to lead to enhanced exploration that is ultimately consolidated through parametric updates.
Gelf-Generated Memory and Tips. We define a memory buffer M = {tip,, tip which stores reflective tips generated by the policy mo during trajectory reflection. Formally, when an episode i of task u terminates at timestep t. the policy takes the final state s; together with a tip- generation prompt input and produces tip, where tip, ~ TT(ss, u, tip-generation prompt). A set of illustrative examples is provided below, while the tip-generation prompt is presented in Appendix and additional examples are included in Appendix E.
4

---

Published as a conference paper at ICLR 2026
Examples of Generated Tips ScienceWorld <power-component> task
You moved between the kitchen and bathroom but did not find green wire or green light bulb to connect.
You focused the red light bulh hut did complete the task of turning on the red light bulb. You are in hallway and need to find a way.
- TTee trajectory involves connecting the battery to the green wire terminals to power the green light bulb, but the connections to air and other objects are irrelevant.
The circuit for the green light bulb was partially connected but still missing the battery connection; the task is not fully completed.
4.2 PARAMETERIZE NON- PARAMETRII UPDATES VIA HYBRID POLICY OPTIMIZATION
Agents can use memory to improve exploration and learning efficiency, but the acquired knowledge needs be internalized into model parameters to enhance inherent capabilities. To this end, we pro- pose two modes for the rollout and update phases, whose combinations yield three hybrid learning modes (Figure
Memory
tips
Retrieve
tips
Retrieve
tips
Policy 10
tips
Palicy 1T.
tips
Palicy 14
Policy nu > Act-
Policy The
Act-
Policy Th
> Act-
Env
Env
Env
State
State State-
State State [ips
Reward
Reward
Reward
((State, Action, Reward))
((State+tips, Action, Reward))
((State, Action, Reward))
[On-pollcy learning w/o memory] [On-policy learning w/ memory]
[off-policy learning]
Rollout Mode: Prompting Without Memory
Rollout Mode: Memory-Augmented Augmented Prompting Update On-Policy
Rollout Mode: Augmented Prompting
Update Mode: On- Policy
Update Mode: Off-Policy
Figure 5: 2MPO² mode combinations. By combining the two rollout modes and update modes, three EMPO mode configurations are possible: on-policy learning without memory, on-policy learn- ing with memory and off-policy learning.
Rollout Modes. During rollouts, the agent samples between the two modes, selecting one mode each step: mode (2) with memory rollout probability p and mode (1) with probability The ablation study of can be found in Appendix F.1
(1) Prompting Without Memory. For each task at each timestep t, the policy my generates actions conditioned only on the current state 8 and the task u: a++1 ~ To( 8t. [t.u). u).
(2) Memory-Augmentedd Prompting. Retr(o:; C M selects tips most relevant to the current state 8t, e.g., via similarity search in the embedding space. We denote the retrieved as [ips).- In memory-augmented prompting. the policy conditions its action on both & and tips number of retrieved tips 10.
For each task u, each timestep
retrieval operator
ac+1 ~ To(: St, u, tips.. We limit the
Update Modes.
Trajectories generated under rollout mode (1) are directly used for updates, whereas those generated under rollout mode (2)=memory-auggneeee prompting-follow one of two update modes chosen at random during the update phase. Mode (b) is selected with off-policy update probability q. and mode (a) with probability 1 The ablation study of can be found in Appendix F.
(a) On-Policy Updates.
On-policy update uses the same prompt in the rollout, and po(a.)) in (=)(
eq becomes po(a
iip
Tew (a 1ps
5

---

Published as a conference paper at ICLR 2026
(b) Off-Policy Updates. In this mode, the stored log- probabilities 2 = log (T( st, tips are replaced with the log bog--rbbabilities assigned by the same policy Te when conditioned only on (st, u), namely e empsiip = l (e St, u). In this formulation, the advantage update is performed based on how natural the action appears under the distribution without tips.
This construction can be interpreted as a form of reward-guider tories sampled under the ips-conditioned policy act as teacher demonstrations, while the student policy TT(- 8, u) is updated to reproduce those trajectories in proportion to their advantage. High-reward trajectories (A, > 0) are reinforeed, while low-reward trajectories (A, < 0) are suppressed, resulting in selective distillation that emphasizes beneficial behaviors. tips serve as an intermediate scaffolding mechanism that improves exploration and trajectory quality, while the reward signal ensures that only advantageous behaviors are ultimately retained. Consequently, the final policy learns to internalize the benefits of tip conditioning without requir- ing tips at inference time. Appendix [ provides an illustrative breakdown and summary table for the calculation the importance sampling ratio.
knowledge distillation. Trajec-
this way,
Stabilizing Off-Policy Training. Off-policy training is prone to instabil- ity and may collapse (see Figure (). In such cases, gradient normalization, entropy loss, KL loss, and policy gradient loss can all diverge to NaN. Prior work. Yang al. (2025) shows that low-probability tokens destabilize train- ing by amplifying gradient magnitudes through unbounded likelihood ratios. Motivated by this, we introduce masking mechanism that suppresses the ad- vantage term for tokens whose probability under Ty falls below a threshold 5. Finally, the loss in Eq. iis modified as
(mars e/meaing Inear.
Figure 6: Mask- ing tokens stabilizes training.
11 min (po ((()))((a A(a clip(p) (6:t), (i (1l-,,++e).. e))((())) 1 70(a 2222
wo-p(l) (7 JooTOas
NT
3 (To((To(llu) || TT)(-
To further encourage exploration, and inspired by prior work exploration-targeted online RL (Burda al. 2018b, Bellemare et al.) 2016; Ecoffet et al. 2011),, we introduce an intrinsic reward based on the novelty of the current state. A memory list stores distinct states, and for each new state we compute its cosine similarity with existing entries. If the similarity falls below threshold, the state is added to memory and assigned reward. The intrinsic reward is defined as Tintrinsie 1/- where n denotes the number of similar past states. This mechanism encourages the agent to explore novel states even when no extrinsic reward is provided by the environment and maintains policy entropy, as shown in Figure]
Intrinsic Rewards for Exploration.
Folky Extrapy
Figure 7: Policy entropy comparison with vs. without intrinsic rewards.
RELATED WORK
LLM Agents in Multi-Step Embodied Tasks. LLM agents for multi-step embodied tasks have been studied under different paradigms. Data-driven approaches (Song et al., [024 Xiong et al. 2024 Qiao et al. 2025 2024 Tajwar et al. (025) enhance decision-making through effective data collection methods and imitation learning. Model-based agents (Tang et al., 2024 Zhou et al. 2024 build world models, often by generating code with large closed-source systems such GPT-
4. Other methods (Lin et al., 2023 Choudbury & Sodhi, 2025) strengthen reasoning through model transitions or by leveraging privileged information provided by the simulation environment. In contrast, our approach reduces reliance on such external resources and emphasizes autonomous growth through the agent's 's own exploration and self-improvement.
Memory for LLM Agents. To enable progressive improvement from past experiences, Reflexion (Shinn et al. 2023) and REMEMBERER (Zhang et al. 2023) leverage external memory. Reflexion stores verbal reflections for later prompting, while REMEMBERER records observations, actions, rewards, and Q-values, retrieving similar cases as fhw-shot exemplars. These methods show that LLMs can improve without parameter updates. However, with fixed parameters, they cannot pand intrinsic knowledge, so adaptation remains short-term (Zhang et al. (023), relying on external memory rather than achieving long-term evolution and generalization.

---

Published as a conference paper at ICLR 2026
Learning by Knowledge Distillation Our hybrid off-poolic update functions reward- reward-guided knowledge distillation during online training. Snell et al. (2022) introduced context distillation, where the model first solves tasks using a Teacher prompt (with instructions, examples, explanations, and scratch-pad reasoning) and then learns to produce the final answer from a minimal Student prompt via offline, SFT-based distillation. In contrast, integrate knowledge distillation into online RL, leveraging online adaptability while enhancing exploration for more efficient training.
RL for LLM Agents. RL provides a robust framework for optimizing LLM parameters through observations and reward signals from environment interactions. Prior work, Retrospex (Xiang et al.) 2024]. showed that offline RL, which learns optimal policies from large logged datasets, can im-
prove LLM agent performance. Recent studies focus on online RL (Shao et al. 2024 Feng et al.
2025b Wang et al. 2025), where agents learn in real time. GiGPO (Feng et al. (025b) advanced
GRPO by grouping rollouts with similar observations, enabling finer credit assignment and stronger performance. Our work advances this online RI. direction by integrating non-parametric memory updates into both on- and off-policy learning, yielding substantially higher sample efficiency.
Enhancing Exploration for Online RL. A central challenge in online RL is effective exploration. Classical methods such as count-based exploration (Bellemare et al. 2016) and Random Network Distillation (Burda et al., 2018b) use intrinsic rewards to encourage novelty. Go-Explore (Ecoffet et al. 2019) stores key states and re-explores from them, solving hard-exploration tasks like Atari games. Its LLM extension, Intelligent Go-Explore (Lu et al. 2025a), achieves strong results in environments such as TextWorld (Côté al.) 2018), but relies large closed-source models and does not perform parameter updates. In our concurrent work, RLVMR (Zhang et al. (025) employs warm-start SFT elicit diverse reasoning types (planning, exploration, and reflection) and provides dense, process-level rewards for each reasoning type during online RL, enhancing exploration and credit assignment. Together, these studies underscore the importance of structured exploration for scaling RL to complex environments.
6 EXPERIMENTS
To examine the effectiveness of EMPO?, we conduct extensive experiments on two widely used LLM agent benchmarks: ScienceWorld (Wang et al. 2022) and WebShop (Yao et al. (022) using Qwen2.5-7B-Instte (Qwen al., 2025) as the base model. The EMPO? performance evaluate is the performance of the trained model without memory at test time.
6.1 SCIENCEWORLD
ScienceWorld (Wang et al. 2022) is an interactive text-based benchmark in which an agent performs science experiments at the elementary school level. Successfully completing these experiments requires long-term multi-step planning. hypothesis testing, and interpretation of outcomes, as well sufficient exploration to determine where the necessary tools and what appropriate actions should be taken. ScienceWorke includes tasks from diverse topics and in our experiments, we cover 19 tasks spanning chemistry, classification, biology, electricity, and measurement.
Baselines. We compare EMPO² with several RL approaches. For non-parametric RI., Re- flexion (Shinn al. 2023) updates memory in a non-parametric manner by incorporating LLM reflections from previous trajectories and using them in the prompt for the subsequent trial. For offline RL, Retrospex (Xiang et al., (0244) leverages an SFT-trained model and uses Q-function learned via Implicit Q-learning (Kostrikov et al. 2022) to dynamically rescore actions. The offi- cial Retrospex paper used the smaller Flan-T5-Larg (Chung et al.) (024) (770M) and incorporated human-designed heuristics to assist the agent during evaluation. In contrast, to ensure consistency and
in our experimental setup, we standardize the base model of Retrospex Qwen2.5-7B-Inssine exclude these hcuristics. Finally, for online RL, we include GRPO (Shao et al. 2024] as represen- tative baseline. Further details are provided in Appendix D
Training Details. Our EMPO" implementation is based verl (Sheng et al. 2024 one of the
representative RL-for-LLM libraries. We extended GRPO verl from a single-step setup to a multi- step setup and incorporated both memory module and an off- off-policy loss calculation component. We use the same hyperparameter configuration for GRPO and EMPO² The prompt used is provided in Appendix B and implementation details are given in Appendix D.2
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison results of Worln. Each task in ScienceWorld contains multiple vari- ants. We use the first five variants for training and evaluate on the 20 unseen test variants. Bold shows the best performance per task, while red shading marks cases where parametric updates score lower than non-parametric updates. The EMPO² performance evaluate is the performance of the trained model without memory at test time.
@wen2.5-7B 5-7--mrrrrt
Parametric
Offline Retrospex
Online
Naive
Topic
Task
Reflexion
GRPO
EMPO7
chemistry-mne mix
32.0
1.220.7
10.8:100
12.4235
42.7:11244
Chem
chemistry-mixx mitiyy-mi---iiiiit tet---pn--secooarr----ooo secondary-colle coloo
33.02447.1
0.0:0.0
27.8.663
7.1228
33.32266
istry
chemistrymixx www.mmiiiinttettrrry--ooo
33.92:443
36.92377
7.6144
42.626.2
39.228.7
find-animal
58.22490.
39.5218
13.
72.4268
Classi
find-iviigg-hing
65. 1:4.1
36.62261
20.6:::88
68.727.1
100.0 1:000
fication
mad-non-llving--ie
35.92288
4.822.0
19.5
24.7266.
100.0:1:0.0
find-plant
1:.1
15.1
23.04335
46.2:779
100.0:10.0
Bio
identify-life-staees
48.91254
9.21224
19.0
17.9:1477
36.2::11.2
logyl
- 50.7
33.8:15.5
11.01:.7
39.5:16.0
56.3:28.1
lifespan longest- lived
61.8
44.6:16.5
55.0+1500
78.2
100.0:10.0
Bio
trespon-longeatsshrrssees sioets-ied
56.2.:63.5
34.1:5.11
38.01:550
62.3:169
100.0:10.0
logy2
fif--pan-shortest-live
66.81.6300
6.1:119
67.012238
20.611:.4
100.0:10.0
power component
99.0
6.34118
8.2+2.4
15.1
9..3=33.6
Elec
component renewable
85.0
11.7=29
10.0
24.6
92.6:20.9
tricity
nonreacwable-cnergy
test-conductivity
86.91:424
13.2:13.1
60.01:000
27.8:16.1
89.5::3.2
lest-conductivitt-oooe sogprduuttiit-----uuooow-----l
81.71:486
2.61::10
65.31:23.
9,5
71.4:16.3
Measu
mocssur--mdtting-pon----ee kubbbb
77..5
11.4:13.0
26.51::6.1
19.8:1..0
27.6:1:4.2
use-thermemeter
43.7
0.9.:0.4
32.5
7.64225
82.7
Average
17.1
the comparison results among baselines. In ScienceWorld failed tasks lead to negative rewards, producing returns between -100 and 100. The baseline Qwen2.5- 7B-Instruct obtains an average return of -61.3, which improves to 17.1 when non- non-parametric RL (Reflexion) is applied. Offline RL (Retrospex) produces substantial performance gains compared to them, but some tasks underperforms compared to non-parametric RL (highlighted in red). Online RL with GRPO also achieves considerable improvements, and its average performance is comparable to that of offline RL. However, unlike offline RL, it never underperforms non parammetrii RI., indicating that online RI. generalizes better unseen variants. Our EMPO² demonstrates substantially higher learning performance compared to all baselines. Among the tasks that initially started with negative rewards, seven reached the maximum achieved more than twice the performance improvement over GRPO, demonstrating its effectiveness in greatly enhancing learning efficiency online RL.
Main Results.
of 100. On average, EMPO?
Adaptation in New Tasks with Memory Updates. An agent post-trained on single task may exhibit limited ability generalize to new scenarios. However, EMPO?, which acquires the ability to explore by leveraging memory, demonstrates significantly stronger adaptability to novel situa- tions compared GRPO, which is trained without learning to utilize memory. Figure X illustrates how model trained on one task adapts when memory is introduced in a new task. In particular, we demonstrate cases with varying levels of topic difference. For relatively similar transition, we Biology 2 life-span--hhorest--ivee + Electricity (lest-conductivity) and
examine Biology tinct transition, we examine Biology Chemistry (chemistry-mit waiiiiippaii---ae paiin--ecoonary-aaooo secondary-color)
For more dis-
Electricity (power-component)
Biology Biology
Biology 2 Electricity
Electricity Chemistry
26.5%
Raturn pecin
70.0%
fett.m
Figure 8: Comparison GRPO and EMPO² adapting to new tasks. Step 0 has no memory, while later steps use accumulated memory as in EMPO? training.
8

---

Published as a conference paper at ICLR 2026
As shown in Figure 8 without memory (step 0), EMPO² achieves stronger baseline performance on novel tasks than GRPO. When memory is enabled, EMPO² adapts rapidly, yielding an average improvement of 136% across three scenarios within 10 steps. GRPO, by contrast, demonstrates notable gains in some cases but exhibits greater variability and, in other instances, fails to adapt to unfamiliar tasks. In certain situations, its performance Instruct base model. Though these findings are preliminary, they indicate that EMPO² has strong potential as RL framework for developing agents that are both more general and adaptable.
falls below that of the Qwen2.5-7B-
6.2 WEBSHOP
WebShop (Yao et al. 2022) is an HTMM--based online shopping environment where agents search, navigate, and purchase items according to user instructions. When the "buy" action selected, final reward is given based on how well the product's attributes and price match the criteria.
Baselines. For the WebShop experiments, we use the same baselines as in the Science experiments, with the addition of one more online RL baseline, GiGPO (Feng et al.) 2025b), as GiGPO does not cover ScienceWorke but provides benchmarking results on WebShop. The scores of Naive, Reflexion, GRPO, and GiGPO are taken from Fong et al. (2025b), while Retrospex results are re-run using the official Retrospex code with the Qwen2.5-7B. -
Instruct model.
Training Details. The WebShop EMPO² implementation builds the official GiGPO (Feng et al. 2025b) code with the same hyperparameters. Further details are provided in Appendix D.3
Main Results. Table 2 presents the baseline comparison results on WebShop. Consistent with the findings in Science EMPO² once again delivers the strongest performance. Although offline RI., online GRPO, and GiGPO each outperform non-parametric RI., GiGPO further enhances GRPO by leveraging additional advantage estimation through grouping similar observations within rollout groups. Despite these gains, EMPO² surpasses all baselines, achieving both higher scores and suc- cess rates than GiGPO. Taken together, these results indicate that EMPO² consistently demonstrates superior performance in web-based environments due its improved exploration.
Table 2: Comparison results of WebShop. Following Feng al. 2025b), average results over three random seeds and report both the mean score and the mean success rate (%). GiGPOw/ denotes the of the normalization factor Foom std, whereas GiGPOwlo snd Fram as specified in Feng et al. (2025b). The EMPO² performance we evaluate is the performance of the trained model without memory test time.
Non- Non-Parametric Refletion
Offine RI.
Online RI.
Qwen2.55-2 B.nn2.57BB.. Instruet
Naive
Retrospex
GRPO
GiGPO w/ std
GiGPO w/o std
EMPO2
Soore
26.4
58.
73.1241
79.3228
84.4129
86.222.6
88.3122
Succ.
7.8
28.8
60.4239
66.11377
72.81332
75.22338
76.914.1
6.3 ABLATION STUDY ON MODE COMBINATIONS
ScienceWorld cchermssyymiiseiissssccornnnyyee task
ScienceWorld
<power-component task
100
me0u
EMD CENI AS Ceppliy lar
Figure 9: Comparison of training curves between EMPO2 and variants that exclude either off-policy learning or on on-policy learning with memory.
As shown in Figure 5 EMPO" incorporates three mode combinations: on-policy learning with- out memory, on-policy learning with memory, and off-policy learning, and we further analyze how leveraging each affects performance two ScienceWorld tasks, where EMPO² shows significant improvements over GRPO. Figure 9 presents training curves comparing EMPO² with variants that exclude either off-policy on-policy learning with memory. As shown in the graphs, removing

---

Published as a conference paper at ICLR 2026
either component results in suboptimal learning, indicating that a balanced integration of on on-policy and off-policy updates is most effective for performance improvement. This highlights their com- plementary roles: on-policy updates contribute to stable learning, while off updates enable reasoning as guided by additional tips, and their combination yields both faster convergence and stronger final performance.
7 CONCLUSION
In this work, we propose EMPO², a novel RL method that enhances exploration in parametric RI. by leveraging non-parametric memory updates. EMPO² integrates both on-poliey and off-policy learning, thereby improving training efficiency and stability. Our experiments demonstrate that EMPO² achieves remarkable gains in training efficiency on ScienceWorld and WebShop, and further shows the ability to adapt rapidly to new domains in a few-shot manner by incorporating additional memory. An ablation study confirms the importance of the three distinct modes of EMPO?.
While our study demonstrates the potential of EMPO² as a RL framework for general agents, our current implementation for memory employs simple similarity-based search for memory retrieval. More advanced retrieval mechanisms may further enhance performance. Moreover, although our experiments primarily utilize @wwen.55-7B-Instteen families and sizes could yield deeper insights into its generality and robustness. In particular, scaling to larger models may further amplify the benefits of our approach. Beyond model scaling, applying EMPO* to new domains such mathematics, coding, multi-hop question answering, and multi- modal RL represents an exciting and challenging direction for future research. In addition, exploring other off- aff-po techniques beyond importance sampling could be of interest to achieve more stable and efficient hybrid optimization.
extending EMPO² to broader range of model
ETHICS STATEMENT
This work evaluates EMPO² on ScienceWorld and WebShop. which publicly available research benchmarks that do not include private data or sensitive information. We complied with dataset licenses and community standards for responsible use and citation, and additional data collection or modification of the environments was performed.
Although method exhibits strong adaptability in exploration and reasoning tasks, online RL sys- tems may be misapplied in safety-critical real-world contexts. To reduce such risks, we confine our study benchmark environments, and for real-world applications, responses generated by LLMs will require more careful scrutiny. We hope that future research will further address safety and broader societal impacts when extending embodied reasoning agents beyond simulation.
REPRODUCIBILITY STATEMENT
We release an Agent Lightning Luo et al. 2025 version of EMPO at agent-Lightning empo2 Additionally, we provide detailed training information, including pseudocode in Appendix [A] the hyperparameters used in our experiments, the hyperparameters for the baseline experiments, the GPU resources utilized, and code snippets for the additional components implemented in Appendix
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying baset exploration and intrinsic motivation. Advances in neural information pro- cessing systems, 29, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. / 12894, 2018a.
10

---

Published as a conference paper at ICLR 2026
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12894, 2018b.
Sanjiban Choudhury and Paloma Sodhi. Better than your teacher: LLM agents that learn from privileged Al feedback. In The Thirteenth International Conference on Learning Representations, 2025. URLbttts:
htt.lattttsppeeeieeeeeeee wttp::///epp.eeeeeeeeemo net forummid=sttaaee
Hyung Won Chung, Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetune lan- guage models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Mare-Alexandre Côté, Ákos Kädár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. Textworld: A learning environment for text-based games. CoRR, abs/1806.11 11532, 2018.
Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507. 19849, 2025.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for expllorxlorainn problems. arXiv preprint 1rXiv:1901.10995 2019.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in Ilms. CoRR, 2025a.
Lang Feng. Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for Ilm agent training. arXiv preprint AXXi:25505.10978 2025b.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501 12948, 2025.
Jeonghye Kim, Sojeong Rhee, Minbeom Kim, Dohyung Kim, Sangmook Lee, Youngchul Sung, and Kyomin Jung. Reflact: World-grounded decision making Ilm agents via goal-state reflection. arXiv preprint arXiv:2505. 2025.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations, 2022. URL https: // Spenreview http:eewwwweeeeeeeeeeeeee www.eeetfoeoo www.ooooooiiiidiim 92ZJFF
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. Swiftsage: generative agent with fast and slow thinking for complex interactive tasks, Advances in Neural Information Processing Systems, 06:23813-23821 2023.
Cong Lu, Shengran Hu, and Jeff Clune. Intelligent go-explore: Standing the shoulders of giant foundation models. In The Thirteenth International Conference on Learning Representations, 2025a. URL StLLtttp::/ httpss////eeewweeeeeocoom. net/ forumm www.mmiiddappeeeeeee....
Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2505. 18719, 2025b.
Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K Qiu, and Yuqing Yang. Agent lightning: Train any ai agents with reinforcement learning, arXiv preprint ar0iv:2508. 03680, 2025.
11

---

Published as a conference paper at ICLR 2026
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. Proceedings of the 36th annual acm symposium user interface software and technology, pp. 1-22, 2023.
Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang. Pengjun Xie, Fei Huang, and Huajun Chen. Agent planning with world knowledge model. Ad- vances in Neural Information Processing Systems, 37:1 14111-14414444 114871, 2024.
Shuofei Qiao, Zhisong Qiu, Baochang Ren, Xiaobin Wang. Xiangyuan Ru, Ningyu Zhang, Xiang Chen, Yong Jiang, Pengjun Xie, Fei Huang, et al. Agentic knowledgeable self-awareness. arXiv preprint mrXiv:2504.03553. 2025.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL httthhtp::/ wttp/aixvvv.... org/ abs/ / 2412. 15115
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1007.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang. Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. / 19256, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances Neural Information Processing Systems, 36:8634-8652 2023.
Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. arXiv preprint 15X1:22009.1189, 2022.
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization of llm agents. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7584-7600, 2024.
Fahim Tajwar, Yiding Jiang, Abitha Thankaraj, Sumaita Sadia Rahman, J Zico Kolter, Jeff Schneider, and Ruslan Salakhutdinov. Training generally curious agent. arXiv preprint a7X14:2502.17543. 2025.
Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning large language models with embodied environments via reinforce- ment learning. In The Twelfth International Conference on Learning Representations, 2024. 2024 URL
hattps://openreet
net
Hao Tang, Darren Key, and Kevin Ellis. Worldcoder, a model-based Ilm agent: Building world models by writing code and interacting with the environment, Advances in Neural Information Processing Systems, 37:70148-702122 2024.
Ruoyao Wang, Peter Jansen, Mare-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? arXiv preprint arXiv:2203.07540, 2022.
Zihan Wang, Kangrui Wang, Qineng Wang. Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding eeol-tiol in Ilm agents via multi-turn reinforcement learning, arXiv preprint N0N0:25044.0073. 2025.
12

---

Published as a conference paper at ICLR 2026
Yufei Xiang, Yiqun Shen, Yeqin Zhang. and Nguyen Cam-Tu. Retrospex: Language agent meets of- fline reinforcement learning critic. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4650-4666, 2024.
Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! Ilm agent learning via iterative step-level process refinement. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp- 1556-1572, 2024.
Zhihe Yang, Xufang Luo, Zilong Wang, Dongqi Han, Zhiyuan He, Dongsheng Li, and Yunjian Xu. Do not let low-probability tokens over-dominate in rl for Ilms, arXiv preprint arXiv:2505. 12929, 2025.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 15:20744-207577 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Leurning Representations, 2023. URUhttpp::/opeereviiee wtw.ieewwnee/fooutttm.
id-WE_ L--WE_lluYUL>
Danyang Zhang, Lu Chen, Situo Zhang, Hongshen Xu, Zihan Zhao, and Kai Yu. Large language models are semi marniiparaetticc reinforcement learning agents. Advances in Neural Information Pro- cessing Systems, 36:78227-78239, 2023.
Zijing Zhang, Ziyang Chen, Mingxiao Li, Zhaopeng Tu, and Xiaolong Li. Rlvmr: Reinforcement learning with verifiable meta-reasoning rewards for robust long-horizon agents. arXiv preprint vrXiv:2507.2284. 2025.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403. 13372, 2024.
Siyu Zhou, Tianyi Zhou, Yijun Yang, Guodong Long, Deheng Ye, Jing Jiang, and Chengqi Zhang. Wall-e: World alignment by rule learning improves world model-based 11 agents. arXiv preprint arXiv:4410004884, 2024.
13

---

Published as a conference paper at ICLR 2026
A PSEUDO CODE
Algorithm [ presents the pseudocode of EMPO². Compared the original GRPO algorithm. EMPO² introduces several new components: memory buffer, and tip retrieval and addition, and two rollout modes and two update modes.
Algorithm EMPO?: Exploratory Memory-Augmentce On- and Off-Policy Optimization
1: Inputs: Initial policy Tea.. memory buffer M, task distribution p(l)), group size N, batch size B, max episode length T
2: for each training iteration do
{// Multi-step rollout)
4: Sample B tasks ~u p(()) and initialize N identical environments (total x N)
5: Sample Madiou ~ (Prompting Without Memory p, Memory-Augment Prompting 1 - p}
6: Initialize state 80
(6) + (1)) for o, N
7: for = 0 to T 1 do
do for 0 Memory-Augmmetedd Prompting then
if Mipoloos
lips
Retr(s ( :M)
11:
Sample
az (5) T8 (-
(6) [tipsp, ()
12:
else
13:
Sample dz
(+) m (. (0 (())
end
15:
Execute observe r, S1+1 end for
16:
17:
end for
18: for i to B N do
Sample tips mg (. 16) (6) lip-generation prompt)
20:
Append tips end for
M
21:
22:
{// Policy update}
23:
if Mhllon Memory- Prompting then
24:
Sample Magdaa [On-Policy Off-Policy 9} if Mupdare OfI-Policy then for i to B x N do
25:
26:
27:
log Te (a
[ipsp, (1) + log To (a 5/ (9)
end for 29: end if
28:
30:
end if
31:
Update policy using the loss function in Eq.
32:
end for
B PROMPTS
The following prompts were used in experiments. The ScienceWorld and WebShop prompts were used identically for both the online RI. baseline and EMPO with the WebShop prompt adapted from Feng al. (2025b). The content inside the curly brackets (() is dynamically filled based on the current progress at each episode step.
Tip Generation Prompt
Thanks for your playing. Now you have ended trajectory and collect some meaningless valuable information from the interactions with environment. Please summary trajectory, and sum- mary what information you get from this trajectory, and how far this trajectory is from fully completing the task. Please response with only one sentence with only one line, do not include any extra words. You sentence should be less than 100 words.
14

---

Published as a conference paper at ICLR 2026
Prompt for ScienceWorld
You have done few science experiments successfully and below are the action history of your experiments with similar tasks. Here 2 examples: (example (example 2) Follow the report of the two example tasks shown to you previously, try solve similar new task.
Task Description: deserreestiption
All your possible action formats are: stailaalo.actionndeseipppioo
you enter an unfamiliar room the time, you can use action 'look around' discover objects in it. Items in your inventory: [inventory]
Important! You only FOCUS actions these items: {focus items). You cannot FO- CUS on any other things. Please only use FOCUS required by the task description. Also, please FOCUS more directly, try not to focus on the container. You could try to explore different actions, especially when you are not sure what the best action for your current observation.
Prompt for WebShop
You are an expert autonomous agent operating in the WehShop e-commerce environment.
Your task is to: ..scripsciii
Prior to this step, you have already taken {step_count] step(s). Below the most recent leigth) observations and corresponding actions you took: /action.history) You are now at step current_step and your current observation is: {current {current.observatte observation).
Your admissible actions of the current situation [[available )
Now it's your turn to take one action for the current step.
You should first missible action best advances the shopping goal. This reasoning process MUST enclosed within <think>> </think> tags.
step-by-step about the current situation, then think carefully which ad-
Once you'l finished your reasoning, you should choose admissible action for current step present within <action>> </action>> tags.
C DETAILED EXPLANATION OF IMPORTANCE SAMPLING RATIOS IN POLICY UPDATES
To further clarify our policy update mechanism, this section details the calculation of the importance sampling ratio Pa. The specific calculation depends whether [tips were used during the rollout and update phases. This leads to three distinct scenarios, as summarized in Table The importance sampling ratio pe is defined as the ratio of the probability of action under the current policy to its probability under the old policy Te... used to correct for the distributional shift off-policy learning.
Table 3: Calculation of the importance sampling ratio P9 for different policy update modes. The which in practice is often calculated using log- probabilities.
ratio is computed as Po
*o(asl-)
Te (a.l-)>
Update Mode
Rollout Condition
Current Log Prob
Old Log Prob
Regular On-Policy
No tips
log (a log me(
St, u)
log Tew (a, u)
On-Policy w/ Tips
With tips
st, u, [ipsss
log mea (at St, u, tips tips
Off-Policy
With
ips
(rollout)
log mo(a;
8t,
log Tewa (ag
St,
15

---

Published as a conference paper at ICLR 2026
The three update modes shown in the table cover all scenarios. An update is considered on on-policy when the policy used to generate actions (Te..) and the policy being updated (Ta) are conditioned on the same information. This applies to the first two modes:
Regular On-Policy:
This is the standard on-policy update. The conditioning context for both the current and old policies identical (st, u), with no tips involved.
On-Policy w/ Tips: This mode is also on-policy because both policies are consistently conditioned on the provided tips (8t, u, ()
The Off-Policy update is the key mechanism through which the model learns from external guid- ance. In this scenario, actions are sampled from the old policy augmented with tip information: T (. 8t, u, [ipse). However, to "internalize" this guidance, the current log-probabilities for the new policy Te are recomputed without the tips, using only (o 8t, u). This mismatch in condi- tioning between the new and old policies makes the update off-policy and allows the base policy to absorb the knowledge contained in the tips.
While the importance sampling variant in Table theoretically unbiased, the distribution shift can still lead instability practice. Therefore, this allows for different implementation choices, such as tuning the clipping scheme or computing the old log-probabilities without tips, in order to better control the bias-variance trade-off.
D EXPERIMENTS DETAILS
D.1 RETROSPEX
In Retrospex (Xiang et al. 2024), the base models differ by environment: Flan-T5-Large (Chung ct at. 20224) is used for Science while Llama-3-BBIInstruct (Grattafiori et al. 2024) is used for WebShop. To ensure consistency in experiments, standardized the base model to Qwen2.5- 7B-Instucc. For this purpose, we utilized the offline trajectories provided by Retrospex and con- ducted SFT with I.L.MA-Fcctory (Zheng et al., 2024). For the IQL (Kostrikov et al. 2022) Q- function, we employed the model released by Retrospex. During SFT training, we tuned the hy- perparameters over learning rates 1.0 x 10-5 5.00 10 1.0 10 6 and epochs 3, 8, and adopted the configuration that yielded the best performance. Each run was conducted using two NVIDIA A100 GPUs with 80GB memory.
Moreover, in our Retrospex Science evaluation, we remove human designed heuristics to re- duce reliance on manual rules. Retrospex normally skips any "focus on" action unless repeated three times or explicitly mentioned in the task, and replaces step-by-step "go to" actions with direct "tele- port" moves. Removing these heuristics ensures the evaluation better reflects the agent's 's inherent capabilities.
D.2 ONLINE RI: SCIENCEWORLIS
We base our EMPO² implementation on the GRPO framework provided in verl (Sheng et al. 2024) while introducing the following key modifications:
Multi-step implementation: In the original GRPO implementation in verl, an LLM rollout termi- nates after generating single response given problem. We extend this to a multi-step setting, where the agent continues interacting with the environment until either maximum episode length is reached or the environment issues a termination signal. This modification allows the agent to perform sequential reasoning and adapt its responses across turns.
Memory buffer integration: To support EMPO2's memory-hased mechanism, we incorporate an explicit memory buffer. During multi-step rollouts, the agent can retrieve tips from memory and append newly generated tips The code snippet for this part as follows:
import numpy as ne, zequeste, uvicorn
Erom festapi import FastAPL
from pydantic import RaseMacal from typing import List, Optional
app PastAPT(
16

---

Published as a conference paper at ICLR 2026
ent, men_list, content_set (b, O)
clasa MenRequeat (BaseModel):
key: [fst([
idx: aptional[ings None
contant: Opticnal [erional(str [str]
acore: Optional [*loac] None
eapp-post ("/nam/")
asyne def men_handlezzreq MenRequest);
global mea_list, content_set. key, idx, content, score = reg.key, zeq-idx, req, concent, req.
if content -- "Reset"; Reset all buffers content_set [i: ()
in ringa
mem_liat = (1: for in (ick)] out [i: for in zange (idx) return ("status": "reset")
1f content is
memory
If content not in (entent__aet(ddd)) (ontent_settidx/ (dde add(content)
mem_lis:: append
("ent": ent lidx', "key": key, "content"; content, "score": score)) [idx]
if len (ime__isst (idx]) > 1000: $ Evict oldest concent_aet [idx, discaadd Imem_list [idx] ["content"]]
nen_list[idkl mem List [idx] 1-000:
return ("status"; "added", "cotal"; ent (idx]]
Retrieve by cosine similarity (threshold 0.5), return tep-10 by score rey_vec np.array(key)
candi dates = []
nen_liat(idk]:
aLvec ap.array (m] (m["key"])
sim (key m_vec) / (gg.Linalgg.nomm/eeyy.woo mg..inalg.nnammn
sim 0.5:
candida:es.appendde
data [x("content"] for sorted (candidates, key=lambda ("*soore"] [:10]]
return ("count": len (datal, "data"; data)
client wiilities --- compresa_ (text): return requests.post
jsee-("sext"; text (json ["key"
def retrieve_ (idx, key); requests. post ("tstp://127.0.0000 return =1 =["count"], =["data";
json- ("key" key, *idx"s idx)) json
def medonyy (idx, key, content, szore):
requasts. post ("http://127.0) 2.10 118801/mmm/",
Json=("key": key, "Idx": idx, "contens": content, "score"; acore))
Iraining-phaae nemozy zetrievs:
if phane in ["on-policy wewolllyy--tt--mmmmeee
"off-policy"1:
text "_" join (o* roler mp.array (compress_ (text)) (-1))
1 for conversational
count, es = reteieve aee__emoryyyooooo buffer_id, kay)
count, memories - 0, []
if nome ss main
run 1os:-70.0.0.0" poct-8001, workers=4)
Listing 1: Implementation of memory buffer integration.
Hyperparameters. All online RL algorithms (GRPO, EMPO²) the same hyperparameter configuration. The maximum response length set to 32 tokens per step and 4,500 tokens in total, and each episode is limited to 30 steps. The actor learning rate is set to 1 x 10-6 For GRPO, the group size is fixed at and the mini-batch size at 16. The KL-divergence loss coefficient is set to 0.0. In addition, the actor rollout parameters specified follows: the clipping upper bound is set to 0.30, the clipping lower bound to 0.20, and the clipping ratio coefficient to 10.0.
Computing Resources. All experiments were conducted using eight NVIDIA A A100 40GB GPUs.
17

---

Published as a conference paper at ICLR 2026
D.3 ONLINE RL: WEBSHOP
We base our EMPO² implementation on the GRPO framework provided in verl-agent (Fnng et al. 2025b), and the modifications for EMPO² the same those described in Appendix D.2
Hyperparameters. All online RL algorithms (GRPO, GiGPO, EMPO*) use the same hyperpa- rameter configuration, following Feng al. (025b). The maximum response length tokens, and each episode is limited to 15 steps. The actor learning rate is configured as 1 x 10 For GRPO, the group size is fixed 8. The rollout temperature
512
1.0, while the validation temperature is set to 0.4. The mini-batch size is 64, and the KL-divergence loss coefficient is 0.01. Finally, the discount factor y set to 0.95.
Computing Resources, GPUs (40GB each).
All online RL experiments were conducted using eight NVIDIA A100
E QUALITATIVE ANALYSIS ON TIPS
E.1 MORE EXAMPLES OF GENERATED TIPS
Following the example of the generated tips in Section 4.1 below more detailed examples of how the tips evolve as the task progresses.
WcienceWorld <power-component> task
You focused on red light bulh did not complete task turning on red light bulb. You are in the hallway and need to find a way; At that timestep, The specific action your took was focus on red light bulb. score 100.00
Trajectory from completion: connected battery red wire not in correct configuration; gained insights on available objects but missed key steps in circuit creation.; At that timestep, The specific action your took was connect battery cathode to red wire terminal 1. score 0.0
Connected green wire to green light bulb but did not find the battery complete the circuit. Trajectory incomplete as the task requires powering
green light bulb.; At that timestep, The specific action your took was connect battery to green wire terminal 1, connect green wire terminal 2 to green light bulb, connect red wire terminal to battery. score 7.0
Connected green wire to green light bulb, but not fully completed due missing battery connection.; that timestep, The specific action your took was connect green wire to green light bulb. score 13.0
Task incomplete, terminal connections incorrect; got red wire, battery, and red light bulb, but terminal connections not successfully made to power red light bulb.; that timestep, The specific action your took was connect battery cathode to red wire terminal score 27.0
I persisted in connecting the green wire the green light bulb but the eircuit was interrupted by other wires, affecting task completion.; that timestep. The specific action your took was connect green wire terminal to green light bulb, score 33.0
You moved to the workshop, connected the circuit, hut the green light bulb is still not powered due the lack of power source and proper connections.; At that timestep, The specific action your took was turn on green light bulb, score 83.0
The green light bulb was turned on. but the door to the workshop was closed repeatedly.; At that timestep. The specific action your took open door to workshop. score 83.0
18

---

Published as a conference paper at ICLR 2026
ScienceWorld <cceenissry..iiiiipie.eeeeeaaarr task
You moved to the greenhouse but did not find the necessary materials to create green paint, indicating the task camnot be completed with the current information and location.; At that timestep. The specific action your took was go to kitchen. score -10.00
Failed to execute the task due to unknown actions; collected information about paint colors and mixing process but unable to complete the task of creating green paint.; At that timestep, The specific action your took pour cup containing yellow paint in art studio in bowl, pour cup containing blue paint in art studio bowl, mix bowl. score 0.0
Failed to find a way to the kitchen or other useful areas: picked up orange but it's not suit- able for making green paint. At that timestep. The specific action your took open door kitchen. score 10.0
Failed to create green paint; mixed blue and yellow paint but action was ambiguous. Col- lected info room layout and objects but not sufficient to complete task.; At that timestep, The specific action your took was open door workshop.
30.0
The agent attempted to pour yellow paint into the bowl in the art studio but could not find the appropriate action, indicating need to directly interact with the paint containers; At that timestep, The specific action your took pour cup containing yellow paint in art studio in bowl and pour cup containing blue paint art studio bowl and mix bowl. score 40.0
The task is to create green paint by mixing blue and yellow paint, but repeatedly tried to pour blue paint into the howl without success.; At that timestep, The specific action your took was pour cup containing blue paint in art studio in bowl. score 40.0
Created green paint by mixing yellow and blue paint, but focus action was unnecessary as the paint was already in the bowl. At that timestep. The specific action your took was pour cup containing ycllow paint in art studio in bowl, score 80.0
E.2 EFFECTS OF TIPS ON EXPLORATION BEHAVIOR
This section provides qualitative analysis of how tips promote exploration.
World <power-component> task
[Without Memory]
[With Memory-Augmented Prompting]
Your task is to turn on the red light bulb. First, focus the red light bulb. Then, create an electrical circuit that powers it on.
Task:
Task: Your task is to turn on the red light bulb. First, focus the red light bulb. Then, ereate an electrical circuit that powers it on.
Observation: This room called the hallway. In it, you see:
Observation: This room is called the hallway. In it, you see:
the agent
picture
the agent a picture You also see:
You also
door to the workshop (that open)
door to the workshop (that is open)
Action: focus on red bulb
Here are some memories you collected in your previous exploration:
tips::
Task Failed
Focus on red light bulb but cannot find it in the hallway.; At that timestep, the specific action your took was focus on red light bulb; Eventually you got the score -100.0/100.
Action: go workshop
Observation: You move to the workshop.
19

---

Published as a conference paper at ICLR 2026
Science
cchemntttyymiippiiiiseeeaaaaroos task
[Without Memory]
[With Memory-Augmented Prompting]
Your task is to use chemistry to create green paint, When you are done, focus on the green paint.
Task:
Your task is to use chemistry to create green paint. When you are done, focus on the green paint.
Task:
Observation: This room called the hallway. In it, you see:
Observation: This room called the hallway. In it, you see:
the agent
the agent
picture
picture
You also
You also
door to the art studio (that is open)
door the art studio (that is open)
Action: pour cup containing yellow paint in
ops|: Here are some memories collected in your previous exploration:
hallway in bowl
Need find green pigment or mixture create green paint.; that timestep, the specific action your took was pour cup containing yellow paint in hallway in bowl; Eventually you got the score -100,0/100.
Task Failed
Need find a green pigment mixture to create green paint.; At that timestep. the specific action your took was open door to art studio; Eventually you got the score -100.0/100. Action: go studio Observation: move to art studio.
As shown the examples above, an agent without memory tends to repeat the same mistakes be- cause cannot incorporate feedback from previous failures into future attempts. In contrast, with memory-augmented prompting, the agent can refer to its past unsuccessful attempts, use them as guidance, and actively avoid repeating those errors. This enables the agent explore novel and more effective behaviors, ultimately expanding its search capabilities and boosting learning perfor- mance.
F MORE ABLATION STUDY
F.1 MODE SELECTION PROBABILITY
As discussed in Section 4.2 EMPO² employs memory-rollout probability during the rollout phase and an off- off-policy update probability during the update phase. We conduct comprehensive ablation studies to systematically investigate the effects of these hyperparameters, and q.
Ablation of echemistry-mixe www.iiiaint-ssoonduueee
Ablation of on <power--coppoentt>
swep jed
smpcl Lanautt Lankiy
jart
(a)
(b)
Figure 10: (a) EMPO2 leamingg curves with varying p. (b) with varying
Ablation p (Memory Rollout Probability): We evaluated
{0.1. (0.1. 0.4, the
chemister miiy-iix-aaintt- secondary-coles task. When p = 0.1. performance degrades significantly because EMPO2 effectively collapses to GRPO, confirming the importance of memory.
20

---

Published as a conference paper at ICLR 2026
Both 0.4 and 0.7 show faster initial learning due more aggressive knowledge internal- ization, although p = 0.7 exhibits some fluctuations in the later stages. Our choice of p = 0.25 provides stable convergence across diverse tasks.
Ablation on q (Ofr-Policy Update Probability): We tested q E {0.3, 0.5, 0.67, 0.85, 0.95) on the power-ccoponeet task. Extreme values (q = 0.3 q 0.95) underperform: very large overemphasizes distillation at the expense of training the memory policy, while small g slows knowledge internaiization. Notably, q = 0.85 achieves faster early exploration than our default y 2/3. This aligns with our expectations, as the default hyperparameters prioritize overall robustness rather than task-specific optimization. Therefore, it is natural that more optimal settings exist for particular tasks, highlighting the robustness of EMPO² within a reasonable hyperparameter range.
These results confirm that EMPO² performs effectively across a broad hyperparameter range. Our default settings represent well-balanced configuration that generalizes across multiple tasks with- out task- specific tuning, while the algorithm remains adaptable when further optimization desired.
F.2
ROLE OF INTRINSIC REWARD
Ablation of Intrinsic Rewards
EMPO ao sorsk toward
EMPO? ircandant Insimal acdlen..
-10
30
250
Figure 11: EMPO² learning curves with different intrinsic reward configurations on ScienceWorld task. We compare our full method against four variants: scaling the intrinsic reward coefficient by 0.5x and 2x, substituting it with a Random Network Distillation (RND) bonus, and its complete removal (w/o Intrinsic Reward).
To further investigate the role of the intrinsic reward in our proposed algorithm, EMPO?, we conduct an ablation study examine its impact. We compare full method against variants with different intrinsic reward coefficients (0.5x and 2x), a complete removal of the intrinsic reward, and its re- placement with standard exploration bonus based on Random Network Distillation (RND) (Burdal et al. 201 8a). For the RND baseline, we adopt the same hyperparameter configuration as in the original work. The results of these experiments are presented in Figure II
Altering the intrinsic reward's scale mainly affects the learning dynamics. A smaller coefficient (0.5x) leads to a smoother but slower convergence. whereas a larger one (2x) introduces minor stabilities. Notably, all variants using an intrinsic to a similar level of final performance. However, removing the intrinsic reward entirely causes learn- ing to plateau at lower level, suggesting its necessity in preventing the policy from collapsing into homogencous behaviors by encouraging sufficient exploration. Overall, these results indicate that EMPO² is robust to the specific mechanism and scale of the intrinsic reward, which primarily influ- ence the stability and speed of leamning rather than the final outcome.
the RND-based one-converge
21

---

Published as a conference paper at ICLR 2026
G ANALYSIS OF COMPUTATIONAL COST
G.1 COST ANALYSIS OF MEMORY-AUGMENTED ROLLOUTS
We analyzed the additional computational overhead introduced by the memory mechanism in EMPO2. During the rollout phase, this mechanism incurs extra costs related to tip generation, retrieval, conducted experiments using the
263.
Tima 63
2122
and storage. For the analysis, Qwen2.5-7BBIInsrruc model on A 40GB GPUs.
Action Comeration
T'p Geneattoo
Hemony Betreva
Homey Storapt
As reported in Figure 12 the memory mechanism adds approxi- mately 50.4 seconds per itcration, which accounts for about 19% of the total rollout time. Among these, tip generation and the sub- sequent storage tips in memory account for substantial por- tion of the cost. Therefore, while have verified that the mem- ory mechanism substantially aids exploration and significantly im- proves learning efficiency, it is more desirable internalize these benefits within the model parameters themselves rather than relying on the mechanism continu- ously-both enhance the model's inherent capabilities and to improve overall efficiency.
Figure 12: A breakdown of the each component spends dur- ing the rollout of each training step.
G.2 COST ANALYSIS OF TOTAL TRAINING TIME
Compared to GRPO, the training time of EMPO is primarily influ- enced by two factors:
ScienceWorld <ppoee--omponennt
The memory component: As discussed in the previous section, the memory component accounts for 19% of the total rollout time. Since eeemory-augmente
Ben.m
prompting is selected with probability (1- p 0.25) (as described in Section 4.2). this implies that, on average, 19% of the rollout time is incurred with a 25% probabil- ity.
Tintral
Figure 13: Time-performanee curves for EMPO2 and GRPO on ScienceWorld cower-compenent task.
The response length: In LLM-based RL training, rollout time constitutes major portion of the total cost. As the response length increases, the rollout itself becomes slower, and the time required for log prg-bbbbabiitty computation and actor updates in- creases accordingly. In our experiments, we found that the response length EMPO² is generally longer than that of GRPO. We attribute this the model spending more time reasoning and ploring when given the tips, which we believe enhances its exploration behavior and ultimately improves performance.
To ensure fair comparison with GRPO from training-timme time perspective, we plot the performance in Figure 13 using training time on the x-axis. The results show that, even under this perspective, EMPO² exhibits substantially higher efficiency than GRPO.
H THE USE OF LARGE LANGUAGE MODELS
We used a LLM polish the writing of the manuscript. The LLM was not employed in any aspect of research ideation, experimental design, or analysis.
22


Reviews:
[{"review_id": "EuJTnLWuIp", "reviewer": "Reviewer_JKue", "summary": "The paper introduces the EMPO2 algorithm, which works as follows\n- agents roll out trajectories\n- at the end they write \"tips\" for solving the environment\n- in future runs, some fraction of the time the agent conditions on hints form past rollouts\n- we train on both types of rollouts, including a 3rd variant where the tips are stripped out of the rollout.\n- they show improvements above prior works on ScienceWorld and WebShop.", "questions": "- The off-policy updates seem like they could introduce stability (as you mentioned). Did you consider either using a real off-policy algorithm or introducing an importance-sampling correction where the numerator is the prob with no tips and the denom is prob with tips?\n\n- The off-policy stabilization approach in Fig 6 is interesting. Would be nice to see error bars or multiple seeds there (to make sure it's not that one seed got unlucky in Fig 6).\n\n- A bit ambiguous to me what numbers are reported in Table 1 (and other results) for EMPO2 -- which rollout mode were used to create these? If the \"with tips\" one, how many rollouts were used to create the memory bank? Same question about baselines that use multiple episodes.\n\n- How were the hyperparameters chosen for sampling from on or off policy?\n\n- Why are there no error bars in table 2 for naive or reflexion?\n\n- Why are there no error bars in figure 8?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "Although each individual part of the proposed method is not original, combining them all together under one framework is an important contribution and has not been done before, particularly in the important but still early field of RL on LLMs.\n\nThe quality is good, mainly focusing on showing saturation of two in-distribution benchmarks. The paper also demonstrated signs of life on out of distribution benchmarks. The paper also sought to understand each component’s importance by doing ablations, as well as proposed future interesting extensions to the work, such as a similarity-based bonus for novel contributions to the memory bank.\n\nI appreciated the clarity of the communication, the plots, figures, and charts all are well-designed and get the most important points across.\n\nThe significance is important, since the standard of RL for LLMs (GRPO) mainly focuses on parametric updates for each training batch rather than encouraging exploration or learning over time across training or episodes.", "weaknesses": "I am confused about the hyperparameter choices for choosing to sample between memory and non memory for rollouts and on-policy and off-policy for updates. There isn’t explanation for these choices (½ and ⅓ respectively), and there are no ablations or sweeps (although 6.3 does ablate the entire components).\n\nSome of the plots could have better reporting. For example, figure 1 B not having error bars across the seeds, or figure 8.\n\nFor some of the baselines, I am concerned about the reported numbers being derived from other papers . For example, for WebShop, the Naive, Reflexion, GRPO, and GiGPO are taken from the paper. The paper states that all of the hyperparameters are the same for the training methods, which absolves most of my concerns, but RL methods are notorious for subtle implementation differences in the algorithm or environment which may not even be highlighted in the paper making big difference.\n\nI also think scaling this to more “non-toy” training and evaluations would improve the paper’s significance (although not necessary since this is proposing a new method).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E8hpPKtNHp", "reviewer": "Reviewer_1fWA", "summary": "This paper introduces a memory-augmented RL algorithm for learning effective LLM based policies. The key idea is to use a memory buffer in sampling (rollouts) and policy learning by using the memory to enable more effective exploration in the rollout phase (by being able to reason about past experiences) and using the memory to enable learning a more generalizable policy by performing a combination of on-policy and off-policy (where main policy is not conditioned on memory) learning. Results demonstrate significant improvement in ScienceWorld and moderate improvement in WebShop benchmarks.", "questions": "Please see questions listed in weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Paper is well-motivated and well-written. Justification for improved exploration in RL for LLMs is sound.\n- Use of memory in both rollout and update phase is simple yet novel in the context of RL for LLMs. \n- Strong results on ScienceWorld which demonstrate the OOD generalization of their method (due to generality of memory).", "weaknesses": "- Lack of ablations. The method introduces additional hyperparameters and components, the effects of which are largely undocumented.\n    - Effect of intrinsic reward component. What is the effect of this component on the performance of the final policy (paper only documents the effect on policy entropy)? How generalizable is this reward term? It seems as if it may require further reward-shaping (i.e. tuning similarity threshold) to generalize to newer domains where naive state similarity may lead to poor performing rollouts (e.g. see static noise TV example from Pathak et al).\n    - Effect of sampling proportion ($p$) between memory-free and memory-augmented rollouts. What is the effect of varying $p$?\n    - Effect of update proportion ($q$) between on- and off-policy updates. What is the effect of varying $q$?\n    - The authors set the KL coefficient to 0.0; does the final model lose its broad generality (e.g. on standard LLM benchmarks)?\n\n[1] Pathak, Deepak, et al. \"Curiosity-driven exploration by self-supervised prediction.\" International conference on machine learning. PMLR, 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IaWIS6n7iI", "reviewer": "Reviewer_c3bg", "summary": "Online reinforcement learning (RL) has recently emerged as a powerful method to improve reasoning and agentic capabilities of large language models (LLMs). However, these methods generally employ on-policy rollouts, and past failed attempts deliver no information other than single scaler reward. For hard tasks where models can be consistently wrong, on-policy samples do not recover any new information, and models may never learn how to solve these tasks via RL.\n\nA possible alternative to this approach is to incorporate memory into LLM agents — LLMs can read their past rollouts, figure out where they had gone wrong/what they could have done differently, and use them to collect future experience. This paper proposes EMPO$^2$, an end-to-end framework where LLMs generate memory in the form of hints from prior rollouts, incorporate them in future rollouts to collect better experience, and uses a mixture of on and off-policy optimization to both improve the model with these hints, and distill the behavior with hints into the model via off-policy learning to retain good behavior on these tasks even when hints are not present. Experiments on multi-turn agentic environments like WebShop and ScienceWorld show superior performance of the proposed method compared to pure on-policy online RL methods like GRPO.", "questions": "(**Question 1: Advantage Estimation**)\n\nBased on the definition of advantage in line 106-107, am I correct to understand that there is no per-turn advantage? It seems like the advantages are calculated using the entire sum of rewards in each trajectory.\n\n(**Question 2: Calculating Importance Sampling Ratio**)\n\nIt is not clear to me how the importance sampling ratios are calculated for off-policy updates. Based on the text and Algorithm 1 from Appendix A, it seems the old log probs ($log \\pi_{\\theta_{old}}$) are calculated using an off-policy manner (i.e., without generated hints). But how is the current log probs ($log \\pi_\\theta$) calculated for the off-policy updates? With or without the hints? More generally, could you elaborate how $log \\pi_\\theta$ is calculated for all different cases (regular on-policy updates, on-policy updates with hints, off-policy updates without hints)? Adding a table clarifying these cases to the main paper would help a lot regarding the clarity of the paper.\n\n(**Question 3: Figure 8**)\n\nI am a bit confused about this figure.\n\nWhy are the EMPO and GRPO plots split across two different panels and not on the plot? It is much harder to compare, at least at the first glance, what the performance difference is.\nWhy does the starting point between EMPO and GRPO vary? Is it because they already went through one round of training with their respective algorithm on the previous task?\nWhat happens if you take the checkpoint resulting from GRPO and run EMPO on top of it, and vice versa (run GRPO on the checkpoint from EMPO) on the new task?\n\n(**Question 4: ScienceWorld**)\n\nWhat does return/reward mean for ScienceWorld? Is there some other metric like task success rate/completion rate beyond just reward coming from different subgoals/components within a task? Could the authors report that?\n\n(**Question 5: Performance Difference between ScienceWorld and WebShop**)\n\nThe proposed method seems to outperform online GRPO quite heavily on ScienceWorld, but not so much on WebShop. Is there a reason/explanation for this?\n\n(**Question 6: Example of how the hints help**)\n\nCould the authors put rollouts with/without hints side-by-side in the appendix, to showcase an example of how the hints help generate better rollouts/avoid common or repeated mistakes?\n\n# References\n\n[1] Learning by Distilling Context, https://arxiv.org/abs/2209.15189\n\n[2] Training a Generally Curious Agent, https://arxiv.org/abs/2502.17543", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "**Overall, the paper is quite strong and I recommend acceptance of the paper.**\n\n## Novelty\n\nThe paper proposes a mechanism for self-generating memory, incorporating memory into the rollout mechanism in order to avoid past mistakes, promote exploration and achieve better rollouts. Moreover, to the best of my knowledge, this paper is the first to use off-policy learning to then distill back these **hint-augmented** prompts back into the model’s parametric knowledge. This is remarkable and also what I was looking for in the online RL system, Kudos to the authors for making it work so nicely.\n\nTo the best of my knowledge, this was only possible in an offline manner using SFT via context distillation [1] (this is an important work that should be cited and discussed however), but no one has done it using online RL before. \n\nThe hints are self-generated and do not use a stronger model, which makes the work more appealing.\n\n## Strength of results\n\nThe results provided on two benchmarks are very strong, showing remarkable improvement over regular on-policy online RL. This may also unlock improvement in cases where on-policy RL has failed, and have potential implications beyond what the paper presents, i.e., on reasoning tasks like math/coding.\n\n## Memory\n\nThe fact that models can generate their own memory and use it in future effectively, **despite not being an entirely novel idea**, is very nice to see in practice.", "weaknesses": "As mentioned above, I really like this paper. However, I would note the following weaknesses:\n\n## Comparison on single turn reasoning tasks\n\nThe idea of off-policy updates using previously generated hints can be useful beyond the tasks used in this paper. Particularly, this can help regarding single turn reasoning tasks like math/coding. \n\nThis is the single most important point where the paper's results can be improved. **If the authors can demonstrate the usefulness of their framework on these tasks, and respond to the other weaknesses/questions I mention below, I am very likely to increase my score on this paper further.**\n\n## Adding a comparison of performance vs GPU hours\n\nThe proposed method is inherently more computationally expensive compared to regular on-policy GRPO. **While Appendix E provides a rudimentary analysis of the breakdown of compute spent on various components of the proposed method, no comparison with GRPO is given.** The authors should include a plot with\n\n1. X-axis: GPU hours/flops/some other measure of compute\n\n2. Y-axis: performance\n\nIn the main paper, to make the comparison fairer with GRPO.\n\n## No ablation for intrinsic exploration reward\n\nThe paper uses an intrinsic exploration reward for novel states, in order to encourage the model to explore unseen/sufficiently novel states. However, I could not find an ablation of the proposed method showing how the performance differs in case this reward is not added/for different choices of the intrinsic exploration reward. To simply put it, it is unclear what the effect of this component of the proposed method is on the performance of the method.\n\n## Experiments using only one base model\n\nAll the experiments in the paper are done using only one base model. While the results are strong, it is unclear if the gains come from model specific pretraining/finetuning for Qwen2.5-7B-Instruct. Including results on models from different companies/different pre-training would make the paper significantly stronger.\n\n## No learning component for reward generation\n\n**This is more of an after-thought for future work instead of a serious weakness.** The proposed algorithm does not incentivize better memory generation. However, some proposed memory/hints can be better at steering future generations compared to others, and the model is never incentivized **directly** to generate better memory/hints (it may get incentivized indirectly via reward on memory-augmented rollouts). This needs to be addressed to make the learning better.\n\n(**Minor**)\n\nAn important prior work for section 3 discussing LLM agents to be information seeking is Paprika [2]. Similarly, context distillation [1] for learning to behave the same way without hints as whenever hints are available is an important prior work that should be cited and discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AcOBCynPvS", "reviewer": "Reviewer_r39R", "summary": "This paper introduces a reinforcement learning framework, EMPO², designed to enhance exploration and generalization in large language model (LLM) agents. Existing RL-based LLM agents often overfit to pretrained knowledge and fail to discover novel states. EMPO² addresses this by combining on-policy and off-policy updates with a self-generated memory mechanism—where the agent reflects on past failures to produce “tips” that guide future rollouts.\n\nIt internalizes useful behaviors into model parameters while maintaining adaptability via external memory through hybrid optimization. Evaluations on ScienceWorld and WebShop show substantial gains over GRPO with strong out-of-distribution adaptability requiring no parameter updates. The framework is proposed to be a step toward building self-improving, memory-aware LLM agents capable of efficient exploration and continual learning.", "questions": "- Q1: Justification of the hybrid update design: Could the authors elaborate on why combining on-policy and off-policy updates yields more stable or effective exploration in LLM agents? A theoretical or empirical rationale (e.g., ablation across different ratios of on/off-policy updates) would strengthen the methodological foundation.\n- Q2: Sensitivity to hyperparameters p and q: The rollout and update probabilities seem chosen heuristically (p=0.75, q=1/3). How sensitive is EMPO² to these settings? Have the authors explored how different sampling ratios affect training stability, exploration depth, or convergence?\n- Q3: Role and quality of generated “tips”: The memory mechanism is central to EMPO². Could the authors provide qualitative examples or a deeper analysis of what kinds of tips are most beneficial? Do these tips generalize semantically across tasks, or do they mainly encode task-specific heuristics?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Novelty in integration of memory and RL optimization: The paper introduces a hybrid framework (EMPO²) that unifies parametric (on-policy) and non-parametric (off-policy) learning, bridging memory-augmented reasoning and reinforcement learning. This combination is conceptually novel and addresses a long-standing gap between reflection-based and RL-based LLM agents.\n\n- Effective exploration mechanism: By incorporating self-generated reflective memory (“tips”), the method enables autonomous correction of past errors and promotes deeper exploration without additional supervision. It is a meaningful improvement over prior static-memory or fixed-parameter approaches.\n\n- The method demonstrates substantial gains on two challenging multi-step reasoning benchmarks, ScienceWorld and WebShop, with comparably significant improvement over GRPO. It also shows good out-of-distribution generalization with zero-shot adaptability.\n\n- The study includes comprehensive ablation analyses, comparisons across offline, online, and non-parametric baselines, and even computational cost breakdowns, supporting the robustness of the findings.\n\n- The paper is clearly written, with structured algorithmic explanations, detailed pseudocode, and implementation appendices.\n\n- Broader significance: EMPO² provides a promising direction toward self-improving, memory-aware, and generalizable LLM agents, with potential applications in embodied AI, web interaction, and general decision-making systems.", "weaknesses": "- The novelty is limited: The core contribution of EMPO² lies in combining existing components, such as memory reflection, on/off-policy RL, and intrinsic rewards, rather than introducing a fundamentally new algorithmic principle or theoretical insight. The innovation is primarily architectural rather than conceptual, which makes its novelty kind of limited. \n\n- The paper does not provide a formal or empirical analysis explaining why the hybrid on/off-policy mechanism stabilizes exploration or improves generalization. Key hyperparameters such as the rollout and update probabilities (p,q) are heuristic, with no clear sensitivity or convergence analysis.\n\n- Evaluation is confined to two benchmarks (ScienceWorld and WebShop), both text-based and reasoning-oriented. The framework’s effectiveness in broader or more complex environments—such as robotics, code synthesis, or multimodal RL—remains untested.\n\n- The study does not analyze the semantic quality of generated “tips” or demonstrate how they concretely guide exploration. Without such analysis, it is unclear whether the model truly learns generalized reasoning strategies or simply memorizes patterns.\n\n- Naming of the Method: the name of the method is kind of confusing at the first time. The \"square\" symbol is like the footnote, which is ambiguous. Since it is an acronym, it is better to show the full name at the first time of the appearance (e.g. in the abstract, you should show the full name at the first time it appears).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)