Meta-RL Induces Exploration in Language Agents
Authors: Yulun Jiang
Liangze Jiang
Damien Teney
Michael Moor
Maria Brbic
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Agent
Reinforcement Learning
Meta Learning
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement learning (RL) has enabled the training of Large Language Model (LLM) agents to interact with the environment and to solve multi-turn longhorizon tasks. However, the RL-trained agents often struggle in tasks that require active exploration and fail to efficiently adapt from trial-and-error experiences. In this paper, we present LaMer, a general Meta-RL framework that enables LLM agents to actively explore and learn from the environment feedback at test time. LaMer consists of two key components: (i) a cross-episode training framework to encourage exploration and long term rewards optimization; and (ii) in-context policy adaptation via reflection, allowing the agent to adapt their policy from task feedback signal without gradient update. Experiments across diverse environments show that LaMer significantly improves performance over RL baselines, with 11\%, 14\%, and 19\%  performance gains on Sokoban, MineSweeper and Webshop, respectively. Moreover, LaMer also demonstrates better generalization to more challenging or previously unseen tasks compared to the RL-trained agents. Overall, our results demonstrate that meta-reinforcement learning provides a principled approach to induce exploration in language agents, enabling more robust adaptation to novel environments through learned exploration strategies.

Published as a conference paper at ICLR 2026
META-RL INDUCES EXPLORATION IN LANGUAGE AGENTS
Yulun Jiang 12+
Liangze Jiang
Damien Teneys Michael Moor
Maria Brbicl.
EPPL. 2ETH Zurich
Idiap Research Institute Equal Advising
Equal Contribution
ABSTRACT
Reinforcement learning (RL) has enabled the training of large language model (LLM) agents to interact with the environment and to solve multi-turn long- horizon tasks. However, the RL-trained agents often struggle tasks that require active exploration and fail efficiently adapt from trial-and-error experiences. In this paper, we present LAMER, a general Meta-RL framework that enables LLM agents actively explore and learn from the environment feedback test time. LAMER consists of two key components: (i) a cross-episode training framework encourage exploration and long-term rewards optimization; and (ii) in-context policy adaptation via reflection, allowing the agent to adapt their policy from task feedback signal without gradient update. Experiments across diverse environ- ments show that LAMER significantly improves performance over RL baselines, with 11%, 14%, and 19% performance gains on Sokoban, MineSweeper and Web- shop, respectively. Moreover, LAMER also demonstrates better generalization more challenging or previously unseen tasks compared to the Rl.-trained agents. Overall, results demonstrate that Meta-RL provides a principled approach induce exploration in language agents, enabling more robust adaptation to novel environments through learned exploration strategies.
INTRODUCTION
Recent advances in large language models (L.LMs) have shifted from building conversational sys- tems to decision Dacision-making agents capable of reasoning about and interacting with their environ- ments (Yao et al., 2023b Shinn et al. 2023 Wang et al. 2025 Feng et al.) (025). To accomplish the goal, language agents operate in multi-turn, textual observation-actionn loops, and must adapt quickly using the memory across turns. Central to such adaptation is exploration, which allows agents to test uncertain actions, acquire new knowledge, and avoid premature convergence strategies. However, unlike humans that can explore systematically and make fast adaptation in new environments (Wilson et al.,, 20144, LIM agents do not robustly engage in exploration without substantial interventions (Krishnamurthy et al. 2024).
suboptimal
Recent works have begun address this limitation by guiding LLMs toward exploratory behaviors at test time. For example, Tajwar et al.] (2025) train models offline to distill exploration strategies from trajectories across diverse environments, While et al. (0024) induce such strategies from offline search traces. /Setur et al. (2025) train models to learn to explore in-context as a better way of spending test-time compute (Snell et al., 2025). However, these works either focus on single-turn non-agentic reasoning problems, or rely on offline data which limits them to imitation rather than active exploration.
In this work, we take step toward agents that can actively explore their environment, gather feed- back, and leverage this experience for more effective exploitation. Since multi-turn tasks often have sparse success signal after an episode, where an episode is the unit of exploration and exploitation. Balancing exploration and exploita- tion can then be naturally formulated as cross-episode reinforcement learning (RL) framework.
consider multi-episode regime (Shinn al. 2023)
Training across many similar but different environments under this framework leads to the meta re- inforcement learning (Meta-RI.) (Duan al. 2016 Wang et al. 2016, Bauer et al. 2023 Beck et al.

---

Published as a conference paper at ICLR 2026
Trajectories
Meta-RL Trajectorics
Meta-RL
70-
0.3
03
Cumulative Probabiity
Success Rate (%)
RL 60-
Meta-RL Rita-RL maining
75% Cumdlatove Probabiity
Probability
0.2
Training
more)
30-
0.2
0.1
Mode
0.0
Trajectory Diversity
Trajectory Index
Trajectory Index
Figure 1: Comparison of RL and Meta-RL training on the MineSweeper environment. Left: Meta- RL training with LAMER retains higher sample diversity from the base model while achieving better success rates, reaching a better trade-off between exploration and exploitation. Right: Distinct trajectories and their empirical probabilities aggregated over multiple sampled trajectories in the Mine WineSweeper environment. Each trajectory corresponds to a sequence of clicks (numbered cell) on the board. Sample diversity is quantified by the entropy of the empirical distribution. The Meta-RL trained model produces more diverse and explorative trajectories.
2025 where the agent is forced to discover general strategies that work in unseen and potentially harder environments.
Building upon this, propose LAMER (LLM Agent with Meta-RL), general Meta-RL frame- work for LLM agent training. LAMER contains two key design principles. First, unlike standard single-episode RL, LAMER is designed around multi-episode structure to train the agent solve the problem through trial and error. In early episodes, the agent is encouraged to gather diverse ex- periences and informative feedback from the environment, which are then used to adapt its policy in later episodes. By maximizing long-term rewards across episodes. the agent internalizes a learning algorithm that explicitly incentivizes exploration for improved downstream exploitation. Second, at both training and test time, the agent effectively leverages the feedback and reflections from previous episodes determine its strategy for the next episode. This essentially implements an RL algorithm in context, making the approach naturally suited for LLM agents. Meta-RL produces more diverse samples while simultaneously achieving higher performance, reaching a better balance between ex- ploration and exploitation than standard RL (Figure D To the best of our knowledge, this is the first time a meta-RL RL framework is used for LLM agent training.
We evaluate LAMER four challenging long- tasks: Sokoban (Racanière al. 2017 MineSweeper (Li et al., 2024). Webshop (Yao et al.] (2022) and ALFWorld (Shridhar ct al. 2021 Using Qwen3-4B Yang et al. 2025).
demonstrate that LAMER consistently outperforms prompting and RL. baselines across all environments. In addition, we observe that the trained model learns to balance between exploration and exploitation, resulting in improved test-time scaling per- formance. In particular, LAMER adapts the trained policy at test time, with 11/14/19% absolute performance gains on Sokoban, MineSweeper and Webshop, respectively, over the RL. Further- more, we show that the LAMER trained model generalizes better to harder and out-of-distribun tasks. In summary, LAMER takes step toward autonomous agents that can actively act to uncover information and improve their decision-making in the new environments.
2 RELATED WORK
LLM ALLM-ss-ggen aseat. As LLMs become increasingly capable of reasoning about complex scenarios Wei
al.) 2022). ) there is a growing interest making them decision -aaiiig autonomous agents. Ear-
lier works rely on prompting frozen LLMs (Yao et al., 20236 Shinn et al.] [023] Park et al. 2023
Wang et al. 2024a| AutoGPP). ReAct (Yao et al. (023b) prompts L.LMs with in-context examples to generate both textual actions and reasoning thoughts. Later, Reflexion (Shinn et al. (023) extends
this principle to the multi-episode setting, where the agent verbally reflects on the last episode and maintains their own reflection buffer for the next episodes. More recent research trains LLM agents
through designing advanced RL algorithms (Wang et al. 2025 Feng et al.) 20225) for multi-tun turn inter- actions, supervised fine-tunine on generated interaction trajectories across diverse tasks (Tjwar jet al., 2025). The evaluation of LLM agents also poses challenges because of fully verbal interac- tions with the environments, Recent benchmarks span wide range of domains, including text-based
2

---

Published as a conference paper at ICLR 2026
embodied environments (Shridhar et al. 2021), e-commerce website (ao et al. 2022), , bandits (Nie
iet al., 2024). classic games (Park et al. 2025 Li et al. 2024) and other tasks (Liu et al., 2024]
Nathami al. 2025 For more comprehensive overview of these efforts, we refer readers recent
surveys (Wang et al. 2024b ZZhan et al. 2025).
Meta reinforcement learning (Meta-RL) (Beck et al. 2025 focuses "learning reinforcement
learn in order to rapidly adapt to new environments. Similar to meta-learning (Thrun & Pratt 1998)
Hospedales et al. (202I) involves an inner-loop that represents RL algorithm (i.e, an adaptation strategy) by itself, together with an outer-loop that updates the meta--arameters so that the inner loop becomes more effective across many tasks. By training across many tasks, the outer loop forces the agent to learn the exploration strategy necessary to solve the tasks, while the inner loop enables the agent adapt quickly based on the exploration. Depending on how the inner loop done, there are in- context methods and gradient-basee
methods. For example, (uuan et al. (2016); Wang et al.] (2016); Stadie al. (2018) represent the inner-loop as a history-dependent the adaptation is thus done 'in-context" through gathering more information stored in the memory states. On the other hand, Finn et al. (2017) leverages a gradient- based approach in which the inner adapts general meta- meta-policy learned by the outer-loop. Our work lies in the former category, where the adaptation occurs entirely in-context at test time, naturally leveraging LLMs' in-context learning abilities.
policy parametrized by an RNN,
Test-time compute. A Meta-RL. framework in LAMER can be considered as amortizing the test- time compute by training tasks in multi-episode manner rather than single episode. In this way, the learned in-context policy adaptation balances exploration and exploitation for a fast adaptation test time. This essentially better way of spending test-time compute (Snell al. 2025
Muennighoff et al. [2255] Wu et al. [2025] Setlur et al. (025)). In our experiments, we match the training compute budget between an RL and a Meta-RL baseline, and show that meta-RL encourages superior test-time scaling behavior (through @ass@k). [Qu et al. (2025) similarly relates meta-RL to
test-time compute, but they are limited to single-turn problems of mathematical reasoning, without leveraging the interactive feedback from the environment.
Reasoning in LLMs. More broadly, this work relates reasoning in LLMs, because language agents must use reasoning as part of their decision-making. A large bulk of recent work on LLM reasoning has focused on more advanced prompting (Wei et al.] 2022 Yao et al., 2023a). post-
training (Cobbe et al.] 2021; Luong et al. [024] Shao al.) 2024 DeepSeek-AL et al. 2025)
or bootstrapping (Zelikman et al., (2022) against verifiers or reward models, inducing structured search behavior (Gandhi et al. 2024] Moon et al.
2024), or reflecting on previous answers (Kumar
let al., 2024 Xiong et al.] | Qu et al. 2024), etc. Most of these works focus on single-turn
math (Hendrycks et al. 200 Cobbe al. 202T) and coding (Chen et al. [02T]] [Heddrycks al.,
202 problems, while we target multi-turn agentic environments where environment feedback is available after every action and the end of the episode.
3 PRELIMINARIES
We consider the scenario where an LLM agent interacts with the environment solve a multi-turn task. This process can be formulated as a Markov decision process M = (S, AA P, R, Tssp), where and denote the state space and action space, and R is the reward function. At each time step t O...., T 1, the LLM agent observes state E S and selects an action a, E A according to its policy d, ~ (-(=))). The environment then provides a scalar reward Tr E R and transitions to the next state $141 according to the transition function P(- St, az). A trajectory is the sequence of states, actions, and rewards over an episode, i.e., т = (so. ao, To: ST ( The objective of reinforcement learning is to maximize the expected discounted return:
E E
Vttep t==
where "sep E [0, 1] is the discount factor. Recent works (Wang et al. 2025 Feng et al. 2025) have shown that RL training has enabled LLM agents to interact with the environment and solve multi- turn tasks. However, such agents often learn a fixed policy during training and struggle to actively explore and adapt their behavior to the tasks at test time (Nie al. 2024
3

---

Published as a conference paper at ICLR 2026
Meta-RL. RL. Conversely, by training distribution of tasks, Meta-RL (uan al. 2016 Wang
et al. 2016 Bauer et al. 2023; Beck et al. 2025) encourages exploration because it optimizes meta- parameters, such that the agent can solve tasks quickly. In case, the meta- meta-paraman parameters the parameters of the I.L.. This necessitates the agent to learn general exploration-exploiattio strategies suitable for the task distribution trained on. For example, for most navigation tasks in partially observable env environments, the optimal strategy is to gather the environment information and locate the target during the first episode, then reach the target as efficiently as possible in the second episode. This explore-then-iee the--eepooo-oo strategy implemented by the agent is itself an RL algorithm, where the policy learned the meta-level encodes how to adaptively switch between information- gathering and For LLM agents operating in multi-turn tasks, the policy can be context (i.e., without parameter updates at test time), naturally leveraging the in-context capability of L.MSs.
behaviors depending on the stage of interaction with new task.
LAMER: A META-RL FRAMEWORK FOR LLM AGENTS
Adopting the Meta-RL principle, we present LAMER, a framework for training LLM agents with the ability to actively explore and adaptively learn from the environment. The framework addresses two central challenges: (i) how to balance exploration and exploitation over multiple attempts at task, and (ii) how to efficiently adapt the policy during training and evaluation. To address the first challenge, LAMER introduces cross-episode training scheme that treats each trial as a sequence of episodes, enabling the agent to explore in early episodes and exploit this information in later ones. Second, instead relying on gradient-based updates, LAMER uses self-reflection as an in- context adaptation mechanism, allowing the agent to summarize past experiences and adjust its strategy accordingly. Together, these two components enable scalable training LLM agents under unified Meta-RL framework, which be optimized with standard RL algorithms.
Cross-episode training framework. In the training of LAMER, each trial consists of N episodes sequentially generated by the agent:
T= (7)) (()
(N-1)), where (()) T& (n) (-), E [0, - 1],
where Th (-) is the policy episode n updated from the accumulated history 7(0) (1-1)
through some adaptation strategy. For simplicity, in our analysis we assume all episodes contain the
(n) (n) (n) s(() (n) -1 (((m) T steps of interactions with the environment, i.e., (n) (5!m) ag
for all n E N - 1]. The rollout process terminates at n ((n) is successful (as indicated by the environment feedback). Otherwise, the agent starts a new episode ((++)) from the same initial state, repeating this procedure until the maximum episode budget is reached. For action a( the discounted return ge within the episode + (n) E ET is:
T-1
g
Σ
(3)
1=!
where Ssep [0, is within-the-episode discount factor.
To enhance the exploration and maximize the long-term reward, in LAMER framework we define the discounted return G(") across the episodes of T as:
N
(
m-- (m)
Yaj
90
within-the-opisode
m==-11
episode
where "uaj [0. 1] is the cross-episode discount factor. Finally, the LLM agent is trained via the following Meta-RI. objective:
N-
T-1
((0)= = ET
Σ
n Tiraj
Istep"
(m)
= ET (600
(5)
t=0
4

---

Published as a conference paper at ICLR 2026
Here, Yua is important factor for the trade-off between exploration and exploitation. Ideally, small Yuaj biases the objective towards early episodes and will lead to rapid exploitation to solve the problem. In comparison, larger Yinj emphasizes long- long-horizon return and therefore encourages more exploration at the early stage.
RL Training:
(0)
(D) T--1
(3)
(3)
%
(K-1)
(R-1)
Agent
Episode (0)
Independent
Episode 7 (1)
Independent
Episode (N-1)
Meta-RL Training:
(0)
(3) -0
(3)
(3) TT-1
(0-1)
XX-33
(N-13
ST-1
Agent
Episade
Reflection
Episode y (1)
Reflection
Episode 7 (N-1)
Q
Yiraj
Ferai
Figure 2: Comparison between the training processes RL (top) and Meta-RI. used in LAMER (bottom). For single task, RL generates group of trajectories independently. In contrast, in LAMER we use Meta-RL and produce the trajectories sequentially and adapt the policy in-context with self-reflection. Trajectory discount factor Tiraj used for cross-episode credit assignment.
In-context policy adaptation with self-reflection. In the Meta-RL, policy adaptation is the inner loop of the learning process of an LLM agent. Therefore, a flexible and efficient adaptation mech- anism plays important role during training and methods like gradient descent (Finn et al.) 2017)
might be too expensive, especially for LLMs. In LAMER, we propose a self-reflection based strat- eSy(Shinn et al., 2023) adapt the policy in-context (Brown al., 2020) Laskin al. 2023).
Specifically, after each episode finishes, we prompt the agent to generate the textual reflection on the previous attempt, providing specific feedback and plan to guide the next episode (see Ap- pendix A for the used prompt). The policy is therefore updated through modifying the context,
To (1)) (+) (-)) where H()) denotes the inter-episode memory that contains both the history trajectories and reflections. Importantly, the self-reflection step is also explicitly trained in LAMER using the reward obtained in the next episode. Note that the content H(x) can be adjusted according the predefined memory buffer reduce the context length and improve the efficiency. By default,
retain both history and reflection in H(m), and provide an ablation study in Section 6.2
Comparison to the RL training. Compared to the RL objective (Eq. D. Meta-RL extends the credit assignment across multiple episodes to incentivize exploration in the early stages. In practice, given a single task, both RL and Meta-RL will sample a group of episodes during training to estimate the advantage. The key difference is that the RL rollouts are independent, whereas in Meta-RL each episode is conditioned on the preceding rollouts within the trial. Figure illlustraees the conceptual difference between the training processes of RL and Meta-RL.
Optimization.
The proposed Meta-RI. objective in Eq. (5) can be optimized with standard policy gradient methods. Given the per-action cross-episode return ((²) defined above, the gradient can be estimated by
N-IT-I ΣΣ
( = ET
V log To(a
(m) 10(0
(n) H(a) (m)
(6)
R=0 t=
where A(n)) is the advantage estimation derived from G (n)
The framework is compatible with and critic-free approaches such as
widely used optimizers such PPO (Schulman et al. 2017)
GRPO (Shao al. 2024) and GiGPO (Feng al. 2025
5

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
In this section, we conduct comprehensive experiments to evaluate LAMER in the Meta-RL frame- work. Specifically, we present the evaluation on: (i) the overall performance of LAMER across different agent environments; (ii) the generalization ability of LAMER to harder tasks; and(iii) the generalization of LAMER under distribution shifts.
5.1 EXPERIMENTAL SETUP
Environments. We evaluate LAMER four challenging and diverse environments: Sokoban (Racanière et al., 2017). MineSweeper (Li et al. 2024), Webshop (Yao et al. (202)) and ALFWorld (Shridhar et al. 202T). Among them, Sokoban a classic grid-based game fully observable. In comparison, the environments of
on planning where the environment MineSweeper, ALFWorld and Webshop are partially observable, requiring the agent explore and plan under uncertainty to finish the task. Specifically, MineSweeper is board game about logical deduction on hidden cells. Webshop simulates realistic web-based shopping tasks, and ALFWorld provides text-based embodied environments. We provide the detailed explanation and prompts of the environment Appendix A All the experiments are conducted with the text modality, though the proposed method can be naturally applied to multimodal environments.
Training details. We use Qwen3-4B (Yang et al. 2025 as our base model for all the experiments. To improve rollout efficiency in agentic loops, eration. Additionally, we validate our method on Llama3. Appendix for the results. For the Meta-RL setting, we use "inj 0.6 as the default trajectory discount factor and explore its influence in the ablation study. We use GiGPO as the default opti- mization algorithm for all the environments with LAMER. Importantly. for Meta-RL RL training we sample N = 3 episodes and set group size to 8 for each task. To ensure fair comparison, we use a group size of in standard RI. training, yielding the same number trajectories used for each gradient update step. All other hyperparameters and configurations are kept identical across RL and Meta-RL for a fair comparison and are provided in Appendix C Our code is publicly available at nttps://githar
the non-thinking mode during trajectory gen-
(Grattafiori et al. 2024), see
cetom//mbiio.m -pt.oo-ppf//ii LaMer
5.2 PERFORMANCE COMPARISON
We compare the performance of the LAMER framework with grompting-based baselines (Zero- shot, ReAct Yao al. 2023b Shinn al. 2023 ). and RL methods (PPO (Schulman al. 2017 RLOO (Ahmadian et al. 2024), GRPO (hao et al. 2024), and GiGPO (Feng et al.) (025) across three environments: Sokoban, MineSweeper, and Webshop. For each method, we report the success rates under 1, 2, and 3 attempts (i.e., pass 1, pass @ 2, and @ respectively). The results are summarized in T
Table 1: Performance on Sokoban, MineSweeper and Webshop environments. The results of p@ @ 2 and @ denote the success rate (%) under 1, 2, and attempts, respectively.
Sokoban
MineSweeper
Webshop
Method
@@1
p@2
p@3
p@1
p@2
p@3
p@ 1
p@ 2
p@ 3
Prompting
Zero-shot
6.8
9.8
12.9
4.5
6.6
8.6
1.4
2.1
2.3
ReAct
7.2
9.6
12.5
6.3
7.0
10.9
3.1
4.5
4.5
Reflexion
6.4
9.8
12.1
5.5
7.2
9.8
2.7
3.3
3.5
Training with RL
PPO
12.5
15.4
16.8
29.7
34.2
35.5
53.1
54.5
54.9
RLOO
13.5
16.6
18.8
48.8
51.2
51.6
67.6
68.4
69.1
GRPO
22.9
26.4
27.0
36.3
40.0
40.4
72.9
73.0
73.0
GiGPO 41.6
43.6
44.1
52.0
54.9
55.1
73.4
74.6
75.2
Training with Meta-RLe (ours)
LAMER
42.4
52.0
55.9
44.1
66.4
74.4
67.8
84.4
89.1

---

Published as a conference paper at ICLR 2026
Meta-RL obtains better performance. Across all three environments, LAMER trained with Meta-RL consistently outperforms both prompting-bait based baselines and RL-training methods on the final pass @3 success rate. On the Sokoban environment, LAMER achieves 55.9% pass @3 success rate, substantially outperforming the 44.1% from the strongest RI. baseline (GiGPO) and 12.9% from prompting-based methods. Similarly, 74.4% pass@3 success rate, which is 19% higher than the best RL-trained model. On the Web- shop environment, LAMER also performs 14% better than the RL-trained methods. Notably, the performance gains are not limited to pass@ improvements are also observed on pass@ for all the environments, and even pass@ for Sokoban. Together, these results demonstrate that LAMER delivers consistent benefit on the trained agents to solve the long-horizon task in the complex envi- ronments.
the MineSweeper environment LAMER reaches
Meta-RL exhibits stronger test-time scaling. LAMER trained with Meta-RL demonstrates re- markable effectiveness test-time scaling, with larger performance gains across attempts according to the results in Table [ For example, from pass @ to pass @3 on Sokoban L. LAMER achieves 13.5% improvement, significantly larger than both RL-trained and prompting- less than 5%). Notably, although LAMER starts with slightly lower pass@) I performance than the RL baseline (GiGPO) in the MineSweeper and Webshop environments, it quickly recovers and sur- passes all baselines by pass @2 and @. The results indicate that the Meta-RL trained model has successfully learned to actively explore in the earlier episodes and adapt effectively from the mistakes, leading significant gains in the subsequent attempts. The illustrative trajectories and reflections produced by the trained agents are presented in Appendix
baselines (which are
Meta-RL induces exploration. To further analyze the behavior of the models, measure the diversity of answer trajectories across environments. For each question, we sample multiple trajec- tories from the agent and group the identical trajectories that have the same states and actions. These groups are used to form the empirical distribution over distinct trajectories, as shown in Figure [ We then estimate the entropy the distribution quantify the trajectory diversity. Figure 3 compares the trajectory diversity of the base model, RL, and Meta-RL agents across environments. We observe that the base model exhibits the highest entropy, indicating it generates wide range of trajectories, though this diversity does not translate into higher success rates (see Table TT. RL-trained agent reduces diversity and converges toward more deterministic behaviors. contrast, LAMER trained with MI preserves higher level of diversity than RI. baselines, allowing more exploration at test time.
Sokoban
MineSweeper
Webshop
Trojectory Diversty
Trojectory Diverstty
Diversity
Model
Meta
Base
RL
Medel
Neta
Figure 3: Trajectory diversity of base and trained models. Compared to RL, Meta-RL preserves more diverse trajectories from the base model, striking a better balance between exploration and exploitation.
5.3 GENERALIZATION TO HARDER TASKS
Next we study the generalization ability of the pretrained models on harder tasks. To this end, we take the models trained with the RL. and Meta-RL and evaluate them on the harder tasks in the environments of Sokoban and MineSweeper. We increase the difficulty by using more boxes for Sokoban and more mines for MineSweeper in the grid. The results are shown in Figure 4 As expected, the model trained with both RL and Meta-RL underperforms on harder tasks with an increasing number of boxes mines in the grid. However, Meta-RL consistently outperforms RL on all the difficulty levels. Notably, on the most difficult setting, the model trained from Meta-RL still outperforms the RI-trained model with 10% performance gap on Sokoban, and 5% performance
7

---

Published as a conference paper at ICLR 2026
Sokoban
MineSweeper
Success Rate (8%)
50
RL
Success Rate (96)
Meta-RL
60
Meta-RL
40
30
40
Trained at
Trained
Number Boxes
Number of Mines
Figure 4: Performance of RL. and Meta-RL. trained model on the tasks with increased difficulty. For Sokoban, we gradually increase the number of boxes and for MineSweeper, we increase the number of mines the grid.
gap on the MineSweeper. The consistent gap indicates that LAMER trained with Meta-RL, not only performs better the training distribution, but also generalizes better to the harder tasks.
5.4 GENERALIZATION TO UNSEEN TASKS
We further study the ability LAMER and alternative methods to generalize out-of-distribution For this experiment, we use the ALFWorld environment (Shridhar et al., (021). As text-based em- bodied environment, ALFWorld contains categories of common household activities: Pick and Place (Pick), Examine in Light (Look), Clean and Place (Clean), Heat and Place (Heaf), Cool and Place (Cool), and Pick Two and Place (Pick2). We use the tasks of Pick, Look, Clean and Hear as in-distribution tasks and use Cool and Pick2 as out-of-distribution tasks. We train LAMER and alternative baselines with instances from in-distribution tasks and then evaluate the model on both in-distribution tasks (with held-out test set), and the examples of disttiibiiiiitoonnnnn tasks. The results are shown in Table 2 As we can see, RL trained model generally performs well on in- distribution tasks and outperforms prompting-based methods by achieving more than 20% improve- ment on Look, Clean and Heat. However, on out-of-distribution tasks, Cool and Pick2, only obtains 58.15 % and 36.0% success rate. LAMER with Meta-RL RL consistently outperforms RL on both in-distribution and out-of-distribution tasks, with notable performance gap on out-of- out-of-distribution tasks. particular, our LAMER framework achieves 23% performance gains on Cool and around 14% on Pick2. Overall, these results suggest that on ALFWorld, Meta-RL trained model could generalize better to out-of-distribution tasks compared to the RL trained model.
Table 2: Evaluation of out-of-distribution generalization on the tasks of ALFWorld.
i.d
0.0.d
Method
Pick
Look
Clean
Heat
Cool
Pick2
Prompting
91.9
52.9
48.4
44.8
42.8
21.2
RL
95.5
83.0
67.9
86.6
58.1
36.0
Meta-RL
97.7
100.0
90.2
89.5
81.0
50.2
ANALYSIS
We further conduct a series of ablation studies on the key design factors of LAMER, including (i) the influence of trajectory discounted factor "ina on the tradc-off between exploration and exploita- tion and (ii) the ablation inter-episode memory configurations. We additionally discuss (iii) the computation budget of the proposed Meta-RL framework compared to the RL training.
6.1 INFLUENCE OF TRAJECTORY DISCOUNT FACTOR
The cross-episodes discount factor "iay controls how rewards are propagated within a trial, thereby mediating the balance between exploration and exploitation in the LAMER framework during train- ing. To understand the effect of the discount factor, we train the agents with LAMER using different values of Tinj on Sokoban, MineSweeper and Webshop environments (Figure 5). We observe that
8

---

Published as a conference paper at ICLR 2026
larger value of "uaa does not necessarily lead better final performance pass@3, instead, the op- timal setting of Yiraj varies across different environments. For Sokoban and Webshop environments, Yuaj
intermediate values (e.g., "iraj 0.6) yield the best results, suggesting that balancing immediate and long-term rewards is more important for these tasks. In contrast, MineSweeper benefits from relatively larger "firaj (e.g., Ttajj 0.9), indicating that extended credit assignment better supports strategic exploration in this environment. Overall, the results show that 'traj provides a practical way control the trade-off between exploration and exploitation across environments.
Sokoban
MineSweeper
Webshop
Sucress (98)
50
70
40
60
Rasegg
692
Pasa@3
0.6 Youj
0.75
0.6
Youj
You!
Figure 5: Success rates of models trained with different Traj- A higher value encourages more exploration during training.
6.2 ABLATION ON THE INTER-
INTER-EPISODE EPISODE MEMORY
In LAMER, the agent policy is adapted in-context through the inter-episode memory H()), which by default contains both the trajectories and reflections of previous episodes. To assess the influence of memory content to the training, we consider two alternative configurations H(()); (i) only his- tory trajectories; (ii) only reflections. The performance of the trained agents in each configurations are reported in Table 3 The results show that self-reflection provides a clear benefit in LAMER, leading 21.6% improvement on Sokoban, 11.0% on MineSweeper and 3.5% on Webshop, spectively. Interestingly, the reflection-only configuration also outperforms the default setting in LAMER (which contains both trajectory and reflection) across all environments. We hypothesize that this is because reflection-only memory presents more concise and focused guidance, leading to more effective adaptation of the agent's behavior.
Table 3: Comparison of LAMER with different inter-episode memory configurations.
Content in H(n)
Sokoban
MineSweeper
Webshop
Trajectory-only
34.8
69.5
89.3
Reflection-only
56.4
80.5
92.8
Both
55.9
74.4
89.1
6.3 TRAINING BUDGET
We next analyze the training budget of RL and Meta-RL, focusing on both data usage and compu- tational efficiency. To ensure a fair comparison, we set the group size for standard RL be three times larger than that of Meta-RL. This adjustment guarantees that the two methods consume the same number of trajectories for each gradient update. Aside from this scaling, all other experimental configurations-suine such as learning rates, batch sizes, and network architectures are held constant. This design choice highlights that Meta-RI. does not require larger data budget compared to RI;; in other words, both methods rely the same total number of trajectories to learn.
Nevertheless, LAMER might still introduce additional training time cost compared to the RL base- lines. In RL training, all the episodes could be sampled in parallel since they are independent. In contrast, LAMER exhibits less parallelism since episodes within the same trial needs to be generated sequentially. As result, observe around twice the training time cost for LAMER in current implementation. This suggests that more sophisticated sampling strategies, such as asynchronous rollout, could further improve the efficiency of LAMER for training L.M agents.

---

Published as a conference paper at ICLR 2026
7 CONCLUSION
Being able to explore and gather information from the environment is crucial in building autonomous agents that can adapt quickly and robustly. We introduced LAMER, general LLM agent training framework leveraging the principle of meta reinforcement learning. Unlike previous RL methods that maximize a single-episode return for immediate payoff, LAMER maximizes discounted cross- episode return, naturally balancing when to explore versus when to exploit to maximize long-term performance. The exploration allowed at training time teaches the agent general explorative strate- gies that enable rapid in-context adaptation at test time. We show across diverse environments that LAMER significantly outperforms RL methods, is able to generalize to harder environments, and scales better with more episodes at test time.
Limitations and future work. Our results raise several promising directions for future work. (i) The generality of our method allows for combining it with other RL algorithms or self-reflection frame- works. We hypothesize that a more advanced advantage estimation strategy or a stronger reasoning model may enhance the performance. (ii) Our approach requires sampling episodes sequentially for rollouts since episodes are dependent in cross-episode training. This eventually leads to longer training time than RL methods. More efficient training strategies will be explored in future work. (iii) Finally, LAMER trained on casier environments can generalize to harder environments of the same kind or relatively similar domains. This ultimately suggests possibilities in building generalist agents that can adapt to completely novel environments.
ACKNOWLEDGEMEENS
M.B. gratefully acknowledges the support of the Swiss National Science Foundation (SNSF) starting grant SMMSGI2.2222527 SNSF grant IC001O 231922, and the Swiss AI Initiative. M.B. CIFAR Fellow in the Multiscale Human Program.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in lims. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
AutoGPT. Significant-
Gravitas/Auto SignificattGGrvitts///a fully autonomous., 2023. URL. /tttp: tttpg/gtttuhh..oo com faaam//ignfffif haa.gniiiiiatattaaaaaaaaaaa
An experimental open-source attempt to make gpt- http:s///ggttto...oo///ffffffffr
Auto-GPT/tree/mase
Jakob Bauer, Kate Baumli, Feryal Behbahani, Avishkar Bhoopchand, Nathalic Bradley-Schmieg Michael Chang. Natalie Clay, Adrian Collister, Vibhavari Dasagi. Lucy Gonzalez, al. Human- timescale adaptation in an open-ended task space. In International Conference on Machine Learn- ing, 2023.
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shi- mon Whiteson. A tutorial meta-reinforeement learning. In Foundations and Trends in Machine Learning, 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis. Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec
10

---

Published as a conference paper at ICLR 2026
Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Me- Grew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. arXiv preprint arXlv:2107.03374, 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.1466,, 2021.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, et al. Deepseek-R1: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
Yan Duan, John Schulman, Xi Chen, Peter I. Bartlett, Iya Sutskever, and Pieter Abbeel. 2I.:: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.00777,, 2016.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-growe group policy optimization for Ilm agent training. In Advances in Neural Information Processing Systems, 2025.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
Kanishk Gandhi, Denise H J Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah Goodman. Stream of Search (SoS): Learning to search in language. In Conference Language Modeling, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Dan Hendrycks, Steven Basart, Sauray Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2021b.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-leaming in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Akshay Krishnamurthy, Keegan Harris, Dylan Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? In Advances Neural Information Processing Sys-
tems, 2024.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes. Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roclofs, al. Training language models to self-correct via reinforcement learning. In International Conference Learning Representations, 2024.
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, maxime gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. In International Conference on Learning Representations, 2023.
Yinghao Li, Haorui Wang, and Chao Zhang. Assessing logical puzzle solving in large language models: Insights from a minesweeper case study. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs as agents. In International Conference on Learning Representa- fions, 2024.
11

---

Published as a conference paper at ICLR 2026
Trung Quoc Luong, Xinbo Zhang. Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Rea- soning with reinforced fine-tuming tuming. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
Seungyong Moon, Bumsoo Park, and Hyun Oh Song. Guided stream of search: Learning to better search with language models via optimal path guidance. arXiv preprint arXiv:2410.0299,, 2024.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zeutlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. In Conference Empirical Methods in Natural Language Processing, 2025.
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vin- cent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. Mlgym: A new framework and benchmark for ad- vancing ai research agents. arXiv preprint arXiv:2502.144, 2025.
Allen Nie, Su, Bo Chang, Jonathan Lee, Ed H Chi, Quoc V Le, and Minmin Chen. Evolve: Evaluating and optimizing Ilms for exploration. arXiv preprint arXiv:2410.06236 2024.
Dongmin Park, Minkyu Kim, Beongjun Choi, Junhyuck Kim, Keon Lee, Jonghyun Lee, Inkyu Park, Byeong-UP Uk Lee, Jaeyoung Hwang, Jaewoo Ahn, Ameya S. Mahabaleshwarkar, Bilal Kartal, Pri- tam Biswas, Yoshi Suhara, Kangwook Lee, and Jaewoong Cho. Orak: A foundational benchmark for training and evaluating Ilm agents diverse video games. arXiv preprint arXiv::2006.03610 2025.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023.
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Yuxiao Qu, Matthew Y. R. Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement finetun- ing. In International Conference on Machine Learning, 2025.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. imagination-augman agents for deep reinforcement learning. In Advances in Neural Informa- fion Processing Systems, 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1xXXv:1707.06344 2017.
Amrith Setlur, Matthew YR Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Sim- chowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-time compute for LLMs. arXiv preprint arXiv:2506.0000 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, 2025.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Re- flexion: language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2026
Mohit Shridhar, Xingdi Yuan, Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference Learning Representations, 2021.
Côté, Yonatan Bisk, Adam Trischler, and Matthew
Charlie Victor Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effecttve than scaling parameters for reasoning. In International Confer- ence on Learning Representations, 2025.
Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations learning to explore via meta-reinforreemen learning. In Advances in Neural Information Processing System, 2018.
Fahim Tajwar, Yiding Jiang, Abitha Thankaraj, Sumaita Sadia Rahman, J Zico Kolter, Jeff Schnei- der, and Russ Salakhutdinov. Training a generally curious agent. In International Conference on Machine Learning. 2025.
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn. Springer, 1998.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024a.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.0576.3 2016.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 2024b.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in I1m agents via multi-turn reinforcement learning. arXiv preprint wrXiv:2504.20073, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 2022.
Robert C Wilson, Andra Geana, John M White, Elliot A Ludvig, and Jonathan D Cohen. Humans use directed and random exploration to solve the explore-exploit dilemma. Journal of Experimental Psychology: General, 2014.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.0077 2025.
Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning. arXiv preprint arXiv:2502.1 /9613, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338, 2025.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information P Processing Systems, 2023a.
13

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023b.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, 2022.
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, al. The landscape of agentic reinforcement learning for LLMs: A survey. arXiv preprint arXiv:2509.02547, 2025.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A TASK DESCRIPTION AND DETAILS
Sokoban. We include the classic video game Sokoban as fully-observable environment. The game is a 2D square board, with N boxes scattered on the board. There are also N target positions marked on the board. The player placed at an initial position, and the goal to push all the boxes to the target positions. There is no correspondence between each box and the target position. When the player walks into a box, gets pushed in that direction (if there's space). Boxes can't be pushed into walls other boxes. Once box is pushed into corner against wall with no way to get behind it, it might become permanently stuck. There is no pull operation in this game. The agent, therefore, has to think several moves ahead to avoid getting boxes stuck in positions where they can't 't reach their targets. The difficulty of this task is controlled by the board size, the number of boxes, and the wall structure of the board. We train
board size 6 x with 2 boxes.
MineSweeper.
We include the classic video game MineSweeper as partially-observat environ- ment. The game is 2D square board, with several mines randomly scattered in the board cells. The goal of the game is to open all the safe cells without revealing the hidden mines. In each step, the agent opens a cell, and the first step is always safe. If mine revealed, the task ends in failure immediately. The state of the opened (safe) cells can either be empty or a number from I to 8, and the number specifies how many mines are adjacent to the specific cell. The agent needs to use the numbers marked on the opened cells to reason about the position of mines. Success is achieved when all safe cells are revealed. Our implementation based on a simplified version [i et al.) (2024). The difficulty of this task is controlled by the board size and the number of mines. We train on a board size x 6 with 3 hidden mines.
Wehshop Yao et al., 2022). We include Webshop as partially-observen text-based environment that simulates online shopping. The agent is given a natural language instruction specifying a prod- uct to purchase with certain attributes. The environment presents simplified e-commerce interface where the agent can search for products, navigate through search results, and examine product pages with details like price, color, size, and customer reviews. The agent must interpret the instruction, search effectively, filter through multiple product options, and select the item that best matches the specified criteria. Success is measured by whether the final purchased item satisfies all the require- ments in the original instruction.
ALFWorld (Shridhar et al.) ) We include ALFWorld as a partially-observal text-based en- vironment that simulates household tasks in interactive fiction format. The agent receives natural language instructions for common household activities. The environment provides text descriptions of rooms, objects, and possible actions, while the agent must navigate through a house, interact with objects, and complete mulli-step tasks. Objects may need to be found, picked up, cleaned, heated, or combined with other objects to achieve the goal. The agent's view is limited to the current room and nearby objects, requiring exploration and memory previously visited locations. Success requires understanding the instruction, planning sequence of actions. and executing them correctly while managing partial observability. We train on the training examples of the activities 'Pick', 'Look', 'Clean', 'Heat'. We evaluate in-distribution on the test examples from the same activities, and we evaluate oft-of- distribbitt on the test examples from 'Cool', 'Pick2' activities.
15

---

Published as a conference paper at ICLR 2026
B EXAMPLE PROMPTS
We provide examples of prompts for each task. There are two types of prompts: (1) the standard version (with name 'Standard Prompt') used for prompting the agent play the game; (2) the reflection prompt used for self-reflection on a past experience (with name 'Reflection Prompt')
There are variables such in the prompts, among with other task-specific hyperparameters. They are omitted in the prompts for clarity. In practice, they will be replaced with the actual content. Note that the (past experience reflection) will be empty for the first episode.
(history.actionn
Similar to (Feng al. 2025), we use <action> </action> block indicate the final decision of the action, and we use <remark> </remark> to indicate the content of the self-reflection.
(see next page for the prompts)
16

---

Published as a conference paper at ICLR 2026
B. SOKOBAN
Sokoban Standard Prompt
You are an expert agent operating in the Sokoban environment.
# Symbols and Their Meaning
Walls (#): These block movement. You can't 't move through or push anything into walls.
Floor (.): Open spaces where you can walk and move boxes.
Targets (O): The spots where boxes need to go.
Boxes (X): These are what you need to push onto the targets.
Player (P): That's you! You'll move around the grid to push boxes.
Box Target (v): A box successfully placed on target.
Player on Target (S): You standing on target.
# Goal
Your goal is push all the boxes (X) onto the target spots (0). Once all boxes are on the targets, you win!
# Rules
Your admissible actions are ['up". "down". "left", "right"]. You can only push one box at a time. You can't pull boxes, so plan ahead to avoid getting stuck.
You can't walk through or push boxes into walls (#) other boxes. To avoid traps, do not push boxes into corners or against walls where they can't be moved again. (example}
# Observations
The initial state of the game is:
0:
4
#
1:
#
#
2:
#
0
3:
#
X
X
5: # #
{past_exx eeessteexeieeeee.. wwwpeiiencereeleettiooio reflection)
You have already taken the following actions: (bistory_actions. actions}
Your current observation is:
0:
#
#
#
#
1:
#
#
4
2:
O
X
3:
#
X
P
4:
5:
Now it's your turn to make moves (choose the next {num actions per turn] actions).
Your response first be step-by-step reasoning about the current situation observe the positions of boxes and targets, plan a path to push a box toward a target, and avoid traps like corners walls.
Then choose {num_actions actions aactions_per_turr pur_turn] admissible actions and present them within
<action> </action> tags (separated by comma).
17

---

Published as a conference paper at ICLR 2026
Sokoban Reflection Prompt
You are an expert agent operating in the Sokoban environment.
Symbols and Their Meaning
Walls (#): These block movement. You can't move through push anything into walls.
Floor (-): Open spaces where you can walk and move boxes.
Targets (O): The spots where boxes need go.
Boxes (X): These are what you need to push onto the targets.
Player (P): That's you! You'll move around the grid push boxes.
Box on Target (v): A box successfully placed on target.
Player on Target (S): You standing on target.
Your Goal
Your goal is to push all the boxes (X) onto the target spots (0). Once all boxes are on the targets, you win!
Rules
Your admissible actions are ["up". "down", "left", "right"].
You can only push one box at a time. You can't pull boxes, so plan ahead to avoid getting stuck.
You can't walk through or push boxes into walls (#) or other boxes. To avoid traps, do not push boxes into corners or against walls where they can't be moved again.
# Your Task
You will be given the history of past experience. Your job is to **ffflect the past sequence**, identify any *iistakes or inefficiencies**. and then devise a **concise, improved plan** starting from the original initial state.
Past Experience
The initial state of the game is:
0:
#
#
#
#
1:
#
#
#
2: #
O
3:
$
X
X
4:
$
5:
#
You have taken the following actions: {history_actee _ctioos)]
The final state is:
1:
#
#
#
0
2: #
O
3:
#
X
P
4:
#
#
5:
#
$
The task is NOT successfully completed.
Now it's your turn to reflect on the past experience and come up with a new plan of action. Your response should first be step-by-step reasoning about the strategy and path you took
to attempt complete the task. Identify where things went wrong or could better.
Then devise concise, plan of action that accounts for your mistake with reference to
specific actions that you should have taken.
Finally, end the response with your reflection and improved plan inside <remark>
</ rema r >> tags, to guide the next trial.
18

---

Published as a conference paper at ICLR 2026
B.2 MINES WEEPER
MineSweeper Standard Prompt
You expert agent operating in the Minesweeper game. You will be given a two dimensional [board_size) by (board_size) board, with (n_mines) hidden mines. The rows and columns are indexed from I to [izer.
# Cell States
Unopened cells (?): cells that are yet to be revealed and may contain a mine.
Blank cells (.): opened and non-mine cells, and they have no neighboring mines Numbered cells (1-8): opened and non-mine cells, and the number indicates how many mines are in the eight neighboring cells, including those diagonally adjacent. For example, a cell with '8' means all its neighboring cells contain mines. Mine cells (*): opened cells that contain mine.
# Your Goal
Your goal is to clear the board by revealing all the cells that don't t contain mines, without detonating any of the hidden mines scattered throughout the board. Use clues about the number of neighboring mines in each field to reason about the position of mines and non-nn mime cells.
Reveal Rules
Your admissible action is to choose ONE unopened cell (?) to reveal per turn. The outcome depends on the content of that cell: Blank cell (.): That cell is revealed, and all contiguous blank cells plus their bordering numbered cells are automatically revealed (auto-cascade).
Numbered cell (1-8): Only that single cell is revealed, showing the count of neighboring mines.
Mine (*): The game ends immediately in loss.
# Observation
The initial state of the game is:
Row 1:
Row 2:
1
Row 3:
Row 4: 1 1
1
2 ?
Row 5: ?
Row 6: ? 1
{past_ewn eee.eeeeeeeeeeeeeee wwwpeeiienceeeelecttinnnnno
You have already chosen the following cells reveal: (6, Your current observation is:
Row 1:
Row 2:
1
Row 3:
?
Rew 4: 1
Row 5: ? 1 1
Row 6: 1
Now it's your
make move.
Your should first reason step-by-step about the current situation observe the status of the board, inferring the states of unopened cells (?).
Then choose ONE unopened cell (?) to reveal. Put the index of cell in the format of "(row, col)" within the <action> </action> tag.
19

---

Published as a conference paper at ICLR 2026
MineSweeper Reflection Prompt
You are an expert agent operating in the Minesweeper game.
You will be given two dimensional [board_size) by (board_size) board, with (n_mines) hidden mines. The rows and columns are indexed from 1 to siierd_size)
# Cell States
Unopened cells (?): cells that are yet to be revealed and may contain a mine.
Blank cells (.): opened and non-mine cells, and they have neighboring mines Numbered cells (1-8): opened and non-mine cells, and the number indicates how many mines are in the eight neighboring cells, including those diagonally adjacent. For example, a cell with "8' means all its neighboring cells contain mines.
Mine cells (*): opened cells that contain mine.
Your Goal
Your goal is to clear the board by revealing all the cells that don't contain mines, without detonating any of the hidden mines scattered throughout the board. Use clues about the number of neighboring mines in each field to reason about the position of mines and non-min cells.
Reveal Rules
Your admissible action is to choose ONE unopened cell (?) reveal per turn. The outcome depends on the content of that cell:
Blank cell ()): That cell revealed, and all contiguous blank cells plus their bordering numbered cells are automatically revealed (auto-cascade).
Numbered cell (1-8): Only that single cell is revealed, showing the count of neighboring mines.
Mine (*): The game ends immediately in loss.
Your Task
You will be given the history of past experience.
Your job now is to **reflect on the past experience** identify any **mistakes or inefficien- cies*# and then devise **concise, improved plan** ** for your next try starting from the original initial state.
# Past Experience
The initial state of the game is:
Row 1:
Rew 2:
1
? Row 5: ? 1
Row 3:
Row 4: 1
Row 6: ?
You have chosen the following cells to reveal: [history actions} The final state is:
Row 1:
Row 2:
1
Row 3:
Row 4: 1
?
Rew 5:
Row 6: 1 1
The task is NOT successfully completed.
Now it's your turn reflect the past experience and up with plan of action.
Your response should first be step-by-step reasoning about the strategy and path you took
attempt complete the task. Identify where things went wrong or could be better.
Then devise concise, new plan of action that accounts for your mistake with reference to specific actions that you should have taken.
Finally, end the response with your reflection and improved plan inside <remark>
/ remark> tags, guide the next trial.
20

---

Published as a conference paper at ICLR 2026
B.3 WEBSHOP
Webshop Standard Prompt
You expert autonomous agent operating in the Wehshop e-commerce environment. Your task is to: Find me slip resistant, non slip men's loafers & slip-ons with rubber outsole, rubber sole with color: 1877blue, and size: 11.5, and price lower than 70.00 dollars. eeperteepeee experrinceerrffectionn ( [history-attions] actionss] Your admissible actions of the current situation are:
"searc[[your queryl', cclicksseacch]
Now it's your turn to take one action for the current step.
Your response should first be step-by-step reasoning about the current situation, then think carcfully which admissible action best advances the shopping goal.
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
Webshop Reflection Prompt
You expert autonomous agent operating in the Webshop e-commerce environment. Your task is to: Find me slip resistant, non slip men's loafers & slip-ons with rubber outsole, rubber sole with color: 1877blue, and size: 11.5, and price lower than 70.00 dollars. You will be given the history of past experience.
Your job **reffect on the past sequence** identify any **mistakes or inefficiencies** and then devise a **concise, improved plan** starting from the original initial state. Below are the last few actions and corresponding observations you have: [history _actions)
The task is NOT successfully completed.
Now it's your turn to reflect the past experience and up with new plan of action.
Your response should first be step-by-step reasoning about the strategy and path you took attempt complete the task. Identify where things wrong could be better. Then devise concise, new plan of action that accounts for your mistake with reference to specific actions that you should have taken.
Finally, end the response with your reflection and improved plan inside <remark>
</ remark> tags, guide the next trial.
21

---

Published as a conference paper at ICLR 2026
B.4 ALFWORLD
ALFWorld Standard Prompt
You are expert agent operating in the ALFRED Embodied Environment. Welcome to WoxtWorld, ALFRED!
You are the middle of room. Looking quickly around you. you see a bed a desk 2, desk 1, drawer 6, drawer 5, drawer 4, drawer 3, drawer 2, a drawer garbagecan 1, a laundryhamper 1, a safe 1, a shelf 6, a shelf 5, shelf 4, a shelf 3, a shelf 2, and shelf 1.
Your task to: put mug in desk.
prast_experience refleccioon/ /history_actions]
Your admissible actions the current situation are: go to bed 1'.
go to desk 1',
go to desk 2',
go to drawer 1',
go to drawer 2'
go to drawer 3',
go to drawer 4'
go to drawer 5',
go drawer 6',
go to garbagecan 1',
go laundryhamper 1',
go to safe 1',
go shelf
go to shelf 2',
go to shelf
go to shelf 4'
go to shelf
go to shelf 6',
'inventory".
'look'.
Now it's your turn to take an action.
Your response should first by step-by-step reasoning about the current situation.
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
22

---

Published as a conference paper at ICLR 2026
ALFWorld Reflection Prompt
You are an expert agent operating in the ALFRED Embodied Environment. Welcome to TextWorld, ALFRED! =
You are in the middle of room. Looking quickly around you, you bed 1, desk 2, desk 1, a drawer 6, a drawer 5, drawer 4, drawer 3, a drawer 2, a drawer a garbagecan laundrybamper 1, safe 1, a shelf 6, shelf 5, shelf 4, shelf 3, shelf 2, and shelf 1.
Your task is to: put mug in desk.
You will be given the history of past experience. Your job is to **reflect on the past sequence**, identify any *mistakes or inefficiencies and then devise a "*concise, improved plan ** starting from the original initial state.
Below are the actions you took and the corresponding observations: {history_actio _actions]
The task NOT successfully completed.
Now it's your turn to reflect the past experience and up with new plan of action.
Your response should first step-by-step reasoning about the strategy and path you took
attempt complete the task. Identify where things wrong could be better.
Then devise a concise, new plan of action that accounts for your mistake with reference to
specific actions that you should have taken.
Finally, end the response with your reflection and improved plan inside <remark>
</ remark> tags, guide the next trial.
23

---

Published as a conference paper at ICLR 2026
C TRAINING DETAILS
LAMER compatible with standard policy gradient algorithms. Without specification, we use GiGPO as the default optimization algorithm. During training, the self-reflection step is also explic- itly trained using the reward in the subsequent episodes. During training, we match the total number of experiences sampled for each example between RL and Meta-RL to ensure fair comparison. Specifically, for each sample N 3 episodes and set group size to 8 for Meta-RL, and use group size of 24 for standard RL training. Besides that, other hyper-parameters and configuration are kept the same between RI. and Meta-RI. training. We use Qwen3-4B the base model and train it with Adam optimizer and learning rate of 6. For Sokoban and MineSweeper, we train the agents with batch size of 16 for 300 epochs. In comparison, Webshop and ALFWorld. The environment reward is set to be 10 for successful trajectories and 0 for unsuccessful ones. We use temperature of 1.0 during rollout and 0.7 during evaluation. The maximum number of output tokens is set to 1024. Our code based on the training framework of verl (Sheng al. 2025) and verl-agent (Feng et al. 2025)
use batch size of 8 and 150 epochs for
D ADDITIONAL RESULTS
D.1 EXPERIMENTS DIFFERENT ARCHITECTURE
LAMER is designed as a general framework built on meta-RL principles and is model-agnostee agnostic. To validate this, we additionally ran experiments on Llama3. 1-8B-Instruct (Grattafiori 2024). showing that our method works well on models in different model architecture and model size.
Table 4: Performance on Sokoban and MineSweeper environments using Llama3. 1-8B- --s-ruutt as base model. The results of p@ 1, p@2 and p@3 denote the success rate (%) under 1, 2, and attempts, respectively.
Sokoban
MineSweeper
Method
p@1
p@2
p@3
p@
p@2
p@3
GiGPO
5.9
6.3
6.3
39.1
41.4
42.2
LAMER
17.6
19.9
20.3
47.7
61.3
65.6
In T we compare LAMER with the strongest baseline GiGPO (Feng et al. 2025), on Sokoban and MineSweeper. We show The results indicate that on Llama-3. 1-88-IInstuct LaMer still out- performs the RL baselines across the environments, demonstrating the general applicability of our method to different model architecture and size.
D.2 COMPARISON TO RL BASELINES WITH
MEMORY
In main experiment at Table 1 we follow previous work and evaluate the standard RI. methods without access to the inter-episode memory. For comprehensive evaluation, we further evaluate the RL trained agents with access to the inter-episode memory (reflections and previous trajectories). The results of pass@3 3 are shown in T We observe that the inter-episode memory enhances the performance of RL trained agents on Sokoban (+3.8%) and MineSweeper (+5.3%), while degrades the performance on Webshop (-1. (-1.2%). Nevertheless, LAMER still substantially outperforms RL baselines across the environments, demonstrating the advantage of the proposed method.
Table 5: Performance of RI. baselines with access inter-episode memory (pass@3).
Method
Sokoban
MineSweeper
Webshop
GiGPO (w/o memory)
44.1
55.1
75.2
GiGPO (w/ memory)
47.9
60.4
74.0
LAMER
55.9
74.4
89.1
24

---

Published as a conference paper at ICLR 2026
E EXAMPLES
In Figure 6 wc provide an example of trajectories and corresponding reflections produced by the agent when solving the MineSweeper game. Here each trajectory is represented by sequence of clicks (numbered cells) on the board. The mines are not visible to the agent and will lead to failure of the game if clicked.
Attempt
Attempt #2
Reflection
Rellccion v?
Reflection
In the past ottompt, slayer revealed cells 1) and (6, 3) The resulting store showed row column row 5, column 2 row umn 3 a and row 6, column is indicated there is mine (8, 3), why the game onded loss.
StepatySipp Reascning
iehete Mistakie and ietecieencies:
Thes reveared indicating that there exsicty one and (5, this information liiitdd and not very helplll steniffingg the ovact location of mine.
""Reveellng ling
neightoring However, since is adjecent
2. "RRevealing 3y This revealed a Indicating that (8, 3) a mine. This is direct loss, but also provides some information obout the surrunndnng
pow bs.
a. "Lack of etratogic foous""; The player did not consider the impilcations of the numbers in row and row 5, which are 1s and 2s. suggeoding that thero
are nearty mines.
New Aation
improve smattegy player should revised
mevaaing
provide moot
risking direct
""Reveal 17"h This cell asjocent 11 11, Reveoling 1) would hoto determine if mine neartyy especialty
indlcoiing that thwre woreclyy mine Fie 8 neighhoiing cells
2 ""Reveal (5, 2ym, This coll adjacent to 16. 2) and 2), would help cenormine ihehere is mine near 2), which as
q. ""Reveal (8. 4ym This oell adjacent 16. (Which a mine) and (5, 4) Reveating 15, 4) would help meterrine it there is mine neartty especiaily
saro ou (6, 3) la mine.
ahe Rellection and Improvuedt Plan cremarke
the past attempt, revealing (S, (S. a because 3) mine. avoid this, the nest step should be reveal 15, 1o check revelling would help any nearty adjacent 3). This approach would provide more Information without riskngg direct Faselly, revealing <teemarko
cliwetll
Reflection N2
ANd Biup-by-SSup Reasoning In the past attempt the player reven lnd the cells (5. 11, (8, 4) 18. 1), (8. 8) and (6. 5) The reautting state showed some prog ress but the tast was comppered sucoossdully by wis anatgre what went wring.
1. ""Revealing 11 This adjccnt row which has three is. Revealing i5. 1) would reveated sumounnting colls but since whohugh Revealing (6, have shown the number mines around but
nearity.
"Reveeling 4ym: This
column
sance
indicated that there least mine neartyy
2. "Russaling 1y This oell adjazent row 6, colume which ls Revealing 18, 1) would have shown the rumber of mines around it, but
since it wes 1. indicaned that theen is leaet one mine neartyy
""Reooeling (8. at", This cael adjeownt to mos 5, colume whith is Revewiing (6, d) wourdd have shown the number of minas sround it, but
pirce was 31. indicated that thore is lcast one mino neartyy
5. "Revealing 51mh This cell adjacent row 6, column which is 1, Revealing (5, 5) would have shown the rumber of mines around it, but
was 1, indicated that thane i lesul one mine neartyy
locue was that player focuse recealing
provide reet rntommation location enough
player rovealed that adjocent minws.
rovealed
kywntiying the wour lncation
and New Flan of Action
To improve the stattggy the player should foc on revealingg cole that are not adjecent to already revealed ools, and that are for enough away avvidd
being aflected by the auto-casce caucaide affect. The goal to kauntlly the location of the nir wa by revisllngg fngg calla that are not anjecent to any already inveated cellss and that are areas where the numbers indicate a higher leellhood of mine.
Horo a revised plan:
sy": This
adgecaet any absady that
enough away being aflected by subb-casradd
adfaccen
2. ""Revel sy"h This cell adiccnt atradd revested colls least mine
avaid affected
cascade effect.
adacent
indicating that
2. "Russal (8, 1y": This oul acjacent now 5, co corumn which a indicazin that there al least one mine rearby. Howaver, since la adjooanc
to mw 6. oskumn 2, which i 1, is important to reveal this onll to confirm the location of the mine.
a. "TReowal (8. 4ym This cell is aejeooet to mo 5, canumn 4, which in.s indrowing thet theme is loos onw mine rwarbyy Hoseoeer sinos is adjeoset
to row column's 3, which a 1. Es important to reveal this cell to confirm the location of the mine.
"Reooel 5y" Thise cell wejecint nov 5, canumn which s indlowing the theme al lwst onw mine reaeby Howsoer since is adfeetent
to now oalumn 4, which aawe
important to reveel this oell to contfim the location of the mine.
By tellowing this plan, the player identry the locatin cremarie
mines more eflycivvly and avaid mistakes made the previous attempt.
unnoonssary information and wrough prog towerd ddentilyngg the wosec lncaiton the minss To improw the strwlegy player ahouls foeus newes celis that was ad jecent to already revealed colls. and that are for enough oway to avaid being affected by the auto cascade effect. The revised plan invelves revoaling colls push as (8, 21, (5. 3), 05, (5. 4) and (8, d) identity the location of the mines more affectivety shvumarks
in the past attempt the player rove aed owlee that aojer to stready revented owlls, leading
Figure 6: Example of trajectories and reflections produced by LAMER trained agents on the MineSweeper game.
25


Reviews:
[{"review_id": "YupAlSFk7D", "reviewer": "Reviewer_3QcN", "summary": "This paper introduces the LAMER meta-RL algorithm which makes agents explore better. The algorithm involves training the agent on sequences of related expisodes. After each algorithm there is a reflection step where the agent can write notes which are given to figure episodes. Since the meta-rl algorithm rewards the agent for reward across the entire sequence, the agent is incentivized to explore in earlier episodes and exploit in later ones.\n\nThey evaluate on Sokovan, Minesweeper, Alfworld, and Webshop, and find that LaMer produces both better exploration/diversity and higher overall reward.", "questions": "- Why are there no seeds or error bars? RL experiments can have quite a lot of variance and it is typical to do at least 3 experimental seeds for each run.\n- How much of the gain is attributed to the self reflection prompting versus the multi episode credit assignment?\n   - Would be interesting to see an ablation of whether the reflection step is important or if merely showing the agent past episodes is sufficient.\n- Qualitatively, did you notice the sampled trajectories to be more diverse? Entropy or similarity is an easily hackable objective. I'd also appreciate a bit more description on how exactly diversity was calculated, this was not entirely clear to me.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-motivated and is written clearly to explain how the authors are tackling the problem of exploration during RL.\n\nWhile the Meta-RL framework itself is not novel, the application to LLMs is. The paper shows significant gains over single episode training across multiple benchmarks. The paper also shows out of distribution generalization compared to non meta learning on unseen benchmarks.", "weaknesses": "See the \"questions\" section.\n\nBeyond this I would be interested in comparing against pass@k metrics for meta-exploration that have previously been explored in RL (for example, see Walder et. al Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems).  (I do not think this is necessary for this to be a good paper! Just a suggestion for extension.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uSIccfCpsB", "reviewer": "Reviewer_73EN", "summary": "This paper uses meta-reinforcement learning (Meta-RL) to train large language models (LLMs) as language agents across multiple game environments. The inner loop of Meta-RL is implemented as a simple reflection step, where the LLM is prompted to analyze and reflect on previous episodes. The outer loop is optimized using the GiGPO algorithm. The authors show that this approach improves performance compared to both RL and prompting-based baselines across various games. Their analysis further suggests that Meta-RL enhances the exploratory behavior of LLM agents, leading them to produce more diverse trajectories.", "questions": "- Would your method also apply to more classical reasoning tasks where LLMs are trained with RL, such as MATH, BigBench, or similar benchmarks?\n\n- Do you think your approach could work even without the reflection step — relying solely on in-context learning from previous trajectories provided in the prompt?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow, with clearly stated hypotheses and a clean experimental design. The idea is simple and promising to help improve LLM-agents to explore in RL tasks. \n\n- It presents strong empirical results showing that meta-RL with reflection substantially improves the performance of LLM agents across multiple game environments.", "weaknesses": "- The paper does not analyze the reflections generated by the LLM. Do they make sense? Do they evolve over time? Is performance improvement mainly driven by changes in the policy or in the reflections themselves?\n- The paper does not analyze the reflections generated by the LLM. Do they make sense? Do they evolve over time? Is performance improvement mainly driven by changes in the policy or in the reflections themselves?\n- All experiments are conducted with a single LLM (Qwen3-4B). It would be important to test whether the results generalize across different model architectures and sizes.\n- Lack of technical details:\n    - The paper does not explain how trajectory diversity is computed.\n    - Key training hyperparameters (e.g., batch size, learning rate, training budget) are not reported.\n    - For procedurally generated environments such as Sokoban, it is unclear whether the same random seeds are used across episodes within a task—both in the RL and meta-RL settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qHbR6IHpsY", "reviewer": "Reviewer_xsGo", "summary": "The paper proposes a Meta-RL framework for LLM agents, combining cross-episode RL and in-context reflection. The idea is to train the model to learn to explore by first gathering information, then exploiting it across the following episodes. Results show clear gains over RL and prompting baselines across Sokoban, MineSweeper, Webshop, and ALFWorld. The approach seems effective at leveraging multi-episode structure and reflection to adapt at test time, though it’s unclear whether the gains truly stem from “learning to explore” or from other confounding factors such as longer contextual horizons or additional information flow between episodes.", "questions": "* How is the “empirical distribution over distinct trajectories” computed exactly?\n* Are prompting baselines also given access to memories across episodes? Reflection only? or histories too?\n* In addition to “bold” indicating best performance, it is useful to make other performance that are not statistically significantly worse than the best (e.g. underlined).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is clear and well motivated. It’s conceptually simple and general. The evaluation is broad and the study of generalization is interesting. The paper is clear and well-written.", "weaknesses": "**What is actually learned?**\nIt’s not so clear to me whether the agent is only conditioned on the reflection from the previous episode or on the previous history too? If conditioned on both, it would be good to ablate the reflection mechanism and see how performance holds. Ablating history and only keeping reflections might be interesting too. \n\nIs the feedback generation capacity trained too? Or just leveraging a frozen model? Maybe the approach doesn’t actually train exploratory behaviors but only improves feedback generation and feedback use in the following episode. \n\nTo make sure it’s not the case, it would be interesting to visualize exploration strategies in the first episode: what are agents doing? Maybe through videos on a website. Looking and analysing generated feedback might be useful too. Overall I would like to know more about the exploratory behaviors this paper claims is being trained. \n\nIt might be interesting too to feed a first episode collected by either 1) random actions, or 2) the base model; then generate feedback and run the second episode with the trained Meta-RL agent. Is there a drop in performance? If so, it means the meta-RL agent truly learned to explore. If not, then exploration doesn’t matter so much, what matters is the feedback offering a “second chance”. \n\nAre the different episodes in a group the same task instance? e.g. with mines in the same spot in mine sweeper? I assume it’s the case, otherwise exploration would be useless, right? In that case, in Sokoban and MineSweeper, it may not be so much about “learning to explore” as it is about giving a second chance the agent can actually leverage (second chances can’t be leveraged in the standard RL condition because there is no inter-episode memory). \n\n\n\n**Missing information about training task distribution**\nIt’s unclear how different training tasks are from each other. If they’re similar, “learning to explore” might just mean reusing the same strategy in all task instances? Are agents adapting their exploration (1st episode) to the task instance? Here again looking at replays would be useful.\n\n**Comparisons**\nThe RL baseline controls for total experience but doesn’t control for the length of experience the agent has access to at any given moment. One way to control for this would be to run the RL agents on longer episodes instead of running it on longer trajectories. \n\n**Generalization to harder tasks**\nI don’t know if you can say that meta-RL generalizes better here: performance drops by similar percentages in both cases. Meta-RL doesn’t “hold better”, it just starts higher. This is a bit of an overclaim. The AlfWorld generalization study is more convincing. \n\n\n**Typos:**\n* fari comparison -> fair\n* Tabel 1 -> Table 1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OUWQLYM0hr", "reviewer": "Reviewer_Qdy5", "summary": "This paper proposes a meta-reinforcement learning framework, LaMer, which is designed to induce exploration and adaptation in large language model agents. Unlike conventional single-episode RL setups, LaMer uses a cross-episode training paradigm, where an agent interacts with the same environment across multiple episodes, balancing exploration and exploitation through a trajectory discount factor. The method leverages in-context policy adaptation via reflection, enabling agents to improve within trials without parameter updates.\n\nThe authors evaluate the method on Sokoban, MineSweeper, WebShop, and ALFWorld show that LaMer outperforms prompting-based and RL baselines (PPO, GRPO, GiGPO). It achieves a good marginal performance gain on complex environments and improved generalization to harder or out-of-distribution tasks. The framework introduces a bridge between meta-learning and RL for LLM-based agents.", "questions": "- Meta-objective design: How sensitive is performance to the trajectory discount factor? Could adaptive scheduling of this factor improve learning stability?\n\n- Reflection mechanism: Did the authors experiment with alternative reflection prompt formats or automatic summarization of past episodes? It would be great to add it in the ablations. \n\n- Training efficiency: Can rollout dependencies be partially parallelized (e.g., via curriculum or pseudo-batch episodes) without breaking credit assignment? It will result higher training efficiency. \n\n- Exploration quality metrics: Besides trajectory diversity entropy, are there other quantitative metrics used to measure exploration effectiveness?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Novel framework design: The cross-episode MetaRL formulation extends standard RL to multi-episode exploration, introducing a new axis of “meta-time” learning that aligns well with LLMs’ in-context adaptation capabilities. The introduction of the trajectory discount factor provides interpretable control over the exploration–exploitation tradeoff.\n\n- Improvements over the current baselines over benchmarks: LaMer consistently outperforms RL baselines across all benchmarks (Table 1), achieving +20% absolute improvement on Minesweeper and WebShop. Figures 3–5 show substantial increases in trajectory diversity and test-time scaling behavior, supporting the exploration hypothesis.\n\n- Good generalization evidence: Demonstrated robustness to task difficulty (Figure 5) and generalization to unseen ALFWorld tasks (Table 2). Shows that meta-learned exploration strategies can generalize beyond the training distribution, which is an important step toward adaptive, open-ended agents.\n\n- Broader impact: Provides a reasonable path to unify RL-based training and in-context learning for future meta-learning language agents. Bridges recent work in test-time compute scaling and reasoning through reflection.", "weaknesses": "- Limited novelty in algorithmic components: While the formulation is clean, most components, cross-episode return accumulation and reflection-based adaptation, are incremental combinations of known techniques. The paper could better articulate what is fundamentally new beyond adapting Meta-RL for LLMs.\n\n- Insufficient analysis of failure cases: The paper lacks qualitative examples or ablation studies on when reflection adaptation fails (e.g., misleading feedback loops or hallucinated self-reflection). As well, it does not have analysis of how context length or reflection prompt quality impacts learning efficiency.\n\n- Sequential dependency and compute inefficiency: Section 5.6 notes that MetaRL training is 2–3× slower due to sequential episodes, but does not propose mitigation strategies or quantify the trade-off precisely (e.g., wall-clock vs. sample efficiency).\n\n- Evaluation fairness concerns: It is unclear whether RL baselines had access to equivalent multi-episode feedback or if they were constrained to single-episode objectives, potentially overstating MetaRL’s advantage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)