Task-Specific Exploration in Meta-Reinforcement Learning via Task Reconstruction
Authors: Radu Stoican
Angelo Cangelosi
Christian Goerick
Thomas H Weisswange
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning trains policies specialized for a single task. Meta-reinforcement learning (meta-RL) improves upon this by leveraging prior experience to train policies for few-shot adaptation to new tasks. However, existing meta-RL approaches often struggle to explore and learn tasks effectively. We introduce a novel meta-RL algorithm that learns to learn task-specific exploration policies for sample-efficient few-shot adaptation. We achieve this through task reconstruction, an original method for learning to identify and collect small but informative datasets from tasks. To leverage these datasets, we also propose learning a meta-reward that encourages policies to learn to adapt. Empirical evaluations demonstrate that our algorithm achieves higher returns than existing meta-RL methods. Additionally, we show that even with full task information, adaptation is more challenging than previously assumed. However, policies trained with our meta-reward adapt to new tasks successfully.

Published in Transactions on Machine Learning Research (04/2026)
Task-Specific Exploration in Meta-Reinforcement Learning via Task Reconstruction
Radu Stoican"
rudu.soiicnnmmmnchsstrrr.c
Manchester Centre for Robotics and AI
The University of Manchester Manckester, United Kingdom
Angelo Cangelosi
angelo. hmpploo.cnnloss@mmmchhete....c..
Manchester Centre for Robotics and AI The Universily of Manchester Manchester, United Kingdom
Christian Goerick
christian porprtn__pprcc@@hhddrr..oom
Honda Research Institute USA, Inc. San Juse, California, USA
Thomas H. Weisswange
thomas. thomms...rissaangggoooooda...d
Honda Research Institude Europe Gmblf Offenbach om Main, Germany
Reviewed on OpenReview: https: openreveee. net/ forum? id= VRRap VeaJH
Abstract
Reinforcement learning trains policies specialized for single task. Meta-reinforcement learning (meta-RL) improves upon this by leveraging prior experience to train policies for few-shot adaptation to new tasks. However, existing meta-RL approaches often struggle to explore and learn tasks effectively. We introduce a novel meta-RL. algorithm that learns to learn task-specific exploration policies for sample-efficient few-shot adaptation. We achieve this through task reconstruction, an original method for learning to identify and collect small but informative datasets from tasks. To leverage these datasets, we also propose learning meta-reward that encourages policies to learn to adapt. Empirical evaluations demonstrate that our algorithm achieves higher returns than existing mela-RL methods. Additionally, we show that even with full task information, adaptation is more challenging than previously assumed. However, policies trained with our meta-reward adapt to new tasks successfully.
1 Introduction
Research in reinforcement learning (RL) has led to many impressive achievements and powerful methods in the past decade (Sutton 2018). However, real-world usage is still not widespread, as the algorithms used to solve RL problems often suffer from low sample efficiency and poor generalization to new tasks (Hospedales) cct al.) 2021 Beck et al.] 2023b). Meta-reinforcemont learning (meta-RI.) is a ro-emerging approach that tackles these issues. It follows the meta-learning approach of "learning to learn" (Schmidhuber 1987 Thru & Pratt 1998), making well-suited for few-shot adaptation settings. In few-shot adaptation, agents aim to become optimal in any new task from given distribution of tasks, after collecting only few episodes.
*Corresoondngg author.

---

Published in Transactions on Machine Learning Research (04/2026)
Instead of directly learning to solve tasks from scratch, meta-RL finds RL algorithms that quickly learn the optimal policy. While meta-RI. agents are general, the algorithms they produce are task-specific. These meta- learned algorithms contain prior knowledge of the task distribution. By leveraging this prior, they quickly learn the optimal method of exploring or solving a task. Therefore, for any task in the distribution, meta- learned algorithms are expected to more sample-efficient than and outperform standard RL algorithms.
Meta-RL has already been successfully applied ad hoc teamwork Zintgraf al. 2021a He al. 2023
Mirsky et al. 2022). robotics (Nagabandi et al. 2018 Zhao al. 2022); human-robot interaction (Gao et al, Ballou et al,, 2023), multi-agent RL (Yang et al. 2022 Xu et al. 2022a and sim-to-real transfer (Arndt 2020 Despite this, meta-RL has not yet fully addressed the challenges it aims to overcome (Stoican 2023 While recent years have seen several improvements, meta-RL still struggles with adapting to complex tasks. A significant issue is the difficulty of designing strategies for few-shot exploration in complex, dynamic RI. environments. The advantage of meta-RL. is that these strategies do not have to be manually crafted, but can be learned from data for each particular task distribution. Often, meta-RL agents interact with task for only few episodes before being evaluated on that same task. These agents can be seen exploring the task by using implicit or explicit meta-learned exploration strategy. However, implicit exploration may be too sample-ineffiiiett for few-shot adaptation (c.g., leading to agents that do not use their task distribution priors effectively during exploration). Recent approaches have shown that explicitly optimizing for few-shot exploration leads to more powerful adaptive agents (Beck et al. 2023b). Moreover, the difficulties of few-shot adaptation extend beyond task exploration. Our empirical results complement those of Beukman et al. (2024) in showing that optimizing policy that leverages learned priors solve tasks can be non-trivial.
2019
ct al.
et al.
To get clear high-level picture of in-context meta-RL, consider separating it into three main parts, shown in Fig. [ The first phase is task exploration. After task is sampled from the task distribution, a policy explore collects a "few" episodes of data from it. Optionally, this policy may be guided by an encoder fur The goal of explore is to collect data that contains useful information about the task. This information is then extracted by a second encoder g during the task learning stage, and encoded into a lask context vector. Finally, in the task-solving phase, a policy attempts to optimally solve the task at hand. This final stage is similar to standard RL, except that is conditioned on the task context.
Task Distribution
Task Solving
Few Episodes
One Episode
Exploration Policy explore
Task
Task-Solving Policy π
Exploration Data
Task
Encoder fa
Encoder g
Context
Task Exploration
Task Learning
Figure 1: A high-level overview of in-context meta-RL.
There are several ways of learning to reinforcement learn. The approach previously described, and used in this work, is called in-context meta-RL. In-context policies explore and use task context information provided by Su and to explore and solve (asks, respectively. Note that this three-phase paradigm is not always cleanly separated in practice, e.g., encoders fa and may share information or not be separate encoders at all, a single policy may be used for both task exploration and solving, etc. Nevertheless, this view helps form an intuitive understanding of meta-RL. Besides aiding intuition, we also use Fig. throughout this paper clarify which of the three components (i.e., task exploration, learning, solving) we discussing.
2

---

Published in Transactions on Machine Learning Research (04/2026)
We introduce Latent Space Exploration via Task Reconstruction (LaSER), novel meta-RL algorithm for few-shot adaptation D The primary objective of LaSER is to learn task-specific exploration strategies. The core idea behind our method is to identify small datasets that capture rich task-specific information with minimal redundancy. We achieve this by introducing a novel approach of meta-learning by lincarly reconstructing these small datasets into much larger datasets collected from the same task. We refer to this process as task reconstruction. It serves as a mcasure of how effective a given small dataset for few-shot adaptation. Following this measure, LaSER meta-learns a latent space for directly identifying such small datasets, without having to first collect a larger one. This allows training sample-efficient few-shot exploration policy that can collect task-specific data online. We then use the context vectors encoded from this data to train a policy that solves tasks. However, we show that previous approaches to training such a policy fail, even when using optimal task contexts provided by an oracle. Moreover, this failure persists even when the task distribution is replaced by a predefined set of tasks. As a result, LaSER's secondary objective is to learn to use task contexts effectively. To achieve this, we propose an augmented RL objective that encourages the task policy to continue exploring until contexts are effectively exploited. Overall, LaSER's s optimization regime and architecture leverage data- and compute-heary training to achieve rapid test-time adaptation requiring low number of environment interactions.
We summarize our main contributions as follows.
1. We formalize an important assumption about the data roquired for few-shot adaptation. This forms the basis of our task reconstruction method. We then show how an encoder built from these ideas can learn representations that facilitate exploration.
We leverage these representations to design an intrinsic reward for training a lask-specific, elficient few-shot exploration policy.
3. We propose using our aforementioned augmented objective to improve the adaptability of task- solving policies through directed exploration of the task context space. This simple change allows mcta-RL policies to be trained with standard RL algorithms.
4. We introduce a hybrid encoder architecture composed of unidirectional and a bidirectional model. The former is used for online task exploration, while the latter offers richer task contexts.
We formalize the meta-RL problem and introduce the necessary background in Sec. 2 In Sec. 3 the objectives described above and introduce LaSER, our novel meta-RL algorithm. We position our work within the meta-RL literature in Sec. 4 In Sec. 5 we provide empirical results by evaluating LaSER against existing meta-Rl RL algorithms. We first assess LaSER's performance and adaptability to new tasks after a short exploration phase (i.c., four episodos). We then analyze each individual component, i.c., exploration policy, encoder, and task policy. Soc. 0 then provides a high-level discussion of the effect and applicability of our work within the broader meta-RL context. Finally, we conclude and provide directions for future research in Sec. 7
we address
2 Preliminaries
Meta-Reiifor Learning. Mcta-RL extends the standard RL problem. Instead of learning from a single task, meta-RL agent trains on a distribution of tasks. Usually, the agent's 's objective is to adapt to new tasks. We consider a probability distribution p(M) over tasks and define each task M, ~ p(M) as a Markov decision process (MDP) (S, T,, R,, 7, H). All MDPs share the set of states S, set of actions A, discount factor 7, and horizon H. However, the transition function, given as the probability (s's s, s.a.M) of transitioning from stale to state s' by laking action in task Mi, is task-dependent. Similarly, the reward function R, is task-dependent. An episode 5 {st, re)/ is a sequence of states, actions, and rewards, such that at cach time-step a tuple (st, (st,at,7 rı) is collected. For distribution over episodes P (T), we denote T ~ PM; to be an episode collected by following a policy <aa 8) in task M,. The return of an episode T is the accumulated discounted reward G(T) = Σ/1 7're
'Qur code publicly available at
3

---

Published in Transactions on Machine Learning Research (04/2026)
In-Context Meta-RL A common approach to solving meta-RL problems is in-context meta-RL, where pretrained policy acts as adaptable RL algorithm (Duan et al. 2016 Laskin et al. 2023 Moeini et al. 2025). Formally, given a context vector C that describes a task M,, the goal is to train a policy я to optimally solve M, by taking in-context actions a ~ x(- | s. cj), conditioned on both the state s and the context. C₁. The meta-RL, agent can therefore learn new policies from Ca even post-training. A standard approach is to concatenate 8 and G,, then optimize the in-context policy a(a (, using standard RL algorithms. However, this simple method may not generalize well across tasks in complex task distributions (Beukman et al.) 2024.. To address this, we propose novel meta-RL optimization algorithm that explicitly uses C₁ to train in-context policies.
Meta-RL for Few-Shot Adaptation. In few-shot adaptation settings, in-context meta-RL attempts to optimize π to solve previously unseen tasks M2 ~ p(M) in a sample-eficciet manner. However, because the meta-RL agent lacks knowledge of the task-specific dynamics T, and R2 it must first collect data from M₁. The few-shot constraint limits this to K episodes per task. We call this K-shot adaptation problem. Following Beck et al. (2023b), we refer to a sequence of K episodes collected in M, as a mela- episode D(A) = {r We can then compute a latent context vector c, = g(D(k)) using an encoder Importantly, c, should capture information about T, and R,, to allow generalization to new tasks. To collect such meta-episodes, we follow Liu et al. (2021) and define a separate exploration policy rexplore, decoupled from the task-solving policy x. For given task, its goal is to collect an informative meta-episode, without necessarily achieving high return. We define xexplore s, s,T,) to be an in-context policy, where I, is a task representation encoded specifically for exploration. Although we choose I; to be the same C₁ it is less restrictive to use a new representation. Given this sctup, we generalize Liu et al. 20221 and consider agents that maximize the meta-RL objective
J(T, n explore g) = E MM w explore Pa V7 (VDD(())))
(1)
where (c:) PM (G(T))
is the expected return in task M, for the in-context policy conditioned on C..
Training Setup. Meta-training is the process of training a meta-RL, agent. The agent seeks a K-shot algorithm to optimize policies for each task drawn from p(M). The agent's ability to generalize is evaluated during meta-testing by solving new tasks from p(M). For each meta-testing task, the agent is first allowed to collect and learn from K episodes. Then, it is evaluated according to Eq. [ We refer to collecting K episodes prpale as task exploration. This type of exploration is meta-learned to be task-specific. Besides task from PM. exploration, the agent still performs the standard RI. exploration. To achieve their objectives, and x explore have to explore their environment during meta-training. This is commonly referred to as meta-oxploration (Beck et al. 2023b). During meta-testing, there is no meta-exploration.
Data Representation. States, actions, and rewards are represented as vectors or scalars. timestep (s,, (g, rt) then a d-dimensional vector, where d is the sum of the dimensions of st, at; and re. We extend this to episodes. An episode 7 E IR' is vector of H concatenated timesteps, where = Hd. We can further extend this to meta-episodes. For a meta-episode (KK) "f'x composed of K episodes, the k-th column D(K)) corresponds to the k-th episode. Episodes and meta-cpisodes can therefore be represented as tensors instead of sequences, when convenient.
.,k
3
Methods
In this section, we present approach to in-context meta-RL and use the ideas introduce to design the main components of our proposed algorithm, LaSER. We structure this section based on the three-stage meta-RL paradigm in Fig. [ Sec.
3.1 corresponds to the task-solving phase, while Secs. 3.2 and 3.3 focus on the task exploration part. Sec. 3.4 continues task exploration but also discusses task learning, with Sec. 3.5 corresponding to the combination of these three stages into a complete meta-RU RL algorithm.
Specifically, in Sec. 3.1 we introduce a novel approach for in-context RL optimization. Given context vector C; for task MG we propose augmenting the standard RL objective with a term that encourages
4

---

Published in Transactions on Machine Learning Research (04/2026)
meta-explooatton whenever can achieve better returns by improving its understanding of c.. Sec. 3.2 then discusses the training of an exploration policy xexplore for collecting the data required to compute It assumnes a given encoder fu for guiding x'xplore, with fu modeled as a unidirectional transformer (Vaswani ict al. 2017) of size dmodel. We propose a novel method of training fu for few-shot task-spocific exploration We first introduce an important assumption ion on the data collected for few-shot adaptation, then show how a practical learning objective, i.c., task reconstruction, can be built. from it.
in Sec. 3.3
Next, we describe our proposed architecture for encoders fu and in Sec. 3.4 We model g as transformer, their ability to process each timestep in the context of an entire meta-episode has been shown to be advantageous in meta-RL scttings (Melo) 2022 Shala et al. 2024). Finally, in Sec. 3.5, we introduce our meta-training and meta-testing algorithms. From this point forward, we simplify notation and drop the task subscript whenever it is clear that we are referring to task-specific data or representations.
3.1 Task Solving
optimizing an in-context policy T(a | St. c) to adapt to new tasks is non-trivial [Beukman et al. 2024 We hypothesize that this difficulty arises because ı fails to leverage the task- specific information in context e when solving M, While standard RL requires exploring the state space to understand a single task, meta-RL demands an additional
As discussed in Sec. 2
exploortto objective: the agent must explore to understand the relationship between 8t and To encourage this directed meta-exploration of the state-context space, we augment rewards with novel term that encapsulates meta-RL performance. This "meta-reward" allows the in-context policy to be optimized using standard RL algorithms.
Following this intuition, we propose method that encourages π to meta-explore more if e is not properly used. Let (( c) and (ss) be the approximated value functions at state st, with and without considering e, respectively. We assume that, for an optimal policy, V(*+) S ((*,, c) should hold at any timestep 4, i.e., information from e should never decrease performance. Whenever π violates this inequality during meta- training for some state 8t, the agent should meta-explore more from 8t. Our solution adopts the standard approach of encouraging exploration by maximizing entropy (Williams & Peng 1991 with the important distinction that this maximization is guided by the context c. We therefore propose the meta-reward
r. = + w/s, c) S[r] c), where w(s c) max (0, tanh (V+))) V(s c) -()
(2)
(3)
is a non-negative dynamic weight on the entropy S of T(- 8t, c), ri is the environment reward, and |3 is a constant. Note that w(st, c) is non-zero only when V(s,) - V( c) 2 C, for a given threshold Ç Ç (-oo, 0].
Intuitively, w(s;, c) > means that the agent is disregarding information provided by and should meta- explore in state 8t- When w(s,, = 0, i..,, V(s,) V(s,, + Ç, we recover the original RL objective, with ri Tt. By simply replacing Tr with r²", can be optimized in-context using standard RL algorithms. In practice, we use the proximal policy optimization (Schulman et al. 2017, PPO) algorithm, stabilized via proximal feature optimization (Moalla et al. 2024 PFO) (see full details in Appendix A.1
This subsection introduced a novel meta- explorxpno method for meta- veta-training task policies. However, it does not address task exploration. We tackle this distinct challenge in the next subsection.
3.2 Task Exploration Policy
While the approach in Sec. 3.1 addresses how to use a task context c effectively, it assumees such a context is already available. We now formally discuss how to obtain via task exploration. The objective of the exploration policy exxplore is to collect a single informative meta-episode D(K) for a task M, ~ p(M). Specif- ically, exploree should gather information about the task-specific dynamics of M; while avoiding irrelevant redundant exploration.
To quantify informativeness, we rely on a latent representation r = fa(D(K)) E (Hddmod)) computed by encoder fu: with P. denoting the representation of the k-th episode ((K) We optimize fo (detailed in Sec. 3.3) such that the cosine similarity Sc(T.,, T.,e)) is inversely proportional to how useful it is to
5

---

Published in Transactions on Machine Learning Research (04/2026)
collect episode D. after having already collected D for any k k' E [K], k' Therefore, an informative meta-episode contains a set of diverse, orthogonal episodes with average pairwise similarity of approximately zero.
We leverage this property of F to optimize explore Let the average causal similarity score ds = R
1 S ,c, .x)) measure how similar the current k-th episode to the history of previously col- lectod episodes {l....,k 1), while hiding similaritics to future episodes. To encourage task exploration, the end of the k-th episode, the agent receives the task- desendepnnd intrinsic reward
ph = exp
di
(4)
where o is a constant, and a reward of zero at each non-terminal timestep. Intuitively, this reward is maximized at d2 = O, encouraging the agent to minimize the similarity between episode and the previous (k- (k-1) episodes. In practice, we train xxxplore to maximize this intrinsic reward using PPO. A more detailed overview of how create the average similarity scores is given in Appendix A.2
For oxplore to collect useful episode, it must have knowledge of the episodes it has already collected. To create an exploration-focatn task context, we denote D(sk;;t) to be an incomplete meta-episode, containing
(k- 1) 1) complete episodes, and the first timesteps of the k-th episode, for k E [K], E [H]. We then condition the in-context policy π explore on the history ((kk,)) = (T((k.::))) and explore episode & by taking action af+1 ~ n (x 1841, p(sk.))). The overall lask exploration process is visualized in Fig. 2
D(tk.8-1)
fu
1)
pk
n explore
Task M₁; episode k 8/
8/ ag r k
Figure 2: Task exploration. Policy π "xxlore explores a task M, by collecting K episodes. At each timestep of each episode k; explore is conditioned on the current state sf and the latent representation ((k,tt-1) of previous timesteps and episodes. Its objective is to take actions that maximize the intrinsic exploration rewards pk Both (1katt1) and f are computed by the encoder fu from the data collected.
3.3 Learning to Explore by Reconstructing Tasks
The exploration policy described in the previous section relies heavily on the assumption that the latent space I produced by fu encodes the informativeness of collected data in its geometry. In this section, we formally introduce training method for fu that ensures this assumption holds.
Throughout this section, our focus is centered on encoders and fu, which enable task solving and task exploration, respectively. Let E Q x be a lensor containing Q meta-episodes collected from lask MM. Each slice B, (shorthand notation for Byaa), for j E [Q], corresponds to a distinct meta-episode D(K), as defined in Section 2 While D(K) was used to represent a single independent meta-opisode, we adopt this change in notation emphasize that each Bj component of the batch tensor B.
Recall that, given By, we use (B,) for task-solving and r = fu (By) for task exploration. Encoders g and fu are optimized based on key assumption about the data used for few-shot adaptation. To formalize this, we first define the K-shot adaptation objective:
Definition 1 (K-Shot Adaptation). A policy ग is K-shot adaptable in a distribution p(M) over MDPs if, for any Mi ~ p(M), л becomes optimal in M; after at most K episodes collected from Mi.
Finding these K episodes directly can be difficult. Instead, we propose the following intuition: if policy is K-shot adaptable in a task M₁ when using the QK episodes in B, then Definition implics that B contains
With slight abuse of notation, use [N] to mean the sequence [1, 2,2,.... N] for any integer N.

---

Published in Transactions on Machine Learning Research (04/2026)
the K episodes required for K-shot adaptation. More precisely, there exists meta-episode B, composed of these K episodes. We create a practical algorithm from this idea by making the following assumption on the relationship between B and Bj (see Appendix A.3 for a more formal discussion):
Assumption 1 (Linear Task Reconstruction). Let B RQ contain QK episodes collected from an MDP M2 such that B is sufficient for K-shot adaptation. Purthermore, let Bj E I x be the slice of B that contains these K necessary episodes. Then, we assume there crists a tensor CE RQ such that
B
(5)
where the operation denotes the batch matria multiplicationn cy R B gus : for all E [Q].
One reason for keeping this assumption linear, besides simplicity, is to create an information bottleneck for task reconstruction. While an expressive enough non-linear reconstruction could reconstruct from any (non-informative) meta-cpisode, our lincar transformation ensures that successful reconstruction relics the content of B
We leverage Assumption to measure how effective a given meta-episode Bj is for K-shot adaptation, quantifying the potential of computing a task context vector e from B, instead of B. We refer to the process of probing the linear relationship between B, and B by searching for coefficients C as task-rrconstruucionn To find a suitable tensor C, we define the task-reconttiuuttion loss
Li-rec = =E Minp((M), B.paloa (j~U(Q) [MSE (B,C, B)],
(6)
P
where U(Q) is uniform distribution over the integers 21,2,.... ... Q), MSE is the mean squared error, and C is computed from B and B, using encoder g (see Sec. 3.4. By minimizing this loss, the agent learns to find C, which is then used to assess the quality of Bj.
Recall that our iltimate goal is to train a function fu that can reliably guide n explore to collect an informative meta-episode B, during online task exploration. While the linear map C assesses the quality of By, a critical limitation that cannot be computed during online exploration, where the agent limited to collecting a single meta-episode. Specifically, computing equivalent to finding an approximate solution to the system of linear equations C B, which requires access to the full batch B.
offline "ground truth" metric that supervises the learning of fu- To this end, we aim to optimize fu to create a latent exploration space that acts as an online proxy for the information contained in C. The first step is to define a target 6y that quantifies the effectiveness of the linear reconstruction Eq. for a given meta-episode B,. We design 8, such that 8, ss 0 implies B, is a good approximation B:
Our solution is to treat C strictly
8, = 11=1een exp (-((B.C-B)²)
where & is constant and the (.) operator denotes the element-wist mean of a tensor.
We use this target to shape the latent exploration space T = (B,). By optimizing fa such that the cosine distance between episode representations T., and T., v is approximately &g (for k k E [K]), we encode reconstruction quality directly into the latent angles. Therefore, collecting a meta-episode where all vectors in { are approximately orthogonal becomes equivalent to searching for a B, that closely approximates B. This property is important because it enables K-shot adaptation, whereas the alternative approach of collecting meta-episodes is impractical in online settings.
Regardless, Eq. 5 and our entire framework naturally extend to non-linear settings by replacing the linear map with parameterized non-linear function. This change should not compromise learning, provided that the expressivity of the non-linear transformation carcfully tunod. However, find that the lincar approximation is sufficient for effoctive task reconstruction in the domains evaluated in this work, while also being simpler and computationally cheaper. For the sake of completeness, provide formal definition of the non-linear extension Appendix C and an ablation study Appendix G.T
7

---

Published in Transactions on Machine Learning Research (04/2026)
To train fu such that the reconstruction score 8, is explicitly encoded into the angles between episode representations, we introduce the loss
Lconte = =E MM)Dp(M)), PN Preaalrre ((U((Q) [MSE (Sc A(,)))
(8)
where r r= Ja,)),
and A())) E RKKK is the matrix with 6, on the off-diagonals and oncs on the diagonal. Eq. is a form of contrastive learning for RL (Eysenbach et al., 2022 Erraqabi et al. 2022), where F and :. are pushed apart or pulled together according to the soft similarity signal 0z. Since this contrastive objective operates over cosine similarity, pushing dissimilar representations corresponds to maximizing the angle between them, while pulling corresponds to aligning similar representations by minimizing the angle.
Through this training regime, the encoder fu is optimized to be used as described in Sec. 3.2 Specifically, it can train an exploration policy x expllore to find an informative meta-episode Bj (denoted as D(K) during online task exploration).
3.4 Architecture and Optimization
Having defined the learning objectives for task exploration and reconstruction, we are now in a position to describe the architecture and optimization of the encoder used to obtaín the latent task context vector c.
To accommodate both sequential online task exploration and offline batch learning, we model g as a dual- transformer architecture Its first component is fu: the online encoder that separates the informative meta-episodes from the non-informative ones collected during task exploration. Since fu is constrained to encode meta-episodes timestep by timestep, as they are being collected, we model it as unidirectional transformer with parameters Wa-
To create more informative contexts once a set of data has been collected, we leverage the second component of g: a bidircctional transformer fo of size dmodel with parameters wy. Given a tensor B of meta-episodes, fo computes two latent representat ions offline: (z, Z) fa wa). On one hand, vector z E (Rdmaddl captures the shared task structure across distribution p(M). To achieve this, z is computed from a batch of B-like tensors collected from different tasks (sce Appendix A.4.1 On the other hand, tensor Z (RX(HK))ddalla is a lask-specific representation of M,, with Z, E IR ( dmmddl encoding mela-episode Bj, for any j E [Q]. While Z and r appear to have similar roles, we note that I is optimized to learn representations that are only useful for task exploration. It may therefore fail to capture structure that is important for solving tasks, but irrelevant for exploration.
For given meta-episode B,, we compute the final context vector c E R Kdduesa)) as a function of z, Zj. and r Specifically, for function hw,, with parameters wh, we encode B, into
c= gB (B,) hww, (),
(9)
where r fuB wu). and Iw has the concatenated parameters w Wy @ Wp wh. In practice, h consists of two attention layers. Our encoder architecture is shown in Fig. 3
To train we attach two pre-training heads (shown in Fig. 4 The task-reconstruction head /t-re,, param- eterized by Wi-ree, computes the coefficient tensor C = fue-rec 9 (B) gw (B,); wi-ree) required for the low-rank approximation Eq. t The reconstruction head /rec; with parameters wree; used during the optimization of Iw to reconstruct data from corrupted inputs. Specifically, dree projects the context e computed from masked mela-episodes back to the input space, attempting predict. the masked timesteps, which is
"Due to overlapping terminology. LaSER can be seen performing in-context learning in two senses. From the mcta-RL perspective, LaSER uses in-context policies conditioned contextual information. From a different perspective, these context vectors are computed using transformers, which, due to their self-attention mechanism,
considered to be in-context learners. 3The reader might note that the ) context vector does not nevessrrlly reduce the size of the input By. A higb-dimenssoonal is beneficial since preserving the structure of B, (i.c., the HK timestops) makes it simpler to train transformers and fo. Despite this, is still uscful representation of B,. Each of the HK output timesteps has been computed from its corresponding imput timestep and the rest. of the input sequence. This enables dimensionality reduction later on. Specifically, we the task policy n reduce to Umodar-dimense
vector.
8

---

Published in Transactions on Machine Learning Research (04/2026)
e
Attention Madule
Unidirectiona
(()
Meta-episode B (t)
Encoder
Bidirectional Transformer
Unidirectional Transformer fu
Episade
Eplsode
Hybrid Encoder g
B
P
explore
p
Mi
(a)
(b)
Figure 3: Two types of meta-RA RL encoders. (a) Unidirectional encoder, common in meta-RL. It processes task exploration data online, step by step, as it is being collected. (b) LaSER's encoder g. We enhance standard architectures by adding bidirectional encoder. Encoder can be used online for task exploration or oflline to compute context c(t) We use (0) to differentiate between meta-episodes and representations belonging to different tasks. Note that we define timesteps of to contain the current state sf but the previous action at and reward rf-1, with of (st._.)) This simplifies the training of unidirectional encoders.
common approach for pre-training bidirectional transformers. Note that both he-ree and hrec are only used during mela-training For environment interactions, these heads replaced by policies explore or π.
Reconstructed Input
a
Reconstruction
Reconstruction
Exploration Palicy
Task Policy
(cktt-1)
(a)
(b)
Figure LaSER's pre-training and policy heads. (a) resosrioonsii head Ni-ree and reconon head bree; used for pre-training encoder g for task exploration and task learning, respectively. (b) Policies explore and n., used for task exploration and task solving, respectively.
We train 9, using masked self-supervision (Devlin et al. 2019 Lewis et al. 2019 The encoder learns useful representations by adding noise to the input, reconstructing the original input, and optimizing a roconstruction loss Lrec. We extend this idea and optimize the entire architecture using the loss
LLASER (w, Wree; wrree) = Crec (wo Wrea) + Ce-ree (w, wi-ree) + Ccoutr (conte( + R(w),,,
(10)
where Cree; Otreee, Ccoutr, CR are weighting cocfficients, and R is regularization term. Following Piratla et al. 2020), R enforces a soft orthogonality constraint between the global representation z and the task-specific representation Zy. Intuitively, 2, and instead focus on task-specific structure. Full details regarding our architecture, composed objective, and optimization approach can be found in Appendix A.4 with specific implementation details provided in A Appendix [
encouraged to avoid capturing the global information already contained in
9

---

Published in Transactions on Machine Learning Research (04/2026)
3.5 Meta- Training and Meta- Testing
The LaSER mela-training algorithm contains two training phases, shown in Alg, 1 We first train the exploration policy explore parameterized by Ø, and the encoder Iw, parameterized by w, for Nexplore iter- ations. Note that these components are trained together because they are interdependent. We alternate between optimizing one while keeping the other fixed. In the second stage, these two components are fixed and only the task policy 2o, parameterized by 0, optimized for Neask iterations.
T
Algorithm LaSER Meta- Training
Input p(M), task distribution
Output Te, task policy; T
explore exploration policy; 9w, encoder; 2, shared component
Ø, 42, Wrec: Werree f initialize randomly for E [Nexplora] do
3:
Wree Whrre + train encoder(p(MM) a
explore explore
Iw Wheas Wh-reo)
Alg. 3
train exploration craaiitioooooooyy(((( (p(ic T
99w)
Alg.
end for explore
Mi Mi p(m))
B
B
M.
Collect a batch of data from multiple tasks
7: 2, [b(B; wp)
8: for n E [Ntakk] do
t- train task (olicy(p(M)) To, T Iw: 2)
explore
Alg. 5
10: end for explore
11: return TT, T :9w= 2
Recall that z captures global information, so it requires data from batch of tasks, which is not available in the online phase. Therefore, we compute a fixed & from the pre-training data and use it to train the task policy and perform meta-testing. With a slight abuse of notation, we then use gw (D(K), 2 hwn (&, Z I) denote Zj and T are implicitly computed from DIK), while á is given separately. Alg. shows the meta- testing phase where, for any M; ~ p(M), the meta-trained agent first collects K episodes and computes the latent context Then, it uses to find the optimal policy for M,
Algorithm 2 LaSER Meta-Testing
Input p(M), task distribution;
ma, task policy; n
explore exploration policy; gar encoder;
2, shared component
1: for M₁ ~ p(M) do explore D(K) PM;
Sample exploration meta-cpisode
3:
c (D(K),
pro(a)s.c) T~ PM.
Sample exploitation episode
Measure return in
end for
4 Related Work
Meta-RL. The earlier successes of modern meta-RL start with MAML Finn et al. 2017 and RL2 Duan
ct al. 2016: Wang et al. 2016a). MAML, together with other MAML-inspired algorithms (Sung et al. 2017 Li et al. 2017 Gupla et al. 2018 Zintgraf et al. 2019), are gradient-best -baseddttbbsddd methods for mela-training policies that adapt to new tasks by taking a small number task-specific gradient steps. Gradient based approaches usually implement an explicit dual-loop algorithm that follows the standard mota-lcarning paradigm of adapting tasks in an inner loop, while meta-learning adaptation strategies an outer loop. On the other side of the spectrum, RI,2 is in-context meta-RL. approach, meta-training a policy to behave like RT.
10

---

Published in Transactions on Machine Learning Research (04/2026)
algorithm that learns from collected task contexts (Laskin et al. 2023 Mocini et al. 2025 This learning is usually represented by a forward pass through the meta-trained policy, with no task-specilic weight updates. Recent meta-RL techniques, including ours, follow this paradigm of first identifying task dynamics and then adapting a task-agnostic policy to them. Generally, in-context learning may lead to higher meta-testing sample-effciency than gradient-based methods, which is ideal in few-shot adaptation (Beck et al., 2023b).
In-Context Policies. To learn task-specific policies, RL uses recurrent architecture that implicitly conditions the policy on task history. Later works make this conditioning more explicit (Rakelly et al, Zhou et al. 2019 Zintgraf et al. 2019). Specifically, they show that in-context policies can be trained using standard RL optimization by simply augmenting the input state with task contexts. Similar idcas have been explored in the closely related area of unsupervised represenlation learning for RL [Ig et al. 2018 Papoudakis et al.) 2021 Botteghi et al., 2025). This approach has since become standard in meta- RL, with subsequent research focusing more on learning informative task contexts than on novel architectures or optimization algorithms for in-context policies. Recently, [eukmann et al. (2024) observed that this simple method may struggle when there is high variation across the optimal (ask-specific policies, which can arise in complex task distributions. As an alternative, [Beck et al. (2023a) and Beukman et al. (022)) propose that meta-training hypernetworks (Ha et al., 2017) may lead to better generalization. Therefore, they introduce hypernetworks that take the task context as imput and generate the weights of a context-dependent policy, While this architectural change may better leverage task contexts, their approach is still limited by the use of standard RL optimization, which constrains task space exploration. Additionally, this method inherits issues related to hypernetworks, such as slow and unstable training (Ortiz et al. 2023 Chauhan et al. 2024 Beukman et al. 2024).
2019
In contrast, our proposed reta-reward is not an adaptation of standard learning methods to meta-RL, but is explicitly designed for the meta-RL setting. Moreover, it is architecture-agnosticc, introduces little overhead, and allows in-context policies to be optimized through standard RL. This may simplify solving meta-RL problems, as practitioners can rely on stable and well-understood RL algorithms.
Exploration in Meta-RL. A considerable body of literature also focuses on exploration in mela-RL. As opposed to standard RL, exploration strategies for few-shot meta-RL can be learned from interactions with the environment and then applied to new tasks. Since identifying and solving RL tasks requires exploration, all mela-RL algorithms learn to explore, at least implicitly. However, several works have shown the benefits of explicitly learning to explore. [aaellly et al.] (2019) use posterior sampling to explore. Zintgraf et al. (2021b) consider Bayes-optimal policies, which optimally trade off exploration and exploitation, and meta- learn approximations of such policies. They later extend their work in Zintgraf et al. (2021c) by encouraging the agent to explore novel hyper-states during meta-training. Some other approaches make exploration more efficient by structuring the exploration space through contrastive learning (Fu et al.] 2021 He et al. 2024 2024 information gain (Liu et al.) 2021 Jiang et al., 2021 Zhang et al. 2021). Similarly, Chu et al. (0244) construct an exploration space through task clustering, enabling exploration via divide- and-conquer strategy of first identifying a task's 's cluster, then the task itself. Gradient- based meta-RL can also explicitly learn to explore. Gupta et al.] (2018) explore by adding structured, meta-learned noise to the policy. Similarly, (Stadie et al./ (2018) enhance MAML and RL2 by adding an exploration term to the meta- RL objective. Gurumurthy et al. 22020) add self-supervised objectives to lower variance during exploration. Finally, several of the works discussed improve exploration even further by docoupling the exploration and lask-solving policies (Zhou et al. 2019: Gurumurthy al. 2020 Liu al. 2021 Fu et. al. 2021 Chu el al., 2024h. Norman & Clune (2024) train such a decoupled policy to meta-learn exploration strategies that know when to sacrifice environmental rewards during exploration to maximize returns during exploitation.
Yu et al.
LaSER also collects data using a decoupled exploration policy and then meta-learns a structured exploration space. However, it uses a novel objective that encourages the collection of a single meta-episode, which serves as a low-rank linear representation of a larger dataset drawn from the same task. An important distinction to numerous previous works is that task exploration depends only on the structure of the data, while being agnostic to the RL objective of the task policy. This may be advantageous for out-of-distribution adaptation, where meta-test tasks may have goals that differ from those seen in meta-train tasks. In such scenarios, data collected for maximizing meta-training return might not always be relevant during meta- meta-testing.
11

---

Published in Transactions on Machine Learning Research (04/2026)
Norman & Clune (2024) is an example of recent work that, similarly to us, attempts to meta-learn structured task exploration. A critical difference, however, is their limitation in collecting multiple exploration episodes effectively. While LaSER explicitly optimizes policy that avoids redundancy within a meta-episode, "First- Explore does not actively explore to enable future exploration" NNoman & Clunc| 2024). Consequently, LaSER is primarily designed for settings with an exploration budget of K > 1, where information from early episodes enables more effective or diverse exploration in subscquent episodes.
Meta-Learning Contexts. Once task data has been collected, a straighttorward way to obtain informative contexts is through recurrent neural networks (Dan et al. 2016 Wang et al.) 2016aa. More sophisticated methods include meta-lcarning latent representations of value functions (Rakelly et al. 2019) or MDP dynam-
ies Zhou et a 2019 Zintgraf et. al. 2021b c), model-based meta-RL (Clavera et al. 2018 Nagabandi et al.
2018), hybrid techniques that combine in-context and gradient- based methods (Imagawa et al.) 2022). using permutation variant and invariant sequence models (Beck et al. 2024); or enhancing tasks by incorporating language instructions (Bing et al.) 2023). Attention mechanisms (Bahdanau et al., 2015) and transformers (Vaswani et al., 2017) have also been adopted by the meta-RL. community. These architect success in in-context learning, long-sequence modeling, and efficient parallelizable training aligns well with the needs of in-context mcta-RL. Earlier rescarch focused solely on meta-learning through attention (Mishra et al. whereas more recent work used transformer architectures (Melo 2022 Xu et al. 2022b Lee et al, Shala et al. 2024 Grigsby et al. 2024). Moving beyond architecture, Zhou et al. (2025) emphasize the importance of measuring the reliability of task contexts to avoid ambiguous or out-of-distribution contexts. The scope of their uncertainty estimator shares similarities with our goals of measuring data quality through task reconstruction. However, while their approach is built. for offline meta-RL for one-shot adaptation, we tackle the online fow-shot adaptation scenario.
2018
2023
A limitation of previous works that attempt to meta-learn task dynamics is that only unddirectional encoders are used. This constraint arises naturally since task exploration and task learning are coupled. Specifically, task data must be encoded online while it is being collected, in order to guide the exploration policy at the next limestep. An obvious limitation is that only interactions between the current and past timesteps are considered. LaSER improves this design by using an additional bidirectional encoder that also considers future timesteps, which could possess useful information and lead to richer representations (Devlin et al., 2019 Banino et al. 2022p. While the aforementioned constraint cannot be avoided during task exploration, it need not restrict the computation of the final task context once all exploration data has been collected.
Another critical distinction between LaSER and prior work lies in the structural assumptions. Many meta- RL algorithms rely on the implicit assumption that task dynamics can be effectively captured simply by encoding interaction histories into a latent variable via the agent's learning mechanism (e.g., recurrence, variational inference, attention). However, without explicit structural constraints, it is unclear whether these unstructured representations can generalize to the entire task distribution apply to new tasks. Additionally, these representations often optimize for state reward prediction, without encoding explicit signals to aid task exploration. In LaSER, we instcad make an explicit low-rank linear reconstruction assumption (Assumption [] The assumption that a small set of basis episodes is sufficient to linearly reconstruct any data collected from a specific task leads to representations that provide a more directed exploration signal. Specifically, LaSER's exploration objective is reduced to finding these basis episodes, leading to a policy that can safely ignore non-basis episodes.
5 Experiments
In this section, we present empirical results for LaSER. We first introduce the environment and algorithms we use in Sec. 5.1 In Sec. 5.2 we compare LaSER with other types of meta-RL algorithms on multiple benchnarks. Next, we perform two ablation studies by analyzing individual stages of the meta-RL pipeline (Fig. D In Sec. 5.3 we evaluate our novel approach to meta-training in-context task policies. This cor- responds to LaSER's task-solving phase, which we perform using ground-truth contexts instcad of task exploration and learning. Finally, we assess LaSER's task exploration and task learning stages in Sec. 5.4 by visualizing the latent task contexts computed during meta-testing.
12

---

Published in Transactions on Machine Learning Research (04/2026)
5.1
Experimental Setup
5.1.1
Environments
We evaluate LaSER on the meta-RL. benchmarks MEWA (Stoican et al. 2023), Meta- World YYu et al. 2020:
McLean et al. 2025). and MuJoCo HopperMass (Rothfuss et al. 2019 Zhou et al., 2019 Nakhaeinezhadfardd ct al. 2025 We briefly discuss these environments here, focusing especially on MEWA. We then provide more details in Appendix including the publicly available code used to implement each benchmark.
MEWA provides a distribution of tasks that share the same central idea: certain states, called critical states, can Icad to mistakes. These mistakes affect the final return negatively. The probability of mistake happening depends on both the type of critical state and the task's dynamics. For each task, meta-RL agents must find the optimal policy for avoiding high-risk mistakes while minimizing delays. Importantly, [Stoican et al. (202)) ensure MEWA's 's task distribution has no globally optimal non-adaptive policy (i.e., no policy can zero-shot solve all tasks). Therefore, the optimal agent must collect new data and adapt. This property makes MEWA ideal for our case, allowing us to evaluate LaSER's 's task exploration ability.
MEWA evaluates agents on their ability to take optimal actions in different types of "critical" states. These critical states provide agents with two options. Consider a critical state s, of type z. The agent's first option is take a "risky" action. This may lead a mistake of type happening, which in turn leads to a large delay in task completion. The probability of type mistake happening depends on the transition function of the lask. The second option is a "safe" action. This provides guaranteed small delay and leads to a state 8x-1 of type x. An important feature shared by all MEWA tasks is that for any < mistakes of type y are less likely to happen than mistakes of type x. Therefore, depending on the task, taking several safe actions before a risky action could be optimal.
We meta-train all meta-RL agents on the narrow task distribution analyzed by SStoccan et al. (0023). This corresponds to a distribution p(M) in which any task Mi ~ p(M) has four different types of critical states and can be described by vector ( E [0, 1]a. For each task Mr, the probability of making mistake of type x E [4] is pa ~ (N,, 0.12), where A is a normal distribution and 2³ = [0.38, 0.28, 0.19, 0.09). Note that we clip all pa to [0, 1]. See Appendix E.1 for more formal definition of MEWA's 's tasks.
(e)
We meta-test on the fixed set of 12 tasks proposed by Stoican et al. 2023).. Their corresponding p() vectors are listed in Appendix F.1 Table 4 We assign a score to each task, given by the average probability of a mistake of any type happening, i.e., 1/4 p." We use this as rough estimate of the similarity between tasks, such that tasks with a similar score require similar (but not identical) optimal policies. Additionally, note that half of the meta-testing tasks are out-of- distribution (OOD) tasks. OOD tasks lie outside the meta-training distribution p(M), so they are more challenging to explore and solve during meta-testing.
Note that no globally optimal non-adaptive policy exists for these 12 meta-testing tasks. Additionally, the difference in performance between the optimal non-adaptive and optimal adaptive policies is high enough to evaluate the agent's 's ability to generalize and adapt.
MEWA offers 3 baselines to compare our agents to during meta-testing. The random baseline represents the expected performance of an agent that takes uniformly random actions. The task-agmostic baseline represents an agent that always takes optimal actions in MEWA's non-critical states. These are states in which the optimal action can be computed even if the policy has no task information, ie., for the optimal value function V*, if (*(s,c) S V*(s) for all c, then is a non-critical state. For any critical state, the lask-agnostic baseline considers all actions that could potentially be optimal in a valid task, then takes one at random. Finally, the optimal baseline gives the expected return of the optimal meta-RL agent, i... the agent that optimally adapts to and solves each task.
To assess the generalixability of our algorithm, we also conduct experiments on Mera-World V2 ML10 and MuJoCo HapperMass. Meta-Wast World is a suite of 10 meta-training and 5 meta-testing robotic manipulation tasks where the goal is hidden and must be meta-learned. To avoid reproducíbility issues, we follow McLean et al. (2025) and use the V2 reward function. MopperMass the meta-RL extension of the popular MuJoCo Hopper environment (Toooovv et al. 2012). The agent's 's goal is to rapidly move forward, taking into account
13

---

Published in Transactions on Machine Learning Research (04/2026)
task variations given by multiplying the body parts' mass, the joints' damping, and the ground friction with weights uniformly sampled from [1.5-1, 1.5"). To assess out-of-distribution (OOD) generalization, we follow Nakhaeinechadfard et al. (2025) and extend HapperMass to OOD meta-testing tasks with weights in 1.."
[1.5- 1.5 ,.5
In some environments, such as HopperMass, low exploitation returns during task exploration may lead the LaSER agent only "surviving" for small number of timesteps, hindering effective data collection. To avoid exploratory episodes terminating prematurely, we let LaSER's rewards for episode k be
pk F + CppT
(11)
where P is the intrinsic exploration reward defined in Eq. 4 (zero everywhere except when is a terminal timestep), re is the environment. reward, and or is a constant weight. For MEWA and Meta-World, we set 0r, 0. In HopperMass, we tune a, such that ra does not dominate P/.
5.1.2 Algorithms and Meta-Training Setup
We compare our method, LaSER, with five other meta-RL. RL algorithms, MAML (Finn et al. 2017). PEARL Rakelly et al. 2019)) VariBAD (Zintgraf et al.) 2021b). DREAM (Liu et al. 2021), and First-Explors Norman & Clune 2024). For each benchmark, we meta-train the algorithms on tasks sampled from the meta- training distribution. For each task M,, the agents are allowed to collect K episodes during the task exploration phase. For both MEWA and HopperMass, we use 3, 0,000 tasks with K 4, while for Meta- World, we use the 10 lasks provided and set. K 2. The agents' performance on M, is then given by single exploitation episode collected by the task policy. Besides this final return, we also analyze agent's ability to improve its performance more data is collected. MEWA tasks have a horizon of H = 50, Meta-World has H = 200, and HopperMass has H = 400. All of the results we present are averaged over 10 seeds.
We meta-train LaSER in two decoupled phases. Following Alg. 1 the exploration policy a explore and encoder IN are first trained for Nexplore iterations. The task policy Ty is then trained separately for Nusk iterations. Note, however, that for the first Avinitial out of Nexplore iterations, we only train fu (and thus gw), without explore since its training is conditioned on representations learned by fu- For these initial updates, we use data collected by a uniformly random policy.
updating T
LaSER's hyperparameters are listed in Appendix H. We selected these based on performance on the meta- training task distribution of each environment. For implementation details and hyperparameters for MAML, PEARL, VariBAD, DREAM, and First-Explore, see Appendix F.
Note that we perform our primary in-depth analysis on MEWA, as its design ensures that task exploration and adaptability are strictly necessary. In contrast, while Meta- World and HopperMass are important benchmarks, their focus is not on exploration. Therefore, their primary role in our work is to demonstrate the general applicability of our algorithm to meta-RL problems.
For the latter two benchmarks, we select only VariBAD as a baseline for two primary reasons. First, prior litcrature has shown VariBAD consistently outperforming MAML and PEARL in continuous control environments (Zintgraf et al. 2021b Demonstrating improvement over VariBAD is therefore sufficient. Second, as indicated by Liu et al. (2021) benchmarks such as Meta-World and HopperMass have not been explicitly designed to test complex task exploration policies, as naive exploration can often be sufficient. Because of this, Liu et al. (2021) highlight that DREAM is not engineered for such environments and is not necessarily expected to outperform simpler baselines. We follow a similar line of reasoning and omit Fxpst-Explore as well, not unlike Norman & Clunc 2024), who also do not consider this type of benchmark.
5.1.3 Complexity and Trade-offs
We analyze the computational complexity of meta-training cach algorithm on MEWA. Specifically, we mea- sure the total wall clock time required to collect environmental transitions and optimize all components of an algorithm. We additionally report the total time divided by the number of timesteps collected, showcasing
14

---

Published in Transactions on Machine Learning Research (04/2026)
the average cost of interacting with the environment once and updating based on that interaction. Note that all seeds of an algorithm are meta-trained for the same number of timesteps. This duration is set by the timestep at which the last seed's performance (i.e., return or loss) stops improving,
We provide detailed measurements, averaged over 10 seeds, in Fig. 15 Regarding total time, LaSER is the most expensive, being 1.34 times slower than the next slowest algorithm, DREAM. This is followed by PEARL, and finally by MAML, First-Explore, and VariBAD. Most of LaSER's time complexity comes from its transformer encoders, which are powerful but require high amount of data. However, LaSER appears more efficient when collecting single timestep and updating. Specifically, LaSER is 11.93 and 5,86 times faster than DREAM and PEARL, respectively. Furthermore, it is only 1.71, 2.31, and 2.64 times slower than MAML, First-Explore, and VariBAD, respectively.
To understand the architectural complexity of LaSER, 3 the number of trainable parameters to those of the bascline algorithms. While LaSER has notably more parameters, approximately 90% of these belong to the two pre-training heads, hrec and ht-rec- Since these are not used beyond the pre-training phase, LaSER agents only require approximately 8 million parameters during meta-testing and task policy meta-training. LaSER's meta-testing parameter count therefore falls between First-Explore and DREAM, with roughly? 2 million fewer than the former and eight times more than the latter.
LaSER's high number of parameters is a consequence of the transformer encoder g, whose additional com- plexity enables more effective in-context computations. Moreover, since the context vector с outputted by 9 is also higher-dimensional than the ones used by PEARL, VariBAD, and DREAM, LaSER's task policy requires a larger network for extracting features from c. Besides this feature extractor, LaSER's policy networks comparable in size to those of PEARL and VariBAD.
Note that, while scaling up the encoders and the latent context spaces of the baselines could arguably lead to fairer comparisons, this is generally non-trivial and impractical. Unlike LaSER's transformers, architectures based on recurrent neural networks or variational auto-encoders (Kingma & Welling 2014) suffer from instabilities and optimization difficulties when scaled to millions of parameters. First-Explore is the only baseline that uses a transformer encoder, so we set its size match LaSER's meta-testing architecture.
While LaSER has higher computational cost than the baselines during meta-training, this trade-off proves advantageous during meta-testing, By shifting the computational burden of learning general representations and exploration strategies for an entire task distribution to the meta-training phase, the meta-trained agent can reduce the amount of data required during online adaptation. While other meta-RL algorithms also attempt to offload computation to meta-training, their smaller or simpler architectures or their coupled approach to learning task exploration and solving may limit their ability to scale. In contrast, LaSER's decoupled paradigm can leverage large quantities of meta-training data to learn complex exploration priors.
Given the design of LaSER, we find it necessary to draw attention to the scope of our evaluation. Specifically, it is important to distinguish between LaSER's high meta-training compute/daa requirements and its highly sample-cfficient K-shot adaptation during meta-testing.
5.2 Results and Analysis
5.2.1
TrtiaaTninnng Convergence on MEWA
To ensure a fair evaluation, we first show that all algorithms converge during meta-training. For this, we evaluate agents on MEWA's mota-training task distribution. As shown Fig. LaSER's exploration policy, encoder, and task policy converge across 10 seeds. The encoder loss is computed by Eq. [0] with Fig. 10 showing results for each of its terms. The exploration return is computed using intrinsic rewards (see Eq. collected by the exploration policy. Note that we first train the encoder and exploration policy in paralle! for 220 and 170 million timesteps, respectively, then the task policy for 7.5 million timesteps.
Appendix G provides convergence results on MEWA's meta- teita-training distribution for all the baseline al- gorithms. All their task policies converge (Fig. 11. togother with PEARL's, VariBAD's, DREAM's, and First-Explore's encoders (Fig. 12), and DREAM's exploration policy (Fig. [3]. Note that MAML lacks an encoder, and DREAM is the only baseline am that uses separate exploration policy with an explicit
15

---

Published in Transactions on Machine Learning Research (04/2026)
0.90
Average Exploratios Reetmm
0.8
Average Encoder Lass
0.14
0.7
Average Retum
0.85
0.12
0.6
0.10
0.80
0.5
0.08
0.75
0.06
0.70
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
Timestep
166
108
Timestep
Figure 5: Performance on tasks from MEWA's meta-training task distribution. For each metric, the shaded areas report the standard error of the average return across 10 random seeds. We use exponential moving average (EMA) smoothing y/ o + (1 (VEMA)uj- for each point Ijy with OEEA 0.6. We first meta-train the exploration policy and the encoders for 2.2c8 timesteps, then the task policy for 7.5e6. Additionally, only start exploring tasks after 50 million timesteps. (left) Undiscounted exploration return explore for intrinsic exploration reward f. (middle) Encoder loss LLaSER- (right) Undiscounted of policy To return of policy me.
intrinsie reward. While First-Explors also has an exploration policy, its optimization is end-to-end via the encoder's loss.
Finally, to verify the validity of our linear task reconstruction assumption (Assumption 1 we train variant of LaSER (on MEWA) that utilizes non-linear transformations to construct its latent exploration space, as formally introduced in Appendix C We analyze the results of this ablation study in Appendix G.1 The non-linear variant achieves a lower task reconstruction loss, which is to be expected, as non-linear mappings are more expressive than linear ones. Crucially, however, this does not translate to an improvement final task performance. This indicates that identifying low-rank linear basis for task's meta-episode space is sufficient, least in the MEWA environment. Moreover, competitive results Meta-World (Sec. 5.2.3) suggest that the effectiveness of our linear assumption may also extend to enviromnents that are highly non-lineat and continuous.
5.2.2 Meta- Testing Performance MEWA
We now evaluate LaSER's ability to solve MEWA's meta-testing lasks. For each algorithm, we mela-test the agent obtained at the end of meta-training. Note that before being evaluated on task, each agent is allowed explore for K episodes. We report average returns over 10 seeds in Fig. 0 and Table [ Because of the inherent randomness in MEWA's tasks, we take an additional step to ensure the results are not due random chance. For each seed, we repeat the entire meta-test. (i.e., both task exploration and exploitation) 9 times per agent for each of the 12 meta-testing tasks. For each repetition, collect 20 exploitation episodes, using the same context vector, and then average over their returns. In addition to the results averaged over all tasks, we report per-task performance in Fig. 14
LaSER (ours)
MAML
PEARL
VariBAD
DREAM
First-Explore
0.8 + 0.021
0.43 + 0.072
0.67 + 0.32
0.08 + 0.262
0.6 0.191
0.52 + 0.018
Table 1: MEWA returns averaged across 10 seeds (+ std).
LaSER achieves the highest average returns among all meta-RL algorithms. It also proves to be the second most stable algorithm, with a post-adaptation standard deviation of just 0.021 across all seeds. LaSER is one of only two methods to ourperform the lask-agnostic baseline, while the other algorithms only outperform
16

---

Published in Transactions on Machine Learning Research (04/2026)
1.0
Random
Task-Agnostic
0.8
Optimal
Averoge
0.4
0.2
LaSER (ours)
MAML
PEARL
Var BAD
DREAM
Aist-Explore
Figure Average returns achieved on MEWA's 's meta-testing tasks. Each agent meta-tested with an exploration budget of Results are averaged over 10 seeds, with error bars indicating standard error. Note that normalize results between the random and optimal baselines.
the random baseline. PEARL learns a strong global policy, but is loss stable and underperforms compared LaSER. While DREAM appears more stable, its performance is slightly below the lask-agnoslic baseline. First-Explore underperforms DREAM, but it has the lowest standard deviation, slightly below LaSER's. In contrast, MAML and VariBAD perform poorly, with VariBAD being both unstable and close in performance to the random baseline.
5.2.3 Meta- Testing Performance on Meta-World and HopperMass
In this section, we evaluate LaSER on Meta- World and HopperMass. As detailed in Sec. 5.1.2 we omit MAML, PEARL, DREAM, and First-Explore in this setting and compare only with VariBAD, the strongest bascline applicable to these environments.
Fig. (left) shows that LaSER achieves competitive performance with VariBAD on Meta-World ML10. For meta- teata-trannin budget of 50 million timesteps, we present the average success rate on the ML10 meta-testing tasks, mcasured at intervals of 4- 10 timesteps.
Note that for LASER, the encoder and exploration policy have been pretrained for Nexplore iterations be- fore starting the task policy training. Therefore, the learning curve of LaSER shows the post-pretraining optimization of the task policy, measuring how quickly the agent learns to solve tasks given pretrained task exploration prior. On the contrary, VariBAD learns from scratch. We chose this type of comparison to explicitly measure adaptation efficiency and maximum asymptotic performance, rather than total sample complexity across the entire meta- veta-traiiing phase.
LaSER appears to have higher meta-training sample-efficiency initially. However, both algorithms end up performing similarly on Meta-World, as suggested by the error bars that largely overlap throughout meta- training. Overall, LaSER's adaptation efficiency and asymptotic performance during meta-training are comparable to VariBAD's, with both agents reaching maximum performance of approximately 0.11.
As shown in Fig. 7 (right), LaSER performs slightly better than VariBAD in HopperMass in terms of OOD performance. We meta-train each agent for 10 million timesteps and measure OOD performance at intervals of 40, 000 timesteps. As before, LaSER's encoder and exploration policy are first. pretrained for Nexplore iterations.
LaSER is more stable and adapts better the HopperMass OOD tasks than VariBAD. Specifically, LaSER achieves maximum average performance of 540, while VariBAD stops improving at 496. Additionally,
17

---

Published in Transactions on Machine Learning Research (04/2026)
Meta-Worid ML10
HopperMass V5
0.150
VariBAD
Average Successs Rate
0.125
LaSER (Ours)
500
0.100
Average Retur
400
0,075
300
200
0.025
0.000
0.2 04
0.6
0.8
Timestep
167
Timeslep
167
Figure Meta-testing performance, averaged across 10 random seeds (+ standard error). We use EMA smoothing with CEEM = 0.1. (left) Average success rate Meta-World ML10 meta-testing tasks for an exploration budget of K = 2. We measure performance every 4 10 timesteps. (right) Average return on HopperMass OOD tasks for K = 4, mcasured at intervals of 40, 000 thousand timesteps.
LaSER's superior stability is shown in the lower average standard error (26.3) compared to VariBAD (30.3) during the final million timesteps.
5.3 Ablation Study: Task Solving
As ablation study, we evaluate our approach for meta-training in-context task policies, which was in- troduced in Sec. 3.1 The results in this section correspond solely to the task-solving phase of Fig, [ We meta-test task policies by rolling them out before and after they receive a task context c. Our primary objective assess adaptability, i.e., a policy's s ability to use e to improve its performance. We quantify this the difference between post-adaptation and pre-adaptation return.
We make two important simplifications to the standard MEWA benchmark.
To ensure a fair comparison of task policies alone, we meta-train and meta-test without task explo- ration or task learning. Instead, for each task, we use a ground truth, oraclo-provided context vector c, which is normally unavailable the agent.
To assess adaptability to non-stationary dynamics, we use simpler, multi-task objective: we perform both mela-training and meta-testing on MEWA's 12 meta-lesting lasks.
Despite these restrictions, MEWA's guarantee for the nonexistence of globally optimal policy still holds. That is, only adaptive policies can achieve maximum return.
LaSER combines PPO and the proposed meta-reward rtt (see Eq. 2 to optimize the in-context policy we- We compare it to a simpler context-aware PPO policy, where the state st and context e are first preprocessed separately, with their representations concatenated into single vector, which is finally passed through a policy optimizod through standard PPO. This corresponds to LaSER's rtt with 3 = We also compare Decision Adapters (Beukman et al. 2024 DAs), which use hypernetworks to generate task-specifie policies D For fair comparison, the context-aware PPO policy has the same architecture also use PPO to optimize DAs, so the main difference between these and the other two methods is the architectural change. Appendix F.6 provides implementation details and hyperparameters for DAs.
LaSER's policy, We
The meta- training results are shown in Fig. 3 and Table 2
Our method achieves the highest average return while also stabilizing meta-training. is also the only method to find policy that adapts optimally, which occurs in out of 10 seeds. Context-aware PPO slightly less stable, presumably because the agent seeks
SSnuc Beukman al. (024 build and evaluate their DAs with ground-truth contexts in mind, their algorithu aguostic to the context- learning mechanism, and, as reported by the authors, sensitive noisy contexts. We therefore only consider DAs as baselines for LaSER's task policy, and not for the entire LaSER algorithm.
18

---

Published in Transactions on Machine Learning Research (04/2026)
single policy that maximizes returns across all tasks. However, without task-specific adaptation, it fails to find such a globally optimal policy. Hypernetworks, on the other hand, are highly unstable. This is likely due
to hypernetworks being inherently challenging to train (Ortiz et al. 2023 Chauhan et al. 2024 Beukman
ct al. 2024). They also achiove the lowest average return.
Policy lours)
Context-Aware
Adepter Hypernetwokk
Tesk-Agnostic
Optimal
1.0
0.5
0.9
Average Deta Return
Average Retum
0.8
0.3
0.7
0.2
0.6
0.1
0.5
0.0
0.4
-0.1
0.3
1.
167
107
Figure Task policy meta-training. Each agent uses oracle-provided lask contexts e, and meta-trains and meta-tests on the same set of 12 MEWA tasks. The shaded areas show the standard error of the average return across 10 random seeds. (left) Post-adaptation return of task policy ma, given task context c. All returns normalized between MEWA's random and optimal baselines. To improve readability, we restrict the plot to the [0.3, 1.0] range and exclude the random baseline (which is positioned at 0). (right) Adaptability, quantified as the difference in return achieved by no with and without access to c.
LaSER Task Context-Aware PPO Policy (ours)
Decision Adapter Hypernetwork
Return
0.97 + 0.006
0.93 0.014
0.86 + 0.149
Delta Return
0.47 + 0.019
0.07 = 0.067
0.12 0.164
Table 2: Average returns and delta returns the last 1e6 timesteps, averaged across 10 seeds (# std). We meta-train and meta-test the same 12 MEWA tasks, and use oracle-provided context vectors
The LaSER task policy also achieves greater task aask-adapt success than the other two methods. After the meta-cxploration phase, the agent stabilizes towards the end of meta-training. The task-conditioned policy then consistently outperforms the pre-adaptation policy. spperreewwrr-ggneeraee context-aware PPO but are also more unstable. Additionally, their low average return limits the benefits of this adaptability. Context-aware PPO does not explicitly optimize for adaptation, its adaptability is approximately five times lower than our method.
policies adapt better than
Note that, at the beginning, our method learns more slowly than context-aware PPO and the DA hypernet- work. That is, it requires more samples to pass the average return thresholds of 0.9 (for PPO) and 0.85 (for DA). We attribute this to additional meta-exploration in the state space, performed in the context of the task space. The meta-reward rf encourages the policy to learn how take in-context. actions that increase the gap between ((s)) and V(st, c), for a state & and context This in addition the meta-cxploration required to only maximize the standard RL objective (ss)). We hypothesize that context-aware PPO and DA cannot surpass our method due to their lack of explicit task-space meta-exploration. The policies they produce optimize standard RI. objectives, so they meta-cxplore accordingly.
We also observe that our method is nearly as computationally efficient as contexl-aware PPO and much faster than hypernetworks. We measure the average wall-clock time to roll out and optimize Ne. per iteration. On average, hypernetwork-basse agents are approximately 3.3x slower than ours, while our method only 1.3x slower than context-aware PPO. See Fig. [or for detailed comparison. We discuss how meta-rewards cause this additional overhead in Appendix A.1.2
19

---

Published in Transactions on Machine Learning Research (04/2026)
Finally, to verify that the benefit of our meta-reward arises specifically from the interaction between the context-usage weight w and the entropy regularization S (see Eq. 21, we conduct an additional ablation study. We compare our proposed objective with two simpler alternatives: a direct non-entropyy-ependeuut penalty for not leveraging contexts effectively (re- - (tt, c)), and an "always-on" entropy bonus that is agnostic to the value lunction's 8 context utilization (r, + 8 (1,)(1,,))) As discussed Appendix G.2 we find that Eq. 2 ooutperorms both of these two simpler alternatives terms of final returns, stability, and delta returns.
5.4 Ablation Study: Exploring and Learning Tasks
To better understand LaSER's ability to explore and encode tasks during meta-testing, we visualize its latent task space. We compare the task contexts computed by LaSER, PEARL, VariBAD, DREAM, and First-Explore for cach of the 12 meta-tosting tasks MEWA. For each algorithm, we show results for only one of the meta-training seeds. However, we obtain similar representations for the other seeds.
We first roll out the corresponding agent's meta-trained policy, collecting 100 meta-episodes per task, with K 4 episodes per meta-episode. Note that in the case of LaSER, we roll out the exploration policy T explore We then encode each meta-episode D(K) into a latent task context vector e using the agent's meta-trained encoder. Similarly, DREAM and First-Explore collect data using their exploration policy. We visualize two-dimensional projections of task contexts, computed using t-SNE (Van der Maaten Hinton 2008),
Fig. g The 12 tasks are sorted by similarity, i.e., average mistake probability, as in Table
LaSER (ours)
PEARL
VariBAD
Task
DREAM
Rrst-Explore
Figure 9: Latent representations of meta-cpisodes collected and encoded by meta-trained agents. Each of the 1200 points is context vector belonging to one of the 12 meta-testing tasks in MEWA. The two-dimensional visualizations are computed using I-SNE. Different tasks have different colors.
For LaSER, we find clusters of task contexts. Specifically, meta-episodes collected from the first five tasks, the next four tasks, and the final three tasks appear to be projected apart from each other. We note that the tasks in each cluster share common traits. In the first cluster, mistakes of type E (3, 4} never occur. In the third cluster, ps 0.9 for any x + 4. The second cluster's average mistake probabilitics lic between those of the other two clusters. The separation appears follow this trend. More details on the tasks' properties shown in Table
20

---

Published in Transactions on Machine Learning Research (04/2026)
In contrast, PEARL's and VariBAD's latent contexts lack clear clustering that could distinguish data col- lected from different tasks. This low separation may make it more difficult to identify task-specific features. DREAM appears to follow a similar three-cluster separation as LaSER. A distinguishable feature is that data collected from any of the first five tasks collapses into a single context reprosentation. Finally, the representations computed by First-Explore indicate the formation of only two cluster types, with contexts from tasks with mid-range mistake probabilities appearing in both. Additionally, the clusters are less densely packed than those generated by LaSER or DREAM, instead forming multiple sub-groups.
We argue that learning higher-quality clusters is not trivial. The difficulty may come from the high variance in episodes collected from the most difficult tasks. These tasks limit the exploration policy's control over the states it encounters. Additionally, the low adaptation budget of K can make it difficult to separate contexts of similar tasks.
To gain an intuition about the behavior of LaSER's exploration policy, we provide a qualitative discussion of some example episodes. We choose to focus on MEWA due to its inherent interpretability and select two of its meta-testing tasks to discuss (see Appendix G.3 for visualizations and detailed discussions).
We find that, in in-distribution tasks where cach type of mistake has moderate likelihood of occurring, LaSER's policy reduces redundancy and increases diversity across K exploration episodes. Specifically, LaSER appears inclined to generate action sequences that are distinct from those in earlier episodes or to experiment with the different types of risky actions available in the MEWA environment. In contrast, VariBAD tends to repeat action sequences and mistakes. In such tasks, the diversity in DREAM's exploration episodes is approximately on par with LaSER's.
However, we also find task setups where DREAM's exploration policy collapses, preventing full understanding of the task. For example, in an OOD task where mistakes of type 4 never occur but other types are highly likely, DREAM repeats the same action sequence for all K episodes, as this sequence leads to high exploitation returns. In contrast, LaSER avoids this pitfall and attempts to sample distinct episodes that better capture the lask context. We attribute LaSER's behavior in this setting to its meta-training objective, which explicitly encourages the agent to avoid redundancy and ignore environmental rewards during exploration.
6 Discussion
This scction positions LaSER within the broader meta-RL domain. We discuss the implications of our design and results, the limitations of our assumptions, and the potential effectiveness of LaSER in various domains.
Implications for Meta-RL. An important strength of LaSER is that, in the setting where contexts are meta-lcarnod, it outperforms other algorithms, despite none of the methods being optimally adaptive. We argue that this is a result of meta-training with higher-quality lask contexts. Note that the main role of these contexts remains to provide task information for adaptation during meta-testing. However, our results suggest they may also enhance performance and sample efficiency during meta-training. We believe this secondary role should also be investigated in future work.
LaSER is built upon the unique properties and requirements of the meta-RL. framework, which we approach through the lens of few-shot adaptation. As a result, we tackle challenges that are specific to meta-RL, i.e., challenges that might not exist in the broader fields of RL or meta-learning. For example, our exploration algorithm leverages a key assumption about the structure of data collected in few-shot RL environments. Similarly, LaSER's task policy is mota-trained with the explicit goal that in-context policies must outperform context-agnostic policies in a meta-RL setting. This idea, realized as a form of extended meta-exploration over the task context space, leads to almost-optimal adaptive policies in MEWA, when mruund-truth contexts are available. We hope our findings inspire future rescarch to leverage the meta-RL framework in novel ways.
While LaSER outperforms other meta-RL algorithms, it still struggles to adapt to new tasks when task con- texts are meta-learned. We discuss this from the perspective of LaSER's s three main components: exploration policy, encoder, and task policy. Empirical results suggest that the first two components are well-optimized for exploring and learning tasks effectively. For instance, the encoder learned to separate MEWA's meta- testing tasks into three groups This implies that LaSER should be able to outperform its pre-adaptation
21

---

Published in Transactions on Machine Learning Research (04/2026)
policy in these tasks. Furthermore, results show that the task policy can become more adaptive when restricting the meta-training task distribution and using ground truth contexts. We therefore propose the hypothesis that effectively combining these three components is non-trivial.
Scope and Limitations. From a practical perspective, LaSER's previonsly discussed shift in computational burden from deployment to meta-training (Sec. 5.1.3) offers clues to the types of domains our work could be applied to. On one hand, this shift makes LaSER particularly suited to applications where task data is scarce, due to expensive dangerous test-time interactions, while data from similar tasks can be obtained easily during meta-training. For example, sim-to-real settings can use simulations during meta-training to enable rapid adaptation during real-world deployment. On the other hand, environments with a moderate amount of data during both meta- whela-taiiing and meta-testing may see LaSER struggle. Its pre-training phase could fail to converge, while its
capabilities (if achieved) may not be necessary.
The set of applications in which LaSER expected to be effective is also dependent on our framework's contral assumption: low-rank linear task reconstruction (Assumption) D This linear relationship has proven e in environments such as MEWA and HopperMass, and even in the more complex robotic manipulation tasks of Meta-Woon World. We therefore conclude that linear task reconstruction is sufficient whenever the information contained in task can be reasonably approximated by set of K "basis" episodes. As previously mentioned, this simple low-rank approximation also acts as an information bottleneck, regularizing the model and preventing overfitting during task reconstruction.
If this linear assumption fails, then the task reconstruction loss remains high, producing a noisy target 6y. Ultimately, this means that the contrastive optimization is unable to generate meaningful geometric struc- ture within the latent exploration space. The meta-episode representation I will then guide the exploration policy to collect uninformative or redundant episodes, crippling K-shot adaptation. We suspect that such a chain of failures may occur in very complex enviromments with a high-dimensional state-action space, or where the task-specific dynamics cannot be captured by a linear basis.
For instance, in roboties domains with high-dinensional image observations, linear relationships bet ween sequences of images may not reveal meaningful task information. Similarly, multi-objective environments, where each task acts that cannot be approximated by linear basis. Long-horizon tasks could also pose challenges, as small variations in early timesteps could lead to vastly different trajectories collected from the same task. Our linear assumption may then require a significantly larger exploration budget K, defeating the purpose of few-shot adaptation.
unique combination of several goals, may exhibit exponential compositionality
To evaluate the impact of our task reconstruction method on real-world application domains, a more exhaus- tive empirical analysis of LaSER would be required. In cases where the linear assumption proves insufficient, the non-linear variant in Appendix C may act as more expressive alternative. We must, however, repeat our warning: an over-expressive non-linear task reconstruction head could trivially minimize the Li-rec loss without creating a bottleneck to capture meaningful information about the data's K-shot potential.
7 Conclusion
Summary of Contribution. We introduced LaSER, a new approach for meta-learning RL exploration. Our results demonstrate that LaSER outperforms previous meta-RL. algorithms on the MEWA benchmark and is competitive on the Meta- Meta-World and HopperMass benchmarks. They also show that LaSER can meta- learn better lask clustering during exploration. Additionally, we propose a novel mela-exploration bonus for training highly adaptive task policies. This outperforms previous approaches of meta-training in-context policies in both accumulated rewards and adaptability. Since our task-solving method is agnostic to how tasks are explored or represented, it can be integrated into any in-context meta-RL algorithm and then optimized using standard RI.
Open Challenges and Future Work. A potential extension to LaSER, tied to the specific structure of the meta-RL framework, is dynamic exploration budgets. Although we use a fixed budget K in this work, our task reconstruction objective implies that a quality metric (e.g., o5, Eq. 7 could enable an autonomous
22

---

Published in Transactions on Machine Learning Research (04/2026)
stopping criterion. This extension would have practical utility, allowing K to be dynamically allocated task-by-task basis based rank and complexity, instead of being selected by the algorithm designer.
As mentioned in our discussion, optimizing the individual components of the standard in-context meta-RL framework is not sufficient. We suggest that future research should seek to explore and understand the behavior of their joint execution.
Finally, we also note that LaSER ignores environmental reward maximization during task exploration On one hand, this keeps meta-RL agents general, discouraging them from overfitting to single goal. On the other hand, this may cause issues when environmental rewards are necessary for the agent's s survival. While we provide a simple method for combining our proposed exploration rewards with environmental rewards in Eq. 11 more complex approaches may be required to find the optimal trade-off between exploration and survival. We leave this interesting problem as future work.
Acknowledgments
This project was supported by grant from the Honda Research Institute to the University of Manch- ester. The authors would also like to acknowledge the assistance given by Research IT and the use of the Computational Shared Facility at The University of Manchester.
References
Karol Arndt, Murtaza Hazara, Ali Ghadirzadeh, and Ville Kyrki. Meta reinforcement learning for sim-to- real domain adaptation. In 2020 IEEE international conference on robotics and automation (ICRA), pp.
2725-2731. IEFE, 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In Yoshua Bengio and Yann LeCun (cds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Anand Ballou, Xavier Alameda-Pineda. and Chris Reinke. Variational meta reinforcement learning for social robotics. Applied Intelligence, 53(2))227249 27268, 2023.
Andrea Banino, Adrià Puigdomenech Badia, Jacob C Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell. Coberl: Contrastive bert for reinforcement learning. In International Conference on Learning Representations, ICLR, 2022.
Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta- reinforcement learning. In Conference on Robol Learning, pp. 1478 1487. PMLR, 2023a.
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon White- son. A survey of mela-reiiforcemment learning. arXiv preprint or 008028, 2023b.
Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, Zheng Xiong, and Shimon Whiteson. SplAgger: Split aggregation for meta-reinforcement learning. Reinforcement Learning Journal, 1:450 469, 2024.
Michael Beukman, Devon Jarvis, Richard Klein, Steven James, and Benjamin Rosman. Dynamics gener- alisation reinforcement learning via adaptive context-aware policies. Advances in Newral Information Processing Systems, 36, 2024.
Zheushan Bing, Alexander Koch, Xianglong Yao, Kai Huang, and Alois Knoll. Meta-reinforcemmen learning via language instructions. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5985-5991. IEEE, 2023.
Nicolò Botteghi, Mannes Poel, and Christoph Brune. Unsupervised representation learning in deep rein- forcement learning: A review. IEEE Control Systems, 45(2):26-68, 2025.
LaSER still accounts for environmental rewards, as these embedded in the context. vector used for guiding task exploration. However, the exploration objective does not explicitly encourage the maximization such rewards.
23

---

Published in Transactions on Machine Learning Research (04/2026)
Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaci, and David A Clifton. A brief review of hypernetworks in deep learning. Artificial Intelligence Review, 57(9):250, 2024.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Docision transformer: Reinforcement learning via sequence modeling. Advances in neural informalion processing systems, 34:15084 15097, 2021.
Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint arXim:1910.07907 2019.
Zhendong Chu, Renqin Cai, and Hongning Wang. Meta-reinforcement learning via exploratory task clus- tering. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 11633-1611 11641, 2024.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model- based reinforcement learning via meta-yolicy optimization. Conference on Robot Learning, pp. 617-629. PMLR, 2018.
Tristan Deleu, Model-Agnostic Meta-Learning for Reinforcement Learning in PyTorch, 2018. Available at: hups://githuub.cmm httam//rsstaaeleepyrreaeae hatpaammeeeeeeppyooooemmmmmmlll
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Tn North American Chapter of the Association for Computational Linguistics, 2019.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Πya Sutskever, and Picter Abbeel. RI:: Fast reinforcement learning via slow reinforcement. learning. arXiv preprint (at::611.02770 2016.
Akram Erraqabi, Marlos C Machado, Mingde Zhao, Sainbayar Sukhbaatar, Alessandro Lazaric, Denoyer Ludovic, and Yoshua Bengio. Temporal abstractions--uugmeneed temporally contrastive learning: An alternative to the laplacian in rl. In Uncertainty in Artificial Intelligence, pp. 641 651. PMLR, 2022.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35:35603- 35620, 2022.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Haotian Fu, Hongyao Tang, Jianye Hao, Chen Chen, Xidong Feng, Dong Li, and Wulong Liu. Towards effec- tive context for rentfrreinor learning: approach based contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35()), pp. 7457-7465, 2021.
Yuan Gao, Elena Sibirtseva, Ginevra Castellano, and Danica Kragic. Fast adaptation with meta- reinforcement learning for trust modelling in human-robot interaction. In 2019 JEEE/RSJ International Conference on Intelligent. Robots and Systems (TROS), pp. 305-33333 312. TEFF, 2019.
Jake Grigsby, Justin Sasek, Samyak Parajuli, Daniel Adebi, Amy Zhang, and Yuke Zhu. Amago-2: Breaking the multi-task barrier in meta-reinforcement learning with transformers. Advances in Neural Information Processing Systems, 37:87473-87565 2024.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Mela-reinforcement learning of structured exploration strategies. Advances in neural information processing systems, 31, 2018.
Swaminathan Gurumurhy, Sumit Kumar, and Katia Sycara. Mame: Model-agnostic meta-exploration. In
Conference on Robot Learning, pp. 910 922. PMLR, 2020.
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworks. In 5th International Conference on Learning Representations, 2017.
24

---

Published in Transactions on Machine Learning Research (04/2026)
Tuomas Haarnoja, Aurick Zhou, Picter Abbecl, and Sergey Levine. Soft actor-critic: Oft-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp.
PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et Soft actor-critic algorithms and applications. arXiv preprint Xr::::1812. 205.0990,, 2018b.
Hongcai He, Anjie Zhu, Shuang Liang, Feiyu Chen, and Jie Shao. Decoupling meta-reinforremmntt learning with gaussian task contexts and skills. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38(11), pp. 12358-12366, 2024.
Jerry Zhi- Zhi-Yang He, Zackory Erickson, Daniel S Brown, Aditi Raghmaathan, and Anca Dragan. Learning representations that enable generalization in assistive tasks. In Conference on Robot Learning, pp- 2105- 2114. PMLR, 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXi::1606.0844 2016.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEFE transactions on pattern analysis and machine intelligence, 44(9):514--516) 2021.
Xiao Shi Huang, Felipe better initialization. In International Conference on Machine Learning, pp. 4475 4483. PMLR, 2020.
Ba, and Maksims Volkovs. Improving transformer optimization through
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational rein- forcement learning for pomdps. In International conference on machine learning, pp. 2117-2126. PMLR, 2018.
Takahisa Imagawa, Takuya Hiraoka, and Yoshimasa Tsuruoka. Off-policy meta-reiinorrcement learning with belief-based task inference. IEEE Access, 10:49494 49507, 2022.
Peng Jiang, Shiji Song, and Gao Huang. Exploration with task information for meta reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 34(8):4033-4P) 1046,,00,44 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations ICLR, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, maxime gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevier net//forum?dd-hyyassWWY
Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofrr Nachum, and Emma Brun- skill. Supervised pretraining can learn in-context reinforcement learning. Advances in Neural Information Processing Systems, 36:43057-43083. 2023.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising generation, translation, and comprehension. In Annual Meeting of the Association for Computational Linguistics, 2019.
pre-training for natural language
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint ar Xie: 1707 7770008333 2017.
25

---

Published in Transactions on Machine Learning Research (04/2026)
Evan Z Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta-reinforrement learning without sacrifices. In International conference machine learning, pp. 6925 6935. PMLR, 2021.
Reginald McLean, Evangelos Chatzaroulas, Lue McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, KR Zentner, Ryan Julian, JK Terry, Isaac Woungang, et al. Meta-world+; An improved, standardized, rl benchmark. arXiv preprint arXi::2505 11909 2025.
Luckeciano C Melo. Transformers are meta-reinforcemenu learners. In indernational conference on machine learning, pp. 15340-15359. PMLR, 2022.
Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano V Albrecht. A survey of ad hoc teamwork research. In European conference on multi-agent systems, pp. 275-2925 Springer, 2022.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. simple neural attentive meta-learner. In 6th International Conference on Learning Representations ICLR, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540))522 533, 2015.
Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. No representation, no trust: Connecting representation, collapse, and trust issues in ppo. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volune 37, pp. 69652 69699, Curran Associates, Inc., 2024.
Amir Moeini, Jiuqi Wang, Jacob Beck, Ethan Blaser, Shimon Whiteson, Rohan Chandra, and Shangtong Zhang. A survey of in-context reinforcement learning. arXie preprint arXiv:2502.07977 2025.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcemmnt learning. In International Conference on Learning Representations, 2018.
Mohammadreza Nakhaeinezhaddfrrd, Aidan Scannell, and Joni Pajarinen. Entropy regularized task repre- sentation learning for offline meta-reinforeement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 19616-19623, 2025.
Ben Norman and Jeff Clune, First-explore, then exploit: Meta-learning solve hard exploration-exxllotttioo trade-offs. In NewrIPS 2024 Workshop Open- World Agents, 2024.
Jose Javier Gonzalez Ortiz, John Guttag, and Adrian Dalca. Magnitude invariant parametrizations improve hypernetwork learning. arXiv preprint orv::2304400644 2023.
Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht. Agent modelling under partial observabil ity for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:19210-19221 2021.
Mary Phuong and Marcus Hutter. Formal algorithms for transformers. arXiv preprint arXiv:2207.09238, 2022.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common- specific low-rank decomposition. In International Conference on Machine Learning, pp. 7728-7738. PMLR, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. URL http:://wwwmieecattaiin.cc
com//resources//pffffccooooooof /ttp///rrceee//////-//p-//1 1
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
26

---

Published in Transactions on Machine Learning Research (04/2026)
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient oft-policy meta- reinforcement learning via probabilistic context variables. In International conference on machine learning, pp. 5331 5340. PMLR, 2019.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy scarch. In International Conference on Learning Representations, 2019.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and P. Abboel. High-dimensional contin- uous control using generalized advantage estimation. CoRR, abs/1506.02455 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithus. arXiv preprint SEf1:1707.000 2017.
Gresa Shala, André Biedenkapp, and Josif Grabocka. Hierarchical transformers are efficient meta- reinforcement learners. arXiv preprint arXiv:2402. (00:::200000002,,, 2024.
Bradly C Stadic, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Picter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via mela-reinforoement learning. Advances in Newral Information Processing Systems, 31, 2018.
Radu Stoican, Angelo Cangelosi, and Thomas H Weisswange. Mewa: A benchmark for meta-learning in collaborative working agents. In 2023 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1435-1442. IEEE, 2023.
Mingfei Sun, Vitaly Kurin, Guoqing Liu, Sam Devlin, Tao Qin, Katja Hofmann, and Shimon Whiteson. You may not need ratio clipping in ppo. arXiv preprint arXi::2202.000 2022.
Mingfei Sun, Sam Devlin, Jacob Beck, Katja Hofmann, and Shimon Whiteson. Trust region bounds for decen- tralized ppo under non-stationarity. In Proceedings of the 2023 International Conference on Aulonomous Agents and Multiagent Systems, pp. 13, 2023.
Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning to learn: Meta-critic networks for sample efficient learning. arXiv preprint arXiv:1706. 17X.v:1706.0952,, 2017.
Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018
Scbastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn, pp. 3 17. Springer, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEFF/RSJ international conference on intelligend robots and systems, pp. 5026- 5033. IEFF, 2012.
Laurens Van der Maaten and GeofTrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11), 2008.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAT conference artificial intelligence, volume 30(1), 2016.
M Alex O Vasilescu. A Mullilinear (Tensor) Algebruic Framework for Computer Graphics, Compuler Vision and Machine Learning. PhD thesis, University of Toronto, 2009.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jane X Wang. Zeb Kerth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundel, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. ar preprint arx 1.05663, 2016a.
27

---

Published in Transactions on Machine Learning Research (04/2026)
Yuhui Wang, Hao He, and Xiaoyang Tan. Truly proximal policy optimization. In Uncertainty in artificial intelligence, pp. 113-122. PMT.R, 2020.
Ziyu Wang, Tom Schaul, Matleo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995- 2003. PMLR, 2016b.
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algo- rithms. Connection Science, 3(3):241 268, 1991.
Haotian Xu, Qi Fang, Cong Hu, Yue Hu, and Quanjun Yin. Mira: Model- based imagined rollouts augmen- tation for son-stationarity in multi-agent systems. Mathematics, 10(17):3059, 2022a.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In international conference on machine learning, pp. 24631 -2645.. PMLR, 2022b.
Jiachen Yang, Ethan Wang, Rakshit Trivedi, Tuo Zhao, and Hongyuan Zha. Adaptive incentive design with multi-agent meta-gradient reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp. 1436-1445, 2022.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsca Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100 PMLR, 2020.
Xuehui Yu, Mhairi Dunion, Xin Li, and Stefano V Albrecht. Skill-aware mutual information optimisation for zero-shot generalisation in reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and Chongjie Zhang, Metacure: Meta reinforeement learning with enpowerment-driven exploration. In International Con- ference on Machine Learning, pp. 12600-12610. PMLR, 2021.
Tony Z Zhao, Jianlan Luo, Oleg Sushkov, Rugile Povceviciute, Nicolas Heess, Jon Scholz, Stefan Schaal, and Sergey Levine. Offline metla-riinforcement learning for industrial insertion. In 2022 international conference robotics and automation (ICRA), pp. 6386-6393 6393. TEEE, 2022.
Hongtu Zhou, Ruiling Yang, Yakun Zhu, Haoqi Zhao, Hai Zhang, Di Zhang, Junqiao Zhao, Chen Ye, and Changjun Jiang. CERTAIN: Context uncertainty one-shot adaptation for context-based offline meta reinforcement learning. In Forby-second International Conference Machine Learning, 2025. URL
https::/ woet/forumidd=cx00oee
Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta. Environment probing interaction policies. In 7th Inter- national Conference on Learning Representations, ICLR 2019, 2019.
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adap- tation via meta-learning. In International Conference on Machine Learning, pp. 7693-7700 PMLR, 2019.
Luisa Zintgraf, Sam Devlin, Kamil Ciosek, Shimon Whiteson, and Katja Hofmann. Deep interactive bayesian reinforcement learning via meta-learning. In Proceedings of the 20th International Conference on Au- tonomous Agents and MultiAgent Systems, pp. 1712-1714, 2021a.
Luisa. Zintgraf, Scbastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: variational bayes-adaptive deep rl via meta-learning. The Journal of Machine Learning Research, 22(1):13198 13236,, 2021b.
Luisa M Zintgraf, Leo Feng, Cong Lu, Maximilian Igl, Kristian Hartikainen, Katja Hofmann, and Shimon Whiteson. Exploration in approximate hyper-state space for meta reinforcement learning. In International Conference on Machine Learning, pp. 12991 13001. PMLR, 2021c.
28

---

Published in Transactions on Machine Learning Research (04/2026)
Appendix Table of Contents
A Extended Methodology Details
30
A.1 Policy Optimization in Mcta-RL
30
A.1.1 PPO Background]
30
A.1.2 PPO for Meta-RL
30
A.2
Matrix Formulation for Task Exploration
32
Formal Matricization of Linear Task Reconstretton
32
A.4
Extended Architecture and Optimization Details
32
A.4.1
Architecture
33
A.4.2
Optimization
33
A.4.3
Masked Sell-Supervised Training
33
B Al Algorithms
34
C Non-Linear Task Reconstruction
34
D Implementation Details
34
D. Encoder
37
D.2 Policies
37
D.3 Meta- Train-r
38
E Benchmarks Details
38
MEWA
38
2
Meta-Worke
39
E.3
HopperMass
39
F
Baseline Algorithms Details
39
MAML
40
PEARL
3
VariBAD
40
DREAM
41
5
First-Fxplors
41
Decision Adapters
G
Additional Results
42
Ablation Study: Linear vs. Non-Linear Task Reconstruction
46
G.2
Ablation Study: Task Solving Alternatives
47
G.3
Qualitative Results for Exploration Policy
47
Hyperparameters
48
29

---

Published in Transactions on Machine Learning Research (04/2026)
A Extended Methodology Details
This Appendix complements the discussion of our novel meta-RL. method in Sec. 3 Here, we provide addi- tional detail, as well as more rigorous formalisms. Additionally, for the reader's convenience, we summarize our main notation and definitions in Table
A.1 Policy Optimization in Meta-RL
LaSER uses the proximal policy optimization (Schulman et al. 2017 PPO) algorithm to optimize both its txpl-expoonation and task-solving policies. Therefore, we provide a short technical description PPO in Appendix/AA.... Then, we show in Appendii[AAI1.2 detailed overview of how PPO can be used in meta-RL, by implementing the method proposed in Sec. 3.1
A.1.1 PPO Background
For a policy my with parameters 0, Schulman et al. 2017 propose the objective
C.PO (0) E, L (0) -. VF (0) (25
CLIP
(12)
where Ct, C2 are constants, CCUP is the main PPO objective, and CYP and S|me] are additional objectives. CVF is a squared-error loss on the value function (s,s, c), while S[TT] is the policy entropy. Note that the sole change from the original description of PPO is that ae(a, St. e) and Ve(s,, c) depend not only on the state St, but also on the task context c. The clipped surrogate objective LCLPP is defined as
CCLIP
E2 min (r. clip(r. clip(r))) -66 +1++)),))]
(13)
for a constant e, policy probability ratio r0), and estimated advantage A.. The probability ratio rr(0) sola.ls..c) is computed using the parameters Oold from before an update. Eq. 13 encourages small, stable Te [e]] policy updates that keep To close to Tams by constraining r0)) to remain within c, c,1
A popular choice for the advantage estimator Å is the generalized advantage estimator (GAE) (Schulman et al. 2015). In the meta-RL setting, the GAE can be defined as AÇAE = "> for an episode T with horizon H. Here, 7 E [0, 1] is the MDP's discount factor, À [0. an additional discount, and
8y - (V(4)1, ((s,, c) is the temporal-difference (TD) error ad. timestep & for reward Tt-
Furthermore, as mentioned in Sec. 3.1
stabilize PPO (Wang al.
2020
Sun et al.
2022
2023 Moalla
ct al.
2024 using proximal feature optimization (Moalla et al. 2024
PFO).
A.1.2
PPO for Meta-RL
In Sec. 3.1
propose a simple change to the policy optimization objective. By replacing the environment reward with our proposed neta-reward rtt, PPO can be used to optimize mo to solve meta-RL tasks. That is, we compute the TD error
()) = + 3 (ss, c) ((1)) e) + + (V(s V(s c).
(14)
where w(s c) max(0, tVnh(((ss.) V(si, c) ()). The estimated advantages then be computed (C) (1A)) Note that, bocause (G) is now function of the parameters 0, the advantages must be recomputed every time 0 is updated, i.e., after each PPO minibatch update. This is in contrast to standard PPO, where AGAE is computed only once, before update, and then kept fixed until new data is collected.
Intuitively, w(st, e) measures how effective the policy wa is at using the context for each timestep 4. The assumption that, while PPO-optimized policies in single-MDP settings might explore the state space sufficiently, there is no guarantee that the task context space is meta-cxplored enough in meta-RL setting, By introducing w(st. c), whenever e does not lead to an improvement above threshold Ç, the policy is urged to meta-explore from state 8t, thus learning more about c.
30

---

Published in Transactions on Machine Learning Research (04/2026)
Notation
Description
Definition
Mcta-RL Framework
M, p(M)
Task (MDP) from distribution p(M) Timestep episode k
oi, R, rt
of Rd, of E Rd r E ER
d
Sum of dimensions in (st at r4) tuple
d = d + do +1
H
Horizon
H E
H'
Total dimension of H steps
H' = Hd
K
Few-shot adaptation budget Episode vector
K E E
7 E RH
PM. (T)
Episode distribution in M; under Mcta-episode matrix (K episodes)
D(K)
D(K) E HR'
B
Tensor of Q meta-episodes from M,
B RQXH'XR
B
j-th meta-episode in B
B, E xK
B
Batch of meta-episodes
B M:
proeeplor
Mr ~ x(M)
Latent Representations
Latent task context
C RR (HKdmrtt
r
Task exploration representation Task-specifie representation
I E R (Hddmdd))
Z
Z (dx)H))
z
Global representation
2 E Rdmddll
Optimization
c(s,,c)
Dynamic in-context weight
w(s;, c) = max(0, (an( Vo c)-()
rt
Augmented meta-reward
rf + B (ws, ((m
dg
Causal similarity between episode and previous (A: - episodes
dN = k Σ ST T.
pk
Exploration reward in k-th episode
ph exp (-1 E Ryxix
C
Tensor of coefficients
B
Linear task reconstruction
5j
Lincar reconstruction target
6, = 1 - cxp (CB)-BB
Lrec
Reconstruction loss
Ci-ree
Task reconstruction loss
Leontr
Contrastive loss
R
Regularizer
CLASER
Unified LaSER loss
Architecture
Task-solving policy
T
explore
Task exploration policy
f
Online unidirectional encoder
fb.
Offine bidirectional encoder
9w
Hybrid dual-encoder combining fu and fo
hrec
Self-supervised reconstruction head
ht-rec
reoostrccontr head
Table 3: Summary of main notation and definitions.
As the agent improves at using e maximize return, the exploration bonus wrss, c)S[ma st. e) at state s; decreases. The standard PPO objective is only recovered when V(st) S V c) + Ç. This signals that the agent has learned how to in state 8t, and additional meta-exploration is required.
31

---

Published in Transactions on Machine Learning Research (04/2026)
A.2 Matrix Formulation for Task Exploration
This section provides the formal matrix operations used to compute the causal similarity scores discussed in Sec. 3.2 As previously established, the goal of task exploration is to collect a K-shot meta-episode D(K) that avoids redundancy and captures the essential dynamics of task M, Ultimately, this data must be sufficient to construct context vector e. that enables the task policy æ to become optimal in M:
The latent representation generated by fu is matrix I E R(Hdmo)))* while the pairwise cosine similarity is a matrix Sc E RK*K, The average column-wise similarity in D(K) is then vector d E R computed as d Sc where 1 is the K-dimensional all-ones vector.
Consequently, dy represents the average similarity between the k-th episode and all other episodes. However, since xexplore must collect episodes sequentially during online adaptation, cannot access the similarity scores between its current episode k and future, uncollected episodes. To resolve this, we apply causal mask M E k defined a strictly lower triangular matrix (i.e., M,, 1 if j < i, and 0 otherwise), by taking the Hadamard product with ST(TT, r). This masked similarity matrix is then used to compute the causal similarity vector d = (TT @ M)1 required in Eq.
A.3 Formal Matricization of Linear Task Reconstruction
In Sec. 3.3 we introduced Assumption (Linear Task Reconstruction) using batched tensor notation for clarity. Here, we reiterate this assumption in more detailed and formal manner.
Recall that our motivation is to sidestep the challenge of having to directly find K episodes that allow K-shot adaptation. Consider rearranging the tensor B E Q xK of Q meta-episodes into a matrix
Bi2 E "=' (QR) of QK episodes via mode-2 matricization (Vasilescu, 2009). That is, (Bpp)), represents
the k-th episode, for & E [QK]. Then, assuming that 2 is sufficient for K-shot adaptation, Definition [
implics that B is composed of the K episodes required for adaptation, together with the (Q- (Q containing redundant irrelevant information. More precisely, there exists a meta-episode Big composed of these K episodes. We can now formally introduce the matrix form of our work's core assumption:
1) episodes
Assumption (Matrix Formulation of Linear Task Reconstruction). Let B E H' contain QK episodes collected from an MDP M2 such that B is sufficient for K K-shot adaptation. Let Big E H be the submatriz of Big that contains these K necessary episodes. Then, we assume there erists a matria C12) E (QKK) such that
Bi Cp) B
(15)
It dircctly follows from our assumption that rank Bp)) ~ K. Consequently, the submatrix B is an optimal full-rank approximation of Big containing the linearly independent columns in B while Cp is its coefficient matrix.
To make our framework practical to implement and simpler to describe, we can casily switch back to the tensor representations used in the main text. Specifically, we use the tensor B instead of B represent Bi: the mela-episode slice Bj E H for a given j E [Q], and rearrange the coefficients C2y] into tensor C E Q Eq. 15 is then equivalent to the original batch matrix multiplication formulation B,C z B.
A.4 Extended Architecture and Optimization Details
Here, we extend the discussion in Sec. 3.4 on our architecture and optimization approach. The implementa- tion details are then given in Appendix
Here, assume that has already been meta-trained and has enough prior knowledge of the task distribution p(M) to use effectively.
32

---

Published in Transactions on Machine Learning Research (04/2026)
A.4.1 Architecture
While encoders fu and fb share the Eq. 10 loss LLSSER, their specific architectures require different op- timization approaches. The unidirectional transformer Ju is trained using autoregressive attention masks (Vaswani et al. 2017; Radford et al. 2018). This masks future timesteps when computing the attention matrix, cnsuring that the agent cannot "cheat" during training by using information that should not be available. This ult ultilllely enables Ju to operate in online settings, where encodings are computed timestep by timestep as data is collected.
In contrast, fo has no online constraints. Specifically, fo always operates on complete mota-episodes, where a masked bidirectional model may offer richer representation. We follow the standard approach of training bidirectional transformers through masked self-supervision, but defer the details of this process to Ap- pendix A.4.3] While we simplified notation in the main text when discussing the computation of the global and task-specific representations (z, (z.Z) fu(B; Wb), note that in practice fo actually takes batch
B = B ~ PM.
explore
Mr ~ Mcnp(m)}
(16)
as input, where each ( E DQ is a collection of Q meta-episodes collected by o explore from a unique task My (as shown in Alg. I This accelerates training in a practical implementation. More importantly,
it allows E cmoddll to capture the shared structure across multiple elements of p(M) by encoding the full batch. Simultaneously, [Q Xdmodel is independently encoded from ((x) capture the properties of Mi.
compute set { (Z [8] /==1 of task-specific representations, where each Z e
A.4.2 Optimization
The dual encoder g employs two pre-training heads during meta-training, with each having different roles for the optimization of the LaSER model. The task reconstruction head ht-re optimized to minimize loss fi-ree (Eq. 6 learning to compute the coefficient tensor C = he-rec (gw (.. wt-ree). That is, hi-rec combines Iw (B) (the representations of Q meta-episodes) with (B,) (the representation of the j-th meta- episode). To increase stability, the tensor C used to compute 5, in Eq. 7 is encoded by target parameters @ which are only updated to Wh-ree every v itcrations (Mnih et al. 2015
We note, however, that we do not backpropegate ft-ree through the parameters w of g. The intuition is that the role of is to create a contrastive-optimiie to compute C, which is infeasible in practice. The sole role of htrre then to project the output of g into a collection of coefficients that satisfy the linear equation in Assumption D
latent space by following loss Lconte (Eq. S not to learn
Conversely, the role of the reconstruction head hrec is to optimize parameters w by learning to reconstruct corrupted inputs. Specifically, hrec outputs an attempt at reconstructing the masked input given to g. necessary for computing the loss Lree-
The final piece left to discuss in our optimization process is the regularization term R in the composite loss CLSSER. We take inspiration from Piratla et al. (2020) and use to regularize the latent spaces generated by fo and fur Specifically, we enforce a soft orthogonality constraint between the vector representations z and Zj,,;: for each j E [Q) and timestep t [HK], by minimizing (z (z..)...)2 As discussed in the main text, this aims to encourage Z to avoid capturing the global information already contained in z. Finally, R also normalizes the vector representations z, Zja.. and F, for all j E [Q], t E [HK], k E [K].
A.4.3 Masked Supervised Training
The bidirectional transformer encoder fe, introduced in Sec. 3 learns uscful data representations by learning to reconstruct its input. Since reconstructing meta-episode B, is trivial when the entire Bj is given input, we use masked self-supervised training. We create masked meta-episode B maskedd by applying stochastic masking function, similarly to Devlin et al. (2019) and Lewis et al. 2019)) More precisely, Biaakked is identical to By, with the exception that each timestep (s, a, r) in B Biaasedd has a 15% chance of being corrupted. The loss Lree measures the ability of So to predict the true value of each corrupted timestep. The encoder must
33

---

Published in Transactions on Machine Learning Research (04/2026)
learn compute these values from the rest, non-corrupted timesteps in the input. We consider two types of corruption.
Masking: with a probability of 80%, the selected timestep is replaced by a special (MASK) timestep, which carries no information.
Replacing: with a probability of 10,, the selected timestep is replaced by another timestop from the meta-episode.
The rest (i.e., 10%) of the selected timesteps are not corrupted but are still used when computing Cree. To optimize this loss, fo must learn general temporal relationships between the timestops in a meta-episode. We compute fre as the MSE between the unmasked tokens in В₁ and the reconstructed meta-episode BJ (c; Wree), where c = [w (gmssked), encoder. When training the task policy me or during meta-testing, we compute c using unmasked meta- episodes.
(Bruskeed) Note that masked meta-episodes are only used to train the
B Algorithms
For the sake of completeness, but also to enable reproducibility and further analysis (Phuong & Hutter 2022), provide pscudocode for proposed algorithm LaSER. Alg. shows our meta-training process. It is composed of three main parts. Each part optimizes one of the three LaSER components: the encoder (Alg. 30, the exploration policy T details on how we meta-test a fully trained LaSER agent in Alg. 2
explore (Alg, 4), and the task policy mo (Alg. 5 Finally, we provide
C Non-Linear Task Reconstruction
In this section, we show a straightforward method of extending the linear task reconstruction formulation from Sec. 3.3 to non-linear variant. Specifically, we relax the linear Assumption [ which assumes the existence of coefficient tensor C such that B,C B.
For the non-lincar extension, we replace the linear mapping with a non-lincar function V., modeled as an MLP parameterized by v.. To obtain these parameters, we repurpose the task reconstruction head hy-rec (defined in Sec. (.4) to output the vector & instead of a tensor C. This effectively turns /t into a hypernetwork that computes the weights and biases vb = (9-ree(g gw (B,); Wi-ree). Assumption can then be replaced with the non-linear variant (B,) B.
The tasttrocontiiuttion loss (Eq. 6 is adapted to minimize the error of this non-linear reconstruction:
Li-ree E MM B~ PM,
P p [SS (V (B,). B)]
(17)
Similarly, the target 6, (Eq. 7 used to shape the latent exploration space becomes:
dj = exp (EE(V, B)*
(18)
Apart from these changes, the LaSER variant with non-linear task reconstruction follows the same formula- tion as linear LaSER (see Sec. 3).
In practice, we take the same approach as the linear variant and process a batch B of Q meta-episodes by independently generating a non-linear transformation for each meta-cpisode. Specifically, hurec outputs set
of Q distinct parameter vectors t one for each target meta-episode B, E B, for g E [Q]. An element-wise concatenation these Q reconstructions would then give the desired approximation: {MLP vy (1 B.
Implementation Details
We provide additional details on our implementation of the LaSER algorithm and our architecture. This
section is complemented by the hyperparameters given in Appendix H
34

---

Published in Transactions on Machine Learning Research (04/2026)
Algorithm 3 train encoder()
Input p(M), task distribution;
explore Ts
exploration policy; Iw. encoder;
Wrees Wt-ree, head parameters
Output W, Wreer Wareer updated parameters
explore
1:
B
Mi
Mi ((M)
> Collect an ordered batch of data from multiple tasks
Bunaeked f- (mask B BB| E B}
3: for u E [encodee
do
4:
2:
((1)
( })
fo(B (Buakeed, wh)
D Get global and task-specific representations
LLASER
0
6: for E
[B| do
B <- B;
B masked f- B Baakked
for j E [Q] do
r - [u(B) maskedd Wa)
10:
C + hwn (z,
r
>> Eqq 9
11:
B' t Anse (c; Wrer)
Compute reconstructed input
12:
end for
13:
j U(Q)
14:
C - hETTO (Aw (B), (B)); wr.ro))
15:
6j - 1 - exp
3)
Eq.
16:
Wu + wh +
17:
L (w, Whees We-rec)
& compute using B, B', C, and 5,
Eq.
8 10
(1)
18:
CLASER f- LLASER [2] LASER (w, Whea; Wi-ree)
19:
end for
20:
Wreer Whrre +- update using CLASER
21:
end for
22: return w. Wrecs Wt-rec
35

---

Published in Transactions on Machine Learning Research (04/2026)
Algorithm train crain__xxploraiinn policy()
)
Input p(M), task distribution; T Output o, updated paramcters
eexplore exploration policy; Iw, encoder
+
for M2 p(M) do
3:
D(K) +
4:
for k E [K] do
5:
for t [H] do
ap ~ To rexplore (-1 st, 1)) D(k,1) t (of, r/) ((cktt) fu (Dihitt); wu)
6:
Collect st s441r rh by taking action of in M; end for end for
10:
11:
r + fu(D(k); wu); mask SCTTTT) r)))
12:
13:
pe < compute for each E [K]
Eq
Replace environment rewards in D(K) with pk
14:
D D [D,
15:
16: end for
17: for u [NPPo] do
18: Optimize & on transitions from D using PPO
19: end for
20: return Ø
Algorithm train task policy()
Input p(M), task distribution;
ne, task policy; n explore exploration policy; Iw: encoder; 2, shared component
Output 0, updated parameters
B
for Mi p(M) do oxptorr
3: D(K) PM:
Sample exploration meta-episode
c+ (Dlk),
5: B+ [B, T ~ PM (ro(e)s.c)
D. Sample exploitation episode
6: end for
7: for u E [NPPo] do
Ag 4
for 7 T do
10:
for LE [H]] do
11:
Compute V(s,,)), c), V(s:), and S(mT)(sE,
12:
c( + max (0, tanh (V(s;) ((s,, c) ())
Eqq Eq.
3
13:
r t Fi + Bu(s,, c)S(Tel(s,, GAE (0) + compute using r?
14:
A
Åg Ao (G)
15:
16: end for
17: end for
18: Optimize 0 on transitions from B with advantages Åg using PPO
19:
end for
20: return
36

---

Published in Transactions on Machine Learning Research (04/2026)
D.1 Encoder
Both fu and fo use the same transformer architecture (Vaswani et al. 2017): except that fu is unidirectional, while fo is bidircctional. Each transformer has size dmodel 128, layers, 16 attention heads, and food- forward networks of size 512, with GELU activation functions (Hendrycks & Gimpel 2016). All imput timesteps are linearly mapped to dimoee-dimensional embeddings, with positional encodings added before they are passed through the transformer. For stable training, we apply the T-Fixup initialization scheme (Huang ct al. )020). as recommended by Melo (022).
The output of the transformer in fu is passed through a single-layer feed-forward network that outputs P. Similarly, the transformer output of fo is independently processed by two separate single-layer feed-forward networks, outputting z and Z. Each of these three networks has size 64 and uses tanh activation functions.
When using data from multiple tasks, I and Z are computed separately for each task. In contrast, is computed with data from multiple tasks, leading to general representation of the entire distribution.
Before computing 2, we apply average pooling to reduce the dimensionality of the output of fo. We treat Z as a batch of Q meta-cpisode representations, such that each slice Zj. for j E [Q], is computed independently by the feed-forward network. Similarly, each of the Q meta-episodes has its own representation r, computed independently by the feed-forward network in Sa-
The encoder Iw has two final transformer layers, fwn. similar to those in fo. The first layer combines Z and Г. The second computes by combining the output of the first with 2. The reconstruction head hree is a single linear layer that transforms each of the HK vectors in e from dmodel to d, thus computing the reconstructed input. The task-reconstruction head ft-ree is a (256, 128, 128) feed-forward network with GELU activations, used for computing the lensor of coefficients C. We have found, empirically, that using only the specifc representations as input for Mrrec is enough. That is, for a given j E [Q], we compute
C= hu-ree Zz,Zji Wt-ree), where (-. () = fo (B;w),
(19)
(20)
and optimize Warre such that C approximates the linear transformation that maps B, to B. To train Jar hyees and hi-ree, we use the Adam optimizer (Kingma & Ba 2015
D.2 Policies
We represent T (128, 128, 128) networks with tanh activations to represent policy Th take 64-dimensional embeddings of st and (okktt) as input, which are computed by a shared, single-layer, tanb-activated network.
explore Vo, Te, and Vo using feed-forward networks. For task exploration, we use two separate
explore and value function Vg. These
We use similar architectures for mo and Vg. However, is first mapped to a lower-dimensional representation by a feed-forward context encoder with tanh activations. The same encoder is used for both Ty and Vg. The hyperparameters of ma and Vo are given in Table g
To optimize both policies, each PPO update is for Nrro epochs, using minibatches. The exploration policy is always meta-trained using NpPo = 4 with 2 minibatches. For the task policy, these two hyperpa- rameters are wwvironment-dependeee r, and the context c before passing them to the task policy. We use the Adam optimizer, and clip the norm of the gradients to 0.5 before updating parameters & and B. Additionally, we set the coefficients in Eq. 12 to Ci = 0.5 and 02 = 0.01. The GAE ÁGAE is computed using discounts 7 = 0.99 and 0.9.
and shown in Table g Moreover, we normalize the state the reward
Finally, as mentioned in Sec. 3.1 we stabilize the optimization of no by adding the PFO term suggested by Moalla et al. (2024) to the PPO objective, weighted by cPFO, which is set based on the environment.. The explore We found that including it made training more difficult, PFO term is omitted when optimizing T
whereas n explore was already sufficiently stable without it.
37

---

Published in Transactions on Machine Learning Research (04/2026)
D.3 Meta-Training Training
LaSER agents are meta-trained in two phases. In the first phase, we alternate between updating the encoder IN and the exploration policy T6 explore for Nexplore iterations. Before each encoder update, we collect a datasct B by following explore in different tasks. Each clement B B is tensor that contains Q meta-episodes from task M; p(M).
"
We update IN on the masked dataset gmasked for N Mencoder epochs (see Alg. 3 To improve meta-training sample efficiency, we use all Q meta-episodes in each tensor B to compute Lree and R. In practice each B is stored a buffer and reused in future updates. To update T exxloore exploration data is collected from multiple tasks as described in Sec. 3.2 Tn the second phase, the task policy my updated for Naak iterations.
For the losses Free and Lirec in Eq. 10 we exclude the reconstructed actions from the loss computation. This ensures that the encoder focuses learning patterns in states and rewards, rather than the exploration policy. Additionally, we scale the loss on states by factor of 0.05.
Note that the encoder In is optimized purely through self-supervised methods. This constraint can be lifted. The RL loss used to update the task policy Te can also be used to fine-tune 9w- This fine-tuning could adapt the general pretrained encoder to align more closely with the task-solving objective. However, similar Zintgraf al. (2021b), we observe no empirical benefits from fine-tuning, so we omit in our experiments.
A closely related idea is to continue improving the context vector e by encoding data from the exploitation episode as it is being collected by the task policy тæ. This effectively turns our setting into "(K + 1)-shot adaptation", where the final episode comes from a different source. We have considered this alternative variant, but observed no significant. changes in terms of empirical performance. However, a more interleaved interaction between the exploration and task policies may lead to more effective data collection.
In practice, LaSER's encoder gw and exploration policy T exxloore are meta-trained using an Nvidia A100 80GB GPU. In the second phase, we switch to Nvidia. RTX 4070 Ti SUPER 16GB GPU meta-train the task policy øg- All environment data is collected using CPU computations only.
E
Benchmarks Details
This section provides additional implementation details for the meta-RL benchmarks used in this work.
E.1
MEWA
We continue the discussion from Sec. 5.1.1 and provide more details on the tasks available in the MEWA benchmark. In our implementation, we use the publicly available official code for MEWA, at https: htt.h/uwbb..o///ttccaaaaoaoo
In critical state 52 of type x, risky action Grisk can either lead to state s' or s'.. We use s' to denote that a mistake has been avoided and the agent has progressed in the task. In contrast, s denotes that a mistake has happened, resulting in a large delay in lask completion and thus, lower returns. The probability of Grisk leading to a mistake depends on both the mistake's type x and the dynamics of task M, ~ p(M). Formally, we define Ti(s', sz, arisks M,) p to be the task-specific transition leading to mistake. Here, (8) for all y < x, denotes the probabilitics of making a mistake of cach type. The probability of avoiding mistake is then simply Ti(s' sz, Grisk, 1 pb.
the vector (()) with ps (6) S p
An agent's second option is to take a safe action asate in a critical state Sg. This comes at the cost of a small delay, and leads to a critical state 82-1, with p (6) task-agnostic transition (s 8z, Gsafe: =1)=1.
S p(). Formally, the dynamics are given by the
Table g provides the 12 meta-testing tasks used for the results in Sec. 5 We describe each task by its corresponding vector of mistake probabilities p(t) E 10, Additionally, for each task Mi, we provide the average probability of mistake of any type happening, computed as 1/4 E pbo As previously
38

---

Published in Transactions on Machine Learning Research (04/2026)
mentioned, use this rough measure of similarity between tasks. Finally, mark tasks that are outside of MEWA's meta-training distribution (i.e., cannot be sampled during meta-training) as OOD tasks.
Task Index
Mistake Probability
Average Mistake
OOD
Type I
Type II
Type III
Type IV
Probability
0
0
0
0
0
0.000
No
1
0.05
0
0
0
0.013
No
2
0.1
0
0
0
0.025
No
3
0.772
0
0
0
0.193
No
4
0.672
0.618
0
0
0.322
No
5
0.622
0.618
0.577
0.434
0.563
No
6
0.622
0.618
0.577
0.534
0.588
Yes
7
0.872
0.818
0.777
0
0.617
Yes
8
0.672
0.668
0.627
0.584
0.638
Yes
9 0.972
0.968
0.927
0.384
0.813
Yes
10
0.972
0.968
0.927
0.784
0.913
Yes
11
0.972
0.968
0.927
0.884
0.938
Yes
Table 4: The configuration of the 12 MEWA tasks used for meta-testing agents. A task is described by 4-dimensional vector. Each dimension denotes the probability of mistake of the corresponding type happening. We additionally compute the average mistake probability across all types. This can be seen as a way of comparing tasks, i.e., tasks with similar average probabilities have similar optimal policies. Finally, tasks marked as OOD are outside MEWA's meta-training task distribution.
E.2 Meta-World
Meta-World World was proposed by Yu et al. (020) Its latest version, which was introduced by McLean et al. (2025), offers a unified and consistent framework for easily evaluating and comparing meta-RL algorithms. The most important change in this version is the option selecting between Meta- World's s initial set of task- specific reward functions (V1) and the newer set (V2). To be more consistent. with the recent literature, we choose to follow V2 in our work. All Meta-World experiments are performed on the ML10 evaluation protocol. Ten tasks, with randomized positions for objects and goals, are selected for meta-training. Five structurally similar novel tasks are then used for meta-testing (for more details, see Yu et al (2020)). To implement Meta- World, we use the official open codebase provided at httpps//githhbb.. htw.oiibcoooo//aaaaaaaaaaaaaa Farama- Fontrama-ouudation, Foundation/Metaaooti Metaworld
Meta-
E.3 HopperMass
Our implementation of MuJoCo HopperMass https://gtaabboom/odddmanaeeeeed Mobbaamaddreaaaaaaae/EERR..
similar to the one provided by Nakhaeinezhadfard et al. (2025). The main difference from other implementations used in the literature is the focus on generating OOD tasks during meta-testing. The hopper model is characterized by four vectors: body mass, body inertia, friction, and degree of freedom damping. When generating a new task, each element from each of these vectors is scaled by a different weight. Weights are sampled from (1.5-1,1.53) for meta-training tasks and from 1.55-1.5 1.5 u 1.5 for OOD meta-testing tasks.
F Baseline Algorithms Details
We provide details for the architectures, hyperparameters, and the mela-training process for the baseline algorithms used Sec. 5 We tune these algorithms and use the best-performing hyperparameters. Where possible, and if performance not negatively affected, the task policy's architecture and meta-training are similar to LaSER's. Otherwise, the task policy is tuned with the rest of the model. All baseline algorithms are meta-trained using Nvidia RTX 4070 Ti SUPER 16GB GPU.
39

---

Published in Transactions on Machine Learning Research (04/2026)
F.1 MAML
For MAML FFin et al. 2017), we use the publicly available code provided for meta-RL by Deleu (2018) at https:/ http:/ggtttuhbhoo ctm.aaioooooooooooeooooeeeee lasks. For each lask, we sample 5 episodes. MAML is meta-trained to maximize returns after one policy gradient update. However, during meta-testing, it performs
We train for 12000 meta-iterations, with batches of 16
gradient updates. We use a discount factor of = 0.99. We also use À 0.9 to compute the generalized advantage estimator (GAE). To ensure optimal results, we did not use the first-order approximation proposed by Finn et al. (2017), but instead computed the second derivatives and backpropagated. The policy is (64, 64) foed-forward network with tanh activations. Due to the computational cost of MAML, we had to use smaller policy network than in the other algorithms. All other hyperparameters follow the ones used by Finn et al. 2017 in their RL experiments.
F.2 PEARL
We use the publicly available code at PEARL [Rakelly et al. 2019). Each agent meta-trained for 350 iterations on a total of 3000 training tasks, with 16 tasks per batch. At cach iteration, both the encoder and the task policy are optimized for 2000 gradient steps, with batches of 64 transitions for the encoder and 256 for the policy.
/oystr for our implementation of
PEARL uses variational approach for computing task contexts When computing PEARL's loss, the KL divergence of the encoder is weighted by 0.1. Additionally, both the task policy and the encoder have a learning rate of 3 x 10-1 and use the Adam optimizer.
PEARL uses Soft-Actor Critic (SAC) (Haarnoja et al. 2018a by, an off-policy RL algorithm. Note that, since SAC is designed for continuous action spaces, we instead use SAC version for discrete action spaces (Christodoulou 2019). We tune SAC's temperature hyperparameter automatically, using the approach introduced by Haarnoja et al. (018b)), since manual tuning can be difficult. We y 0.99 and larget smoothing coefficient for SAC of 0.005. At each iteration, PEARL adds data collected from 5 randomly selected tasks to replay buffer of size 1000000 timesteps. For each of these tasks, it collects 400 timesteps by conditioning its policy on a context c sampled from a prior distribution over tasks. It additionally collects 600 more timesteps using a a sampled from its meta-trained posterior over tasks. However, this latter data is only used to update the lask policy, not the encoder. Before training starts, the replay buffer is populated with 2000 timesteps per task, collected by following a uniformly random policy,
discount factor of
PEARL's encoder is (200, 200, 200) feed-forward network with ReLU activations that computes 5- dimensional latent task context vectors The task policy is (128, 128, 128) feed-forward net with tanh activations.
F.3 VariBAD
We use the code at train for 2350 (MEWA), 18750 MMeta-Wordd), or 3125 (HopperMass)) iterations. In MEWA, we meta-train on 3000 tasks. For any benchmark, at each iteration, we collect 200 timesteps per task from 16 different tasks. These timesteps are stored in a buffer of 10000 episodes. Additionally, before training, a uniformly random policy adds 5000 timesteps to the buffer. This buffer is later used update the encoder.
http:tuwwb..oommmiittgnraaaaa/// varibad for VariBAD (Zintgraf et al. 2021b). We meta-
VariBAD uses variational auto-encoder Kingma & Welling, 2014 VAE) to encode data collected from tasks into latent context In MEWA, this encoder is optimized using Adam with learning rate of 0.001. At cach iteration, the encoder is updated for 3 steps, using batches of 15 cpisodes, sampled from the buffer. The encoder is a recurrent neural network of size 128 that computes 5-dimensional task contexts e. The decoder takes e as input, together with the current transition s, a, and reconstructs the reward r. We use a (64, 32) food-forward network to represent the encoder. The state, action, and reward inputs are preprocessed into representations of size 32, 16, and 16, respectively, by separate single-layer networks with ReLU activations. Both the encoder and decoder have such pre-processing layers.
40

---

Published in Transactions on Machine Learning Research (04/2026)
Since VariBAD uses PPO to optimize its policy, we use the same network architectures and hyperparameters for MEWA as LaSER's task policy (see Appendix D.2 and Table gh. However, we do not use the additional PFO objective. Finally, before passing s and с to the policy, we embed them into 64-dimensional represen- tations using separate singlo-layer networks with tanh activations. Similarly to LaSER, all experiments use the discount. factor 7 = 0.99.
For Meta-Worke and HopperMass, we use the hyperparameters proposed by Beck et al. (2023a) for ML10 starting point for our tuning. In Meta-World, we use 2 PPO epochs with 8 minibatches with learning rate of 3 10-4, In HopperMass, have 4 PPO updates with 8 minibatches and a learning rate of 3 1-104. The VAE for both World and HopperMass computes 10-dimensional latent context vectors e, which are used by (256, 256, 128) task policy. Moreover, we normalize the state s; the reward r, and the context c before passing them to the task policy. The rest. of the hyperparameters are the same as for the VariBAD models meta-trained on MEWA.
use batches of 2 episodes to update the VAE. For the task policy, we
F.4 DREAM
We use the public code provided by Liu et al. (2021) at DREAM. We meta-train for 100000 iterations. At each iteration, the model first collects K 4 exploration episodes and updates the exploration policy. Afterwards, the task policy, conditioned on the exploration meta-episode, is used to collect an exploitation episode, and then updated.
/draam//maaam to implement
The decoder used by DREAM takes history of transitions of type rs,), input. Each transition is individually embedded, using layers of size 64 for states, 16 for rewards, and 16 for actions. For each transition, we then concatenate these embeddings and apply a linear layer of size 64. Then, an LSTM with hidden dimension 128 encodes the history of embedded transitions. Finally, the output of this LSTM is postprocessed by (128, 64) feed-forward network with ReLU activations.
Both the task and exploration policies are trained with dueling double Q-learning (Wang et al. 2016b; Van Hasselt et al.) 2016), parameterised as deep recurrent Q-networks. The deep Q-network (DQN) used to train the task policy takes the current state, history, and task context as inputs. The state and task context are scparately embedded using matrices of size 64. The history is encoded using a process similar to the one used by the encoder. However, following Liu et al.] (2021), the history is composed of timesteps instead of transitions (i.e,, no next state s'), rewards are not embedded, actions use a 32-dimensional linear layer, and the hidden size of the LSTM decreased to 64. A foed-forward network of size (256, 64) with ReLU activations is then applied to the concatenation of these output embeddings. The task DQN has two final linear heads for computing the value and advantage functions. The exploration policy's 's DQN has similar architecture
Each of the two DQNs has its own replay buffer, each storing up to 16000 sequences of 50 timesteps. For each sequence, the timesteps are stored in consecutive order. We train both DQNs using the Adam optimizer with a learning rate of 1 x 10-4, At each iteration, we update the DQNs 4 times using batches of size 32, clipping the norm of the gradients to 10. The DQNs are then synchronized after every 5000-th update. Finally, in all experiments, we use the discount factor 7 0.99.
F.5 First-Explore
Our implementation of First- Eisst-Explore uses the publicly available code at
Explore provided by the authors (Norman & Clunc 2024). During meta-training, the agent is optimized for 93 iterations on tasks sampled from pool of 3000 training tasks. At each iteration, First-Explore collects 80000 timesteps, then performs 50 updates. Each update contains batch of 4 distinct tasks, with two meta-cpisodes (one from the exploration and one from the exploitation policy) formed of K = 4 episodes of length H- = 50 per task, for a total of 1600 timesteps per update.
We follow the implementation of Norman & Clune 2024 and use a unidirectional causal transformer encoder, similar to the GPT-2 architocture (Radford al. 2019). This acts as a Decision Transformer (Chen et al. 2021). To match the size of LaSER's meta-testing architecture, the encoder is a 12-layer transformer of size dmodel = 128 with 16 heads. The feed-forward networks have size of 3072, while the embeddings are
41

---

Published in Transactions on Machine Learning Research (04/2026)
768-dimensional. The transformer uses GELU activations. This encoder then acts as both the explore and exploit policies, which share parameters everywhere except in the final linear layer.
This architecture is optimized using the authors' novel gequencc-modelinng based approach. Similarly to them, the exploration and exploitation policies both have successor policy variants. The main policies are then updated to match their successor every T = 10 iterations. During meta-training, we set the enviromment-specffe baseline reward hyperparameter
b -20 which is lower than the minimum return of MEWA episode of length H = 50. Finally, we use the Adam optimizer with linear learning schedule. The learning rate linearly increases from zero to 1 10 over the first 10% of timesteps collected during meta-training, then lincarly decreases back to zero over the course of the optimization process. Additionally, we clip gradients to the [-0.25, 0.25] range.
F.6 Decision Adapters
We implement the hypernetwork-based task policy using the code provided by Beukman et al. (2024) at
heee//Deoiiiiaapppppao Adapter The policy is a (128, 128) feed-forward network, similar to LaSER's task policy (see Appendix D.2p. It also meta-trained with the same hyperparameters (see Table However, this policy only takes the state 8 as input. Moreover, between the policy's 's last hidden layer and output layer, we introduce an additional (32, 32) network with tanh activations. The weights of this final network are generated by hypernetwork, which takes the task context. as input. Following Beukman et al. (2024), we also use skip connection between the input and output of this network with bopernetworkggeeeraae
weights.
The hypernetwork itself is a (64, 64) feed-forward network with tanh activations. The output weights of the hypernetwork are computed in chunks of size 16. Before passing the input c, we pre-process it into a 4-dimensional representation using a single tanh-activated layer. Finally, the actor and the critic have separate hypernetworks.
G Additional Results
We provide results that complement those in Sec. Note that for Fig. 11 LaSER uses a pretrained encoder and exploration policy, while the baseline algorithms are trained from scratch. The choice to display adaptation efficiency measurements in this manner follows similar argument to the one given in Sec. 5.2.3.
Acrrage Losa (kg seale]
10-3
Median Reconstruction Loss
Reconstruction
10-1
Avcrege
10"4
0.06
10-4
0.0
1.5
1e6
4.00
Median Contrestive Loss
30
/vg. Regularizatio Loss
3.00
20n4
2.00
20
0.00
0.5
1.0
2.0
0.0
1.0
2.5
2.0
Timesteo
lcs
Timestap
1c6
Figure 10: Encoder loss for LaSER, computed on lasks from MEWA's meta-training distribution. We show separate plots for each term in the encoder loss function. For 10 random seeds, we provide the standard error as the shaded arcas, average the second and fourth metrics, and report the modian for the other two. We found the median be more appropriate, one of the seeds was unstable towards the end of meta-training, for approximately 0.2e8 timesteps. For this same reason, we restrict the plot of the contrastive loss to the range 1-2.5e 6, 5e-5). To aid readability for exponential losses, the reconstruction and regularization losses use a logarithmic scale. All metrics are smoothed using EMA with GEMA = 0.2.
42

---

Published in Transactions on Machine Learning Research (04/2026)
Averagee Return
LaSER (ours)
VariBAD
DREAM
-0.2
First-Explore
0
2
3
5
6
Timestep
1e6
Figure 11: Undiscounted returns of task-solving policies on tasks from MEWA's 's meta-training task distri- bution. The shaded areas report the standard error of the average return across 10 seods. We use EMA smoothing with dEMA = 0.6.
Avg. Encoder (leg scale)
10
Avg Encoder Loss (log scale)
109
PEARL
Var IBAD
10-1
10
10-3
10-4
10
60
limestep
1e6
limestep
DREAM
740
First-Explore
20
Avg. Encoder Loss
Avg. Encoder Loss
15
700
10
4 Timestep
Timestep
1e6
1e5
Figure 12: Encoder loss for PEARL, VariBAD, DREAM, and First-Exxlors on tasks from MEWA's 's meta- training task distribution. The shaded areas report the standard error of the average loss across 10 seeds. We use logarithmic scales for PEARL and VariBAD, their losses decay exponentially.
43

---

Published in Transactions on Machine Learning Research (04/2026)
-6
Average Exoloration Return
-10
-12
-14
-18
DREAM
-20
Timestep
1e6
Figure 13: Intrinsic exploration return for DREAM's exploration policy MEWA's meta-training task distribution. Shaded areas are the standard error of the average over 10 random seeds.
trerapo Necm
0.2
LaSth
MANI
VerBAD
DRTAM
RestiCustree
Randorr
Figure 14: Average per-task returns for each of the 12 meta-testing tasks in MEWA. Each agent is meta-tested with an exploration budget of K = 4. Results are averaged over 10 seeds, with error bars indicating standard error across seeds. The three baselines are computed on a per-task basis. We normalize returns between the random and optimal baselines. For better visualization, we also restrict the plot to the [-0.02, 1.0] range.
44

---

Published in Transactions on Machine Learning Research (04/2026)
250000
Average Wall Clock Time (s timmestep))
0.025
Average Wall Clock Time
200000
0.020
150000
100000
0.010
50000
0.005
LaSER (ours)
MAML
PEARL
VariBAD
DREAM First-Explare
LaSER (ours)
MAMI
PEARL
VariBAD
DREAM Rist-Explore
Figure 15: Wall clock time required to meta-train cach method. We report the standard error of the mean over 10 seeds. (left) Total meta-training time. (right) Meta-training time averaged over all timesteps.
Algorithm
Parameters
Component Breakdown
Meta- Training 78, 78,39,22 281
Meta- Teta-Testing 8, 311,
Component
Paramcters
LaSER (ours)
Encoder Iw
4, 484, 672
Encoder Heads hrec he-rec explore
70, 282, 967
Exploration Policy T
109, 193
Task Policy (No Context Extractor) w
141,901 961
Task Policy Context Extractor
3, 375, 488
MAML PEARL
5, 676 233,27 271
5, 676
Task Policy
5, 676
233, 271
Encoder
87, 210
Task Policy
146, 061
VariBAD
186,580
178, 243
Encoder
76, 538
Decoder
8, 8,337
Task Policy (No Context Extractor)
101, 101,001 001
Task Policy Context Extractor
704
DREAM
080, 546
1, 080, 546
Encoder
426, 880
Exploration Policy
214, 809
Task Policy
438, 857
First-Explore
10, 541, 192
10, 54,1, 192
Encoder
10, 541, 192
Table 5: Number of parameters for each algorithm. All architectures for the MEWA benchmark. "Meta- Training" refers to the total number of parameters required to meta-train the model, which can differ from the final number of parameters during "Meta- "esting. We additionally offer component wise breakdown.
45

---

Published in Transactions on Machine Learning Research (04/2026)
Average Wall Clock Time (5)
8
Policy rollout
Policy update
6
0
LaSER Task Policy (ours)
Context PPO
Decision Adapter Hypernetwork
Figure 16: Wall clock time results for task policy meta-training. We measure the average time per iteration (in seconds) required to roll out episodes and optimize the policy. The results are averaged over 10 seeds. The bars represent standard error across all seeds.
G.1 Ablation Study: Linear vs. Non-Linear Task Reconstruction
Fig. 17 compares LaSER with a variant that adopts the non-linear approach to task reconstruction, as introduced Appendix C Since the head ft-ree now acts as hypernetwork, we increase its size from a (256, 128, 128) to (256, 128, 128, 320, 128) MLP, to compensate for the increase in complexity. Apart from these changes, the linear and non-linear variants share identical hyperparameters. We model Ve, the output of hi-ree; as Q distinct (16, 16) MLPs with tanh activations.
Avg. Task Recomstructioo Loss
LaSER (Lincar)
0.10
LaSER (Non-Linear)
Return
0.08
0.80
Avg.
0.04
0.70
0.5
1.5
2.0
2
3
1eB
le6
Figure 17: Performance comparison between LaSER with linear and non-linear task reconstruction on MEWA's meta-training task distribution. The meta-training setup and metrics are identical those Fig. 5. (left) Task reconstruction loss ft-ree for linear (Eq. 0 and non-linear (Eq. 17] variants. We use EMA smoothing with dEMA 0.2. (right) Undiscounted exploitation return of policy ne. We use EMA smoothing with QEMA 0.6.
The non-lincar variant achieves lower task reconstruction loss compared to lincar LaSER, indicating that its latent context vector e may capture more complex task structures. However, this increase in expressivily does not translate to improved performance during the task-solving phase, as the task policy achieves returns similar to those of lincar LaSER. This supports our lincar assumption, at least in environments similar MEWA. Consequently, the additional complexity in terms of implementation and computation of the non- linear approach is unnecessary in this setting.
46

---

Published in Transactions on Machine Learning Research (04/2026)
G.2 Ablation Study:
Task Solving Alternatives
Fig. compares LaSER's lask policy from Sec. 5.3 oplimized using meta-rewards from Eq. 2 with two simpler alternatives to those meta-rewards. All models are trained with ground-truth contexts on a fixed set of tasks. The penalty-only baseline, defined as rt - w(1,, c), removes the entropy bonus. It penalizes the policy directly when the context c underutilized (according to w), but lacks an explicit mechanism to encourage exploration that corrects this behavior. The entropy-only baseline, defined as Yx + , [.5[=] c), encourages constant exploration, regardless of how effectively the context is being used. While the policy still receives as imput, the meta-reward is agnostic to how effectively this context is used. Note that both baselines in this ablation study use the same architecture and hyperparameters the LaSER task policy, except / is reduced by factor of 50 in the entropy-only sctting to improve stability.
LaSER Policy
BSinlss,
Tesk-Agnstiic
1.0
0.5
0.9
Rverage Return
0.0
0.7
0.2
0.6
Average
0.5
0.4
0.3
0.0
0.6
1.0
1.2
0.2
0.8
1.0
14
Timestep
Timestep
Figure 18: Ablation study on the meta-reward formulation. The experimental setup and metrics follow those in Fig. 8 (ground-truth contexts, 12 MEWA tasks). (left) Average post-adaptation return. (right) Average delta return.
Our results demonstrate that the proposed LaSER meta-reward outperforms both baselines. The penalty- only agent eventually stabilizes but converges to suboptimal return. We hypothesize that, while this baseline is able to identify context misuse, it cannot correct it due to the lack of targeted exploration. The entropy-only baseline learns well initially, but loses stability in the second half of training. This is likely due the non-decaying entropy bonus, which beneficial at the start, but eventually leads to excessive exploration after the agent has learned to use the context. Critically, both baselines have average delta returns close to zero. This indicates that, compared to policy trained with the meta-rewards in Eq. 2 the ablated agents ignore misuse the provided context, leading to no post-adaptation improvements.
G.3 Qualitative Results for Exploration Policy
In this section, provide qualitative discussion with step-by-step visualizations (see Fig. 19 of the meta-episodes collected during task exploration by three fully meta-trained agents. We use the MEWA meta-testing tasks with indexes 5 and 7 in our evaluation (see Table
Task is in-distribution and has an average mistake probability of 0.563. acts as a "middle ground" task in the MEWA curated distribution: complex enough allow frequent mistakes, without having almost unavoidable mistakes.
In Fig. 19a VariBAD adopts simple exploration strategy for task 5, with episodes that follow similar patterns. Specifically, it takes only red and blue actions in the first half of each episode, then green and orange in the second half.
The visualized episodes suggest that DREAM can achieve more diverse exploration than VariBAD, as it experiments with three out of four mistake types. DREAM only appears struggle at the beginning of episodes, where the sequence of the first actions is always identical, while the action at timestep appears to depend whether mistake has happened at timestep 4. This may suggest that the diversity seen in
47

---

Published in Transactions on Machine Learning Research (04/2026)
this meta-episode may partially be consequence of the environment, rather than being purely driven by DREAM's ability to collect distinct data.
LaSER demonstrates a structured approach to task exploration that appears on par with DREAM. While it chooses to focus mostly on mistakes of types 3 and 4, it experiments with three distinct mistake types in the very first episode, and eventually probes the final type in the third episode. Therefore, as opposed to DREAM, explores all four mistake types. additionally avoids taking the same sequence of actions in the first five timesteps of each episode. This is likely due to LaSER being explicitly optimized to avoid redundancy by learning to collect episodes that are orthogonal in the latent exploration space, while DREAM is only exploring to maximize future exploitation gains.
Task 7 is an out-of-distribution task with an average mistake probability of 0.617. A special property of this particular task is that mistakes types 1, 2, and 3 have a relatively high chance of occurring, while mistakes of type never occur.
In Fig. 19b VariBAD utilizes the same exploration policy in the first three episodes. Specifically, the agent continuously attempts to avoid mistake of type 2, then one of type 1, and finally one of type 3. This results in three episodes that can only be distinguished based on the environment's reactions, rather than VariBAD's 's active choices. The agent adapts its exploration strategy in the final episode. However, it never attempts a risky action that could lead to a type 4 mistake, and thus fails to learn that these never occur.
DREAM collects a meta-episode composed of four identical episodes. Since the DREAM agent has been meta- trained collect episodes that maximize subsequent exploitation returns, it fails diversify its behavior in this task. Instead, it chooses to follow the simplest approach of exploiting the task's type four mistake, leading to a collapse into deterministic policy. Consequently, the lack of experimentation with mistakes of other types implics that this specific meta-episode is not sufficient to learn the true specifications of this task.
LaSER avoids falling into the same pitfall as DREAM, and successfully collects a diverse meta-episode. The first episode similar to those of DREAM's, quickly highlighting the low likelihood of type 4 mistake occurring. However, the agent then decides to also experiment with mistakes of types 2 and 3 in subsequent episodes. This exploration strategy is likely result of LaSER's votaattaainin, where diverse task exploration is explicitly encouraged, regardless of the task's 's structure or of the optimal exploitation policy. Its exploration policy is thus capable of ignoring distracting environmental rewards, instead collecting set of K episodes that, according to the agent, best approximates a large dataset constructed from this task.
H Hyperparameters
Tables 6
show the hyperparameters used by LaSER in each of the three benchmarks.
General Hyperparameters
MEWA
Meta- World
HopperMass
Notes
Nexplore
10, 000
700
600
Ninitial
2, 000
20
30
H
50
200
400
Horizon
K
4
2
4
Episodes per meta-episode
Table 6: LaSER hyperparameters used throughout meta-training and meta-testing.
48

---

Published in Transactions on Machine Learning Research (04/2026)
LaSER (ours)
Ep.
Ep.
18
88
85
60
VariBAD
980
68
Ep.
B0
Fp.
10
30
35
DREAM
Cp.
Timesteo
Padcing
Avoided Mistake type
Action
Action
Mistake Made
(a) MEWA's 's 5-th meta-testing task; 0.563 average mistake probability.
LaSER (ours)
Ep.
B0M
B0M
50W
Ep.3
DREAM
Ep.
Timestep
Action
Action
Padding
Mistake Avalded Mistake Type
Action
Action
Mistake Made
(b) MEWA's 7-th meta-testing task; 0.617 average mistake probability.
Figure 19: Qualitative comparison of exploration behaviors on two MEWA tasks. We plot color-coded sequences of actions for exploration mela-episodes of K = episodes. Solid colors highlight risky actions (i.e., those that may lead to mistakes). The symbols indicate whether mistake occurred or was avoided after risky action, while numbers indicate the type of mistake. Each episode is padded to the maximum length of its respective meta-episode, and each episode terminates automatically after H = 50 timesteps.
49

---

Published in Transactions on Machine Learning Research (04/2026)
Encoder
MEWA
Meta- World
HopperMass
Notes
Learning rale
x 10
5 x 10-
Noncuder
50
2
Encoder updates per iteration
Q
20
Mela-episodes per task # of tasks per batch Maximum # of tasks in buffer
B
5
10
Buffer size
100, 000
1, 000
2,000
Cren: Ct-rees Coontr: CR
0.5, 1, 1, 0.125
Coefficients for loss LLSSER Eq. 10
0.05
Constant in Eq.
250
25
Update delay for paramcters
Table 7: LaSER hyperparameters used for meta- veta-trainning the encoder.
Exploration Policy T
explore
MEWA
Meta-Worke
HopperMass
Notes
Learning rate
x 10-5
8 x 10-3
0.025
0.01
0.003
Constant in Eq.
0.1
PPO clipping (exploration): Eq. 13
Oz
0.01
Environmental reward weight; Eq.
Table LaSER hyperparameters used for meta-training the exploration policy.
Task Policy
MEWA
Meta- World
HopperMass
Notes
Learning rate
1 1x 10
6 x 10
Ntask
10, 000
9.000
2, 500
B
1.5
Coefficient in Eq.
<
-0.13
Threshold in Eq.
3
0.2
PPO clipping (exploitation); Eq.
13
CPFO
0.1
0
PFO cocfficient
NpPo
2
4
# minibatches
Layers
(128, 128, 128)
(256, 256, 128)
Context
(128, 128, 128)
(256, 256, 128, 10)
Encoder for context vector c
encoder layers
Table 9: LaSER hyperparaameters used for meta-training the task policy.
50


Reviews:
[{"review_id": "YIGkg1ss1C", "reviewer": "Reviewer_soxu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a full Meta RL approach, that learns an exploration policy that aims to \"meta-explore\", in particular, to find informative episodes considering the episodes it has already collected. Beyond this they develop a context extraction approach based on a linear task reconstruction assumption, and use this to train a task solving policy that effectively uses the context. This is done by incentivising it to have higher entropy when it is not fully using the contextual information.\n\n##### Strengths\n- I like how the work explicitly separates the task exploration and the task solving parts of the meta RL process, and uses quite different techniques for both.\n- The results in MEWA seem reasonable, in that Laser is one of the only algorithms that beats the task agnostic baseline.\n- The linear subspace idea is neat.\n- Using the difference between a context-conditioned value function and an unconditional one to determine if the context is being used is also a clever idea, and I suspect that can be used on its own in several settings.\n- The idea of incentivising the different episodes to be informative, and different from each other, is a good idea.\n- The ablation results in section 5.3 seem more convincing. particularly how large the delta return is when given access to the context\n- The visualisations in fig 9 are also quite useful\n\n\n##### Weaknesses\n\n- It would be great to compare against the much simpler algorithm from this paper, or explain why it wouldn't be a fair comparison. In particular, because this paper also has this separation between exploration and exploitation policies.\n\t\t- \"First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs\"\n- The method is quite complex and has so many moving parts, that I wonder if it will hinder adoptability and other research building on it. The open source code is useful though\n- The results in meta-world do not seem much different from varibad, although this perhaps just means that the benchmark is not challenging enough.\n- The meta training number of parameters is significantly more than all other baselines (e.g. more than 300x the number of parameters in PEARL). I wonder if those baselines were given more parameters or more compute if they would do better. While the authors do mention that there are fewer meta test parameters, the difference is quite substantial.\n- It took me a few passes to properly understand what was going on in section 3. I read it as very dense, but also verbose at the same time, and I found it hard to follow all of the different parts. I don't know if I have a great suggestion for how to address this though. I think perhaps one part of this is that there are so many components\n\t- In particular, the last part of section 3.3 makes sense but is quite dense and quite hard to understand on a first pass.  Also the jump from how to find $g$ to how to find $f_u$ is quite sudden", "claims_and_evidence_yn": "Yes", "claims_explanation": "I think this is partly true.\n\nIn particular, the exposition is somewhat unclear, and the paper is quite verbose / long, which dilutes it a bit in my opinion.\nHowever, the claims are reasonably supported, although the positive empirical results rely mostly on one benchmark, MEWA. I don't think that is a massive problem though.\n\nTheir new algorithm, laser, does learn the context using a small dataset, and the objective of the exploration policy seems to incentivise it to collect a diverse set of experience.\n\nTheir architecture also makes use of the context more effectively when solving task problems.", "audience_yn": "Yes", "audience_explanation": "Yes, this paper does present a new meta learning approach that seems to effectively explore, and make use of contextual information, which some other approaches don't do as well. It has a number of interesting ideas that I think are worth sharing.", "requested_changes": "##### Major changes\n- I think section 3 is quite unclear and it  make the whole paper hard to follow. I am unsure how to improve it, but perhaps making it feel like less of a jump between subsections could help.\n- Figure 8: Having a baseline that is just state and context concatenated, using normal PPO, would be great.\n\n###### From the weaknesses:\n- It would be great to compare against the much simpler algorithm from this paper, or explain why it wouldn't be a fair comparison. \n\t\t- \"First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs\"\n- The meta training number of parameters is significantly more than all other baselines (e.g. more than 300x the number of parameters in PEARL). I wonder if those baselines were given more parameters or more compute if they would do better.\n\t- A discussion or experiment for this would be useful.\n##### Small Points that are not critical but I think would improve the work\n- Switching the notation from $D^{(K)}$ to $B_j$ is a bit confusing in section 3.2 -> 3.3\n- This can be made clearer I think:\n\t- \"We therefore optimize $f_u$ to learn a latent space that encodes this information.\"\n\t- The paragraph after equation (8) does a good job of explaining this, and I wonder if it makes sense to have that, or something similar, before the dense maths and notation\n-  I think it would be clearer to replace \"The latter head\" on page 8 with \"$h_\\text{rec}$\", and specifying in more depth how the reconstruction works, like figure 4(a) would be beneficial here too.\n-  I wonder if it would be better to move the meta training complexity results of 5.1.3 to after the main results, but I am not very tied to this idea.\n-  I think the discussion of figure 7 (left) reads too much into the small spikes. The curves of variabad and laser overlap with the error bars, so I would argue that they perform the same on metaworld. On hopper mass I would say laser performs slightly better, and is a bit more stable.\n- In section 3.2, it is mentioned that $d_k$ is the similarity between the $k$-th episode and all the others, but then you mention that you use causal masking to compute this for only the episodes $i < k$; I would perhaps suggest using different notation in equation (4) then to indicate this.\n- One other experiment that would be nice is showing that the exploration policy qualitatively or quantitatively explores differently to other algorithms / that it does some intelligent exploration based on the existing episodes it has already explored. This would nicely complement the results showing that the laser task solving policy does actually use the context.", "broader_impact_concerns": "None", "additional_comments": "##### Questions:\n- Do you train each method for a different number of timesteps?", "extra_scores": {}, "extra_text": {}}, {"review_id": "LvJsHjgcmi", "reviewer": "Reviewer_X7vr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes LaSER, a meta-RL framework aimed at improving few-shot adaptation through task-specific, sample-efficient exploration. The key idea is to formalize informative exploration as identifying a small set of episodes that can linearly reconstruct a larger dataset collected from the same task, under a low-rank assumption. Based on this perspective, the authors introduce a task reconstruction objective to supervise representation learning for exploration, an intrinsic exploration reward that encourages episode-level diversity in a learned latent space, and a novel meta-reward to address the issue that in-context policies may fail to exploit task contexts even when they are available. The method is evaluated primarily on the MEWA benchmark, where exploration is essential, and additionally on Meta-World and HopperMass to demonstrate generality. Empirical results show that LaSER significantly outperforms established meta-RL baselines on exploration-centric tasks, while providing competitive performance on standard benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Pros.\n- Clear and novel conceptual framing of exploration in meta-RL. The idea of low-rank task encoder is clean and technically sound. This is a novel perspective of task modeling in meta-RL.\n- Well-motivated optimization contribution addressing context underutilization. The proposed meta-reward explicitly tackles the empirically observed but under-discussed issue that in-context policies may ignore task context even when it is informative.\n- The implementation is provided to improve reproducibility.\n\nCons.\n- Limited qualitative analysis of exploration behavior. While quantitative results are strong, the paper lacks trajectory-level or qualitative examples illustrating how the learned exploration policy behaves differently from baselines. Given the interpretability of MEWA, concrete behavioral visualizations or walkthroughs would substantially strengthen the claims about task-specific exploration.\n- Assumption of linear task reconstruction is not empirically stress-tested. Considering the complexity of tasks in the wild, the linear task reconstruction could be unrealistic if task identity is not representable as a low-rank combination of episode-level information, such as in environments with highly continuous, nonlinear, or noise-dominated task variations.\n- Missing discussion of recent meta-RL works. Zhou, Hongtu, et al. \"CERTAIN: Context Uncertainty-aware One-Shot Adaptation for Context-based Offline Meta Reinforcement Learning.\" Forty-second International Conference on Machine Learning. Chu, Zhendong, et al. \"Meta-reinforcement learning via exploratory task clustering.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 10. 2024.", "audience_yn": "No", "audience_explanation": "At least a substantial subset of TMLR’s audience would be interested in this paper’s findings, particularly researchers working on meta-reinforcement learning, exploration, representation learning, and in-context adaptation. The paper addresses a well-known but under-explored failure mode in meta-RL: inefficient task exploration and the inability of in-context policies to effectively exploit learned task contexts, and proposes a principled solution grounded in a clear modeling assumption.", "requested_changes": "Please see the cons.", "broader_impact_concerns": "This paper does not raise broader impact concerns. It focuses on algorithmic contributions to meta-reinforcement learning, specifically task exploration, representation learning, and policy optimization in simulated environments. The proposed methods are evaluated exclusively on standard benchmark tasks and do not involve human subjects, personal data, deployment in real-world decision-making systems, or applications with foreseeable societal, ethical, or safety implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kubYZ5gr9I", "reviewer": "Reviewer_EqNm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses an important limitation in existing meta-reinforcement learning methods: even when informative task context is available, learned policies often fail to actually exploit it. The authors argue that this problem arises from two coupled issues—inefficient task exploration and weak optimization on context-conditioned policies to meaningfully use the inferred task information.\n\nThe paper proposes LaSER, a meta-RL framework that explicitly separates task exploration and task exploitation at inference time. During exploration, a dedicated policy $ \\pi_{\\text{explore}} $ is trained to collect a fixed number $K$ of informative episodes. Informativeness is defined through a task reconstruction objective: a small set of episodes is considered useful if it can approximately reconstruct a larger dataset from the same task. This yields a principled, reconstruction-based notion of exploration quality that goes beyond diversity or novelty measures.\n\nTo address failures in context exploitation, the authors introduce a meta-reward that penalizes the task policy for ignoring the inferred task context. Concretely, the method selectively encourages entropy when conditioning on the context does not improve value, pushing the policy to further explore the context–action relationship instead of collapsing to context-agnostic behavior.\n\nEmpirically, the paper shows that LaSER outperforms or matches strong meta-RL baselines across several benchmarks. The experimental section includes  oracle-context evaluations and ablations, which support the claim that both improved exploration and improved context exploitation are necessary for effective few-shot adaptation.\n\n**Strengths**\n\n- Introduces a principled definition of informative exploration grounded in task reconstruction, rather than heuristic notions of diversity or information gain.\n- Clearly identifies and empirically validates an underexplored failure mode in meta-RL: policies ignoring task context even when high-quality context is available.\n- Integrates exploration learning, representation learning, and task policy optimization into a coherent framework.\n- Includes strong diagnostic experiments (e.g., oracle-context evaluations) and ablations that justify key design choices.\n\n**Weaknesses**\n\n- The method relies on a strong inductive assumption that task data lies in an approximately low-rank *linear* subspace with fix rank $K$. While this assumption is central to the reconstruction objective, the paper provides limited evidence that it holds beyond the considered benchmarks. It remains unclear how robust the approach would be in task families with inherently nonlinear or compositional structure, or whether reconstruction quality correlates with downstream adaptation performance in such settings.\n- The exploration budget $K$ is fixed and specified during meta-training, which limits flexibility at test time. In practice, different tasks may require different amounts of exploration, and the framework does not currently allow the agent to adaptively decide when sufficient task information has been collected.\n- The exploration policy and encoder are pretrained before task policy learning, effectively shifting data and computation to earlier stages. This makes comparisons of early-stage sample efficiency with baselines difficult to interpret and would benefit from alternative evaluation protocols (see Requested Changes).\n- Baseline coverage becomes narrower in some experimental sections without explicit justification, particularly when earlier baselines are omitted despite being evaluated elsewhere in the paper (see Requested Changes).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experimental results largely support the paper’s main claims. The authors report performance improvements across multiple benchmarks and include useful diagnostic experiments, such as oracle-context evaluations, that help isolate key failure modes. The presented ablation studies also support the importance reconstruction-based exploration objective and the proposed meta-reward for encouraging context usage. Overall, the empirical evidence is mostly consistent with the paper’s stated motivations and conclusions.\n\nHowever, several aspects of the empirical evaluation would benefit from clearer justification and more careful framing. In particular, comparisons of sample efficiency are difficult to interpret without explicitly accounting for the pretraining of the encoder and exploration policy, which provides the task policy with a significantly more informative initialization during meta-training. This effectively shifts data and computation to an earlier stage and should be discussed more explicitly when making efficiency claims.\n\nAdditionally, in Section 5.2.3, results are reported against only a single baseline, which was not consistently the strongest performer in earlier experiment. Without a clear justification for this choice, it is difficult to assess the strength of the reported gains in this setting. Including additional baselines or explaining their omission would substantially improve transparency. \n\nOverall, these issues do not fundamentally undermine the paper’s claims, but addressing them would make the empirical evaluation more convincing and the overall narrative more complete.", "audience_yn": "Yes", "audience_explanation": "Although the method introduces strong inductive biases like low-rank linear reconstruction assumption and a fixed exploration budget (which is common among many existing meta-RL approaches), the overall framework is well motivated and technically insightful. In particular, the explicit decoupling of the exploration policy and the exploitation policy represents a clear conceptual contribution and helps expose important failure modes in context-conditioned policy learning that are often obscured in end-to-end approaches.\n\nMoreover, the results reported in MEWA show a non-trivial performance gap over the selected baselines, which supports the practical effectiveness of the proposed method. While the assumptions underlying the approach may limit its applicability in some settings, the ideas introduced in this paper are likely to be of interest to the TMLR audience and to stimulate further work on structured exploration and context utilization in meta-RL.", "requested_changes": "1. **Improve clarity and narrative structure.**  \nThe paper would benefit from substantial rewriting to improve clarity and overall narrative flow. While the core ideas are interesting, the current presentation is dense and does not consistently convey the main intuitions. I recommend shortening the main paper and moving secondary technical details to the appendix, with greater emphasis on motivating the problem, explaining key design choices, and clarifying how the different components of the framework fit together.\n\n2. **Clarify baseline selection in Section 5.2.3.**  \nIn Section 5.2.3, only VariBAD is included as a baseline. It is unclear why other baselines evaluated earlier in the paper are omitted in this setting. Providing an explicit justification for this choice, or including additional baselines, would improve transparency and strengthen the empirical comparison.\n\n3. **Clarify sample efficiency trade-offs.**  \nThe paper would benefit from a clearer discussion of sample efficiency. Since LaSER pretrains the encoder and the exploration policy, it effectively shifts computation and data usage to the training phase. Even if this leads to lower overall sample efficiency, such a trade-off can still be valuable in practice. The authors should clarify whether LaSER is particularly helpful in *data-scarce adaptation settings*, where test-time interaction is limited, and explicitly discuss the benefits and drawbacks of this design choice.\n\n4. **Additional ablation studies on the meta-reward formulation.**  \nThe task-policy meta-reward is defined as\n$r_t^{+} = r_t + \\beta\\, w(s_t, c)\\, \\mathcal{S}[\\pi](s_t, c),$\nwhere $ w(s_t, c) $ measures context underutilization and $ \\mathcal{S}[\\pi] $ denotes policy entropy.  \nTo better isolate the contribution of this specific design, it would be helpful to include ablation studies comparing against simpler alternatives, such as\n$r_t^{+} = r_t + \\beta\\ \\mathcal{S}[\\pi](s_t, c),$\n(always-on entropy bonus), and\n$r_t^{+} = r_t - \\beta\\ w(s_t, c),$\n(direct penalty for context underutilization).  \nThese comparisons would help clarify whether the primary benefit arises from entropy regularization, the context-usage signal $w$, or their interaction.\n\n5. **Discussion of structural assumptions.**  \nThe paper would benefit from a more explicit discussion comparing LaSER’s *explicit low-rank linear reconstruction assumption* with the *implicit structural assumptions* made by baseline methods. Such a discussion would help contextualize the inductive biases introduced by LaSER relative to existing meta-RL approaches.\n\n6. An important open question concerns the generalizability of the proposed framework beyond linear reconstruction. While the current formulation relies explicitly on a low-rank linear structure to define the informativeness of exploration data, it would be helpful for the authors to discuss how the approach might extend to settings where task structure is inherently nonlinear, or to clarify the regimes in which the linear assumption is most appropriate.", "broader_impact_concerns": "I do not identify any significant ethical or societal concerns arising directly from this work.", "additional_comments": "Although the framework is built around a fixed exploration budget $K$ which is common in many meta-RL baselines, the overall structure of LaSER appears well suited to more flexible formulations. In particular, the reconstruction-based objective naturally suggests the possibility of adapting or selecting $K$ dynamically, for example based on marginal gains in reconstruction quality. Allowing the model to determine when sufficient task information has been collected could significantly strengthen the approach and broaden its applicability.\n\nFinally, while the paper deliberately separates exploration and exploitation into two distinct phases with two policies, it may be interesting to consider formulations closer to standard reinforcement learning, where exploration and exploitation are interleaved rather than strictly phased. For instance, allowing the exploration policy to continue contributing data during task execution, with the task encoding updated online, could enable more progressive adaptation and reduce reliance on a hard phase boundary.\n\nOverall, these points represent promising directions for future work and do not detract from the paper’s core contributions. However, addressing them, even at the level of discussion, would further strengthen the paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)