Learning Task-Aware Abstract Representations for Meta-Reinforcement Learning
Authors: Louk van Remmerden
Zhao Yang
Shujian Yu
Mark Hoogendoorn
Vincent Francois-Lavet
Venue: Accepted by TMLR ()

Abstract: A central challenge in meta-reinforcement learning (meta-RL) is enabling agents trained on a set of environments to generalize to new, related tasks without requiring full policy retraining. Existing model-free approaches often rely on context-conditioned policies learned via encoder networks. However, these context encoders are prone to overfitting to the training environments, resulting in poor out-of-sample performance on unseen tasks. To address this issue, we adopt an alternative approach that uses an abstract representation model to learn augmented, task-aware abstract states. We achieve this by introducing a novel architecture that offers greater flexibility than existing recurrent network-based approaches. In addition, we optimize our model with multiple loss terms that encourage predictive, task-aware representations in the abstract state space. Our method simplifies the learning problem and provides a flexible framework that can be readily combined with any off-the-shelf reinforcement learning algorithm. We provide theoretical guarantees alongside empirical results, showing strong generalization performance across classical control and robotic meta-RL benchmarks, on par with state-of-the-art meta-RL methods and significantly better than non-meta RL approaches.

Published in Transactions on Machine Learning Research (12/2025)
Learning Task-Aware Abstract Representations for Meta- Reinforcement Learning
Louk van Remmerden Deportment of Computer Science Vrije Universiteil Amsterdam
Lamalbil@w..l
Zhao Yang
z-yangg
Department Computer Science Vrije Universiteit Amsterdam
Shujian Yu
s.yu3@vu.n
Department of Computer Science Vrije Universiteit Amsterdam
Mark Hoogendoorn Department of Computer Science
m.hhoogendoom@uv..ll
Vrije Universiteit Amsterdam
Vincent Frangois-Lavel Lavet.
whccent.franccrilavee.......
Department of Computer Science Vrije Universiteit Amsterdam
Reviewed OpenReview: /ttps://openrevieee net/ pdfp id= 3CWyTh4hJ4
Abstract
A central challenge in mela-reinforcement learning (meta-RL) is enabling agents trained on set of environments to generalize new, related tasks without requiring full policy retrain- ing. Existing model-free approaches often rely encoder networks. However, these context encoders are prone to overfitting to the training enviromments, resulting in poor out-of-sample performance on unseen lasks. To address this issue, we adopt an alternative approach that uses an abstract representation model to learn augmented, task-aware abstract states. We achieve this by introducing novel architocture that offers greater flexibility than existing recurrent network- approaches. addition, optimize model with multiple loss terms that encourage predictive, task-aware repre- sentations in the abstract state space. Our method simplifies the learning problem and pro- vides a fiexible framework that can be readily combined with any off- efi-the-shelf reinforcement learning algorithm. We provide theoretical guarantees alongside empirical results, showing strong generalization performance across classical control and robotic meta-RI. RL benchmarks, on par with state-of-the-art meta-RL methods and significantly better than non-meta RL approaches.
mutext-cconditiona policies learned via
1 Introduction
Achieving robust generalization to new enviromments is challenging for agents trained within a single setting. While such agents may perform well the training environment, they often fail when faced with even minor changes dynamics. Mota-reinforcement learning (meta-RL) addresses this limitation by training across distribution of tasks, thereby enabling the few-shot adaptation of classical RL agents (Beck al., 2025 Nagabandi et al. 2018)

---

Published in Transactions on Machine Learning Research (12/2025)
One of the most promising directions in meta-RL is context conditioning ZZingrrff et al. 2021 Lee et al, 2020). The core idea is to learn a context representation that captures task-specific informat ion from distribution of environments. At test time, the agent can explicitly infer the context of a new, unseen environment and adapt its behavior accordingly. Recent work (Lee et al.] 2020; Rimon et al., 2024) has shown that context-conditionel dynamics models are particularly ellective to meta-RI. seltings.
A key limitation of context- based meta-RL approaches is that the agent's performance depends heavily on the encoder's ability to accurately distinguish between tasks. Consequently, if the context encoder overfits to the training distribution fails to generalize its out-of-sample performance can degrade substantially (Zintgraf et al. 2021
This work aims to improve generalization in meta-RL by leveraging abstract representation models (ARMs), which learn high-level representations of environmental states using a pretrained model. Our goal to produce task-aware, augmentod abstract state representations. Such an augmentod abstract state a latent representation that may include additional task information, analogous to a belief or hyper-state. Accord- ingly, this differs from the more conventional usage in which abstraction typically implies compressed state with reduced information.
We obtain task- states with a neural architecture that encodes states into an abstract space through a state-space encoder and infers task encodings with a recurrent neural network (RNN). These representations are optimized jointly using a set of complementary loss functions. Our approach offers two main advantages: train and generalize well, reducing reliance on the context encoder; and (2) they integrate seamlessly with any off-the-shelf model-free reinforeemen-learriing algorithm, giving the agent direct accoss to task information.
(1) ARMs are straightforward
Our network architecture integrates abstract state representation learning with task inference in unified framework. Unlike previous approaches that rely primarily on task-encoders that are trained separately, we introduce a joint optimization framework based on predictive losses in the abstract space- capturing transitions, rewards, and lask structure.
We call our methodology Environment-aware Meta Encoding and Representation Abstraction for Latent Domains-ENar EMERALD. It is novel reinforcement learning framework that learns task-aware abstract representations capturing both environmental dynamics and lask-specific factors, enabling improved gener- alization and task disambiguation in meta-RL settings Our key contributions are:
EMERALD's performance is par with state-of-the-at performance meta-RL methods across suite of meta-RL benchmarks and outperforms classical RL approaches in few-shot adaptation and transfer tasks, and is compatible with standard reinforcement learning algorithms such as SAC and PPO.
We provide theoretical justification for EMERALD's design, showing how incorporating contextual information into the abstract state space promotes fast generalization to unseen tasks.
2 Related Work
Meta Reinforcement Learning Menfrreenforrement learning (meta-RL) enables RL agents to rapidly adapt to new, unseen tasks environments (Beck et al. 2025 Nagabandi et al. 2018). The goal is to learn policy trained
set of tasks T, p(Turaiu) that can quickly adapt to new tasks. Achieving this requires the agent to generalize evfectively across tasks, leveraging experience from the training distribution to perform well in novel settings (Finn et al. 2017). Meta-RL can also be viewed through the lens of partially observable Markov decision processcs (POMDPs), whore the true task identity is unobscrvod and must be inferred through interaction, making conlext inference analogous to belief state estimation (Humplik el al. 2019 Rakelly et al. 2019).
I.BB Our of "task-aware" follows the meta-RL convention, where the representation is couditioned on au inferred context voctor. In the continual learning litcrature, the same term typically denotes access to an explicit task idcutity (and "tass-agnostie" the absence thereof). 2Code base can be found htttp::/ http://hhhuuccmmm/mmmmmss///aaaa
2

---

Published in Transactions on Machine Learning Research (12/2025)
Context-based Meta-RL Context-based meta-RL approaches learn context encoder to capture variations across lasks. PFART, (Rakelly et al. 2019) and VariBAD (Zintgraf et al. 2021 formulate the context- encoding problem using a Bayesian Adaptive MDP (BAMDP), where a context variable z is modeled as a belief state inferred from past transitions and used to condition the policy, yiclding ((& z). VariBAD passes an augmented s consisting of the current state and the belief variable to the policy. This allows the agent to adapt its behavior based on the inferred task identity. We build on this idea of an augmented state space, but rather than concatenating the state and context, we feed the learned context variable directly into the state-space encoder of the representation model.
Model-based Contextual Meta-RL MAMBA (Rimon et al. 2024) extends VariBAD to the model- based setting by integrating context encoding into Dreamer's latent imagination process. A related method, CaDM (Lee et al. 2020), learns contextualized dynamics models by conditioning both the reward and tran- sition models on context, and employs a backward-anddforwarrd transition mechanism to stabilize training. Our approach builds on these methods, but uses only forward predictions in the abstract space. As Dreamer-based methods such as MAMBA, we allow the policy to operate in an abstract space with lower cardinality than the raw state space, [ < |S x TV. However, our state task encoder also be deployed in model-free settings via abstract representation models. Figure [1] illustrates the high-level architectural differences between our approach and other context- context-based methods.
Abstract Representation Models in Deep RL Abstract representation models (ARMs) are widely used in deep reinforcement learning to improve generalization and sample efficiency (Botteghi et al. 2022 Starre et al.) 2022; Ni al. 2024p. These models map dighnsimensinal states to compact latent spaces X via encoders & > X, enabling more structured learning. Unlike "standard" auto-encoders, abstract representation models in the context of RL are typically used to obtain generalizable abstractions of the reward and transition dynamics. Several works illustrate this: for instance, Zhang et al. (2018) decouple dynamics and rewards for domain generalization, while Li et al. (2021) apply abstract models to adversarial scttings with shared dynamics but varying visual obsorvations. We build on those insights and extend ARMs
to the meta-RL selting. Similar to prior combined RL approaches (François- Lavel el al. 2019 Lee el al. 2020), our method is flexible and can be integrated with both model- free and model- based agents.
Separate Tisk Approach
RNN--ausd Approach
EMERALD
x
h
hg w
RNN
Recurval Network
PC
Pally Connected Layers
Task aware Abstract State
FC
RNN
RNN
FC
Task-Eucoding
State Space
luput
h Full Trajectory up to
Costert Costrat (Racode Laasel
Segaraie Costest (RIN Leated)
Auguctted State Spuce (RMK Eucode)
Figure 1: Comparison of state and context representation strategies. Left: Methods such as CaDM learn state (s) and context (z) representations independently, without temporal modeling. Middle: VariBAD-like approaches use RNNs to learn from trajectory history h1-5 but still treat s and as separate components. Right: Our proposed EMERALD ARM architecture integrates the trajectory-derived context & with the current abstract state to form a lask-aware shared latent state representation x, enabling unified and task- aware modeling of the environment dynamics.
3 Task-aware Abstract Representation Models
Notation We define a task T, as a Markov Decision Process (MDP), represented by the 5-tuple T, = AS 7)), where S the state space, A the action space, P.(8 8,, a;) the transition kernel, and ra the reward function mapping state-action pairs to real-valued rewards. The discount factor denoted 7 (0,1). We index time steps by the subscript I, e.g., 8t and 81+1-
3

---

Published in Transactions on Machine Learning Research (12/2025)
We sample N independent and identically distributed (1..d.) training tasks T, (()). From task, collect data sets of transitions D; = {(SK,t, Ok,t; Sk,++1 7k,+) The combined dataset across all N tasks is denoted D = - D.. We let m, |D;] denote the number of transitions collected from task T. We denote the entropy of random variable X H((),, and the conditional entropy H[(X Z). (: denotes the loss of model with parameters 0 in single environment and (T= the loss over all tasks in the distribution.
Model Components The abstract state space denoted as X' and the task embedding space as 2. Note that, in our architecture, &' contains both the information about the task as well as the state in that lask. The ARM has four components: (1) the contextual state-space encoder &i S x + X, (2) the transition function T : x A X, (3) the reward predictor p : X A + R. Lastly, (4) learned encoder Φ (8 A 1 x + 2, mapping history of transitions to environment embedding Z R² 2 (h) where ho---1 (su, do, 11, $1 01, ......, 8t-, -. 8t). Figure (right) shows schematic overview of the model architecture.
X + A. The value function (*(x) is defined the expected relurn when starting from abstract state r and following policy x.
Policy We denote the policy
V (x) = E
Σ r
The Q-function (z, denotes the expected return when taking action in abstract state 2, and thereafter following N.
= Ex
r'r
2o = x,
Goal The aim to learn a policy that maximizes the expected return across tasks sampled from the task distribution p(T). Formally, we optimizo:
00
max E TT 8,0
E B(mn.ao.ri...).r. JTT
r
where the expectation taken over both the task distribution and the rollout distribution induced by the policy x across a task distribution, operating the abstract state space
3.1 Aware Augmented State- Represeepresntttioonns in Meta-RL
Necessity of task-aware ARMs in meta-RL Despite the capacity of representation models to generalize well in simple meta-RL settings such as variety of different mazes, we argue that only through the addition of task-specific information can their potential be fully realized, a feat well-ostablished in standard meta- RI. literature Beck et al. (2025); Ni et al.] (2024). Within our framework specifically, the intuition is that without task-awareness (mcasured by how much task information is retained by the encoder), the abstract representation model will not be able to learn representation that can differentiale based solely the input it receives. As such, task-agnostic state-space encoders essentially introduce many-to mapping, which collapses into a single point during optimization. We provide an illustration of this insight in Figure
How can task-awareness be achieved? We introduce novel approach that integrates task information directly into the abstract state space X' via the encoder e), producing representations to be both compact and lask-relevant. Unlike prior meta-RL methods that condition downstream components (such as dynamics, reward models, or policies) on context (e.g., VariBAD, PEARL), our method performs early context injection by embedding task identity during the state abstraction phase. This design allows our agent to resolve task ambiguity upstream, providing strong inductive bias and leading more robust representations that generalize better to unseen tasks.
4

---

Published in Transactions on Machine Learning Research (12/2025)
5g - "Ridit"
X3
- "Right"
Xi
vs.
X
Raw States
Optinall Encoding
Suboptimal Encoding
Illustration of the main intuition that onc-to-one task-aware mappings produce better encodings. Raw input stales, where S1 maps to stale S2 in task red and to S3 in task green. Left: Optimal encoding where both follow-up abstract states are mapped to different regions in space. Right: Suboptimal encoding we would expect if the encoder does not retain contextual information. In that case, we may observe a collapse of the two states, because v(sg') and (sa²²) are pushed to the same point in order to minimize the loss.
Figure 2:
By shifting task uncertainty into the encoder, we also simplify the overall architecture and training objective, eliminating the need for amortized inference or latent variable optimization at test time. Furthermore, this abstraction-firs formulation is highly modular and integrates seamlessly with both model-basod and model- free RL algorithms. We formally justify this design through the lens of entangled versus disentangled modeling: in the latter, state abstraction and task representation are learned separately, whereas in the former our setting they are integrated into single representation. We then empirically show that our approach leads to improved generalization across standard meta-RL benchmarks.
Proposition 1 (Entanglement Yields Lower Entropy) Let Ti and To be two tasks such that for some state-action pair (s,, (s), a.), the transitions differ: Pi | St, ) + P2(8141 st, 0z). Let 2 = o(ha,)) E Z be a deterministic task embedding computed from transition history. Consider two model variants: (1) task-state encoder, where x; = <b(s1,2), 2), the transition prediction is given by r+t1 = It + T(1,, and the eslimaled reward fr. - p(xr, and (2) a slate encoder, where I, = =b"((,,, It-1 R It. + (t,;) and the estimated reward fr = p² (æ,, 2). Then, assuming v p and T well as v², p² and T" are deterministic and trained to perfectly fit the transition and reward losses, the total entropy of the representations satisfies:
(()(se,, I (s)()
That is, the task-state version yields lower joint entropy due to resolving task ambiguity in the representation, resulting in simpler transitions and reward functions.
We provide the full proof in Appendix Intuitively, Proposition D states that entanglement causes context ambiguity to be resolved before it reaches downstream tasks (e-g.; before computing transition probabilities or state-to-action mappings via the policy). As such, following a general "conditioning reduces entropy" MacKay 2003), which has been explored extensively in the
rule (Cover & Thomas 1999 Gray 2011
machine learning literature (Bounoua et al.
2025 Pandey & Dukkipati 2017 Shamir et al. 2010), early conditioning effectively ensures that all model components are one-to-one mappings. We call the task-aware rast state space produced by the the state-task encoder the augmented abstract state space.
4 EMERALD: Architecture & Objectives
ARM: Architecture Blocks We introduce parameterized abstract representation model, consisting of (1) state-space encoder v (2) latent transition function т, (3) a reward predictor p, and (4) a transitions-to-contexx function & to adjust for environment-specife dynamics (see Appendix for a more detailed overviow). Formally, we have four parameterized model components:
(st, 2x) -> Ft, F++ = Dt + T(z, ax; 0r), A = p(rr, as 0p), [hhi 0z).
We combine the model parameters into a jointly optimizable model via 0, where 0 (0w, Or, O,r 0z)
5

---

Published in Transactions on Machine Learning Research (12/2025)
Objectives The transition and reward dynamics are effectively captured by the loss functions defined below. We define the transition and reward losses as:
ARM:
[(0) transition
= Elsneneen (I (1 (;))s, (1,)) as 0.:
((0) reward = B(sc p(e,
The formulations above follow standard practice (o.g., François-Lavee et al. 2019 but we introduce context
vectors retrieved from the context encoder and include directly into the optimization objective. To prevent potential state space collapse, we also include regularizer:
[(0)reg exp -xpppcc (-Callbb(st 2;00)
(= - ( +(
where s and are two randomly sampled states and Ca is a constant. Intuitively, this term enforces that any two stales some distance apart.
Putting everything together, we obtain the following joint objective:
(()) [(0)transition
transition C(0)rwward reward - r(0)reg:
where determines the regularization strength. We minimize this objective using standard mean squared error losses for the transition and reward terms. Algorithm | shows the pscudocode for the ARM training process. Throughout the experimental section, sample the offline data for training (line 1 of the algorithm) the model using random policy. However, in practice, the offline data might also be collected via non-random or expert policy. A Python implementation can be found at https://githu:: com/ 1 jjmmabi1/EMERAL
Training of the policy The policy is trained after ARM training. We freeze the reprosentation model's parameters and pass the online transitions through the model. The augmented abstract state approach allows combine the learned representations with any reinforcement learning algorithm (e.g., SAC, PPO). We define the optimal policy
arg max V (18(81, 0,)), Vsa E.
Algorithm 2 shows the pseudo-code for policy training (full version shown in Appendix D
5 Experiments
Our evaluation consists of four complementary experimental sets, each designed to test a distinct aspect of EMERALD. First, we compare EMERALD with strong baselines on vul-of-distribution lasks following the evaluation protocol of Lee et all 22000) Second, compare EMERALD to another set of baselines the more recent Meta- World ML1 and MI10 suites (Yu et al. 2020 Third, evaluate the effect of training multiple environments by measuring performance on various tasks. Lastly, we perform two ablation studies: (i) we examine how the quality of the ARM affects policy learning, and (ii) we visualize the learned task-aware latent space to assess how effectively the policy exploits it.
5.1 Comparative benchmarks
Setting 1: We largely follow the experimental protocol of Lec et al. (2020), using variants of the continuous-
control CartPole lask and several MuJoCo benchmarks (oooorov el al.] 2012). The learning algorithms are trained on a set of configurations and evaluated on previously unseen ones. For CartPole, for example, we train on pole lengths 1.0, 1.5, and 2.0 and test. the out-of-distribution performance on poles of length 0.5 and 2.5. HalfChectah (Volume) HalfChoctah setting where we shrink or expand the Choetah's volume, depending on the exact. treatment. regimen. The three evaluation domains illustrated in Figure
(and

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm EMERALD Abstract Representation Model Training
Require: ARM (transition encoder, reward predictor, abstract state encoder, task-encoder), task distribu- tion p(T), learning rate ag, batch size B, epochs per task Repochs; initialize replay buffers Dj for every
task Tú context horizon H
Offline Data Collection: For each task Ti, collect dataset Do
{((s, at 8141, rr) of offline
transitions following policy
while not converged do
D. ARM training loop
3:
for each task T: do
4:
Reset context window: h +- for epoch = to Repochs do
5:
for each batch ((t, ot 8141, vz)) Doflline do
Sample task embedding 2 66. (h;0ss)
Encode states: 2, (,(5,, 0v) rtt1 = 2;;
Predict transition and reward: B4t1 Is + (1i
10:
Predict reward: f; p(xte az; 0p)
11:
Compute loss: La = Leransition + Lreward + 3 Lreg
12:
Update vp parameters: Og 0s 0x0 Σ B
Vog. fri
Parameter updates
13:
Update & parameters: Os t 0s 00, B i=
B
Vo,Li Li
14:
Update P parameters: 8, 0 as, B Li=1
E
Vo,Li Li
15:
Add to context window:
c e U ((0), 0t, I', 8t41)
until |b*| S H
16:
end for end for
17:
18:
end for
19:
end while
20: return Trained ARM.
Algorithm 2 EMERALD Policy Learning
Require: Trained ARM, tasks Ti p(T), learning rates Oa,, av, batch size B, rollout length nr, updates horizon H; parameters Oz, fy; buffers D, + 0
1: while not converged do
2: for all T do
Rollout
B - 0, h +
for 1... n. do
Φ( h:; 0a); da ('(8, 2; 0p), at no, ( xx)
env step + Tr, Te+1 = (1(411,1 0p)
7:
add (I,, a, (1, to B and h² (truncate to H)
8:
end for D; < UB end for
9:
10:
11:
for 1. ... ny do for all T, do minibatch b² D2; compute L; Liv
Update
12:
13:
14:
end for
15:
Og + as Ve. Li 0V + 0v -o--vvey av E. Li
16: end for
17:
end while
18: return no, Vol

---

Published in Transactions on Machine Learning Research (12/2025)
bbo
Figure Illustration of the Experimental Setup. Left: Cartpole with different sizes, HalfChectah with different volumes. Right: Pendulum with different lengths.
Middle: Three
details are provided in Appendix L.. EMERALD is paired with Soft Actor- Critic (SAC) Haarnoja et al. 2018) (EMERALD SAC) and Proximal Policy Optimization (PPO) (Scullman et al. 2017) (EMERALD PPO). The ARM is pre-trained offline with 10k CartPole transitions, 11x 4.5k Pendulum transitions, and 100k HalfChoctah transitions. In order give our model an unfair advantage, we collected the shows the effect of using an expert
offline transitions under a random policy (ablation study in Appendix policy).
Following Lee et al. 2020). evaluate all methods on unseen environments under two regimes (see Ap- pendix for complete specifications):
Moderate: Unseen test tasks differ only slightly from the training distribution.
Extreme: Unseen test tasks deviate substantially from training task.
In the second setting, we compare EMERALD the same set of baselines on the World ML10 suite Yu et al. (2020). Meta-World is benchmark environment for evaluating multi-task and meta- reinforcement learning algorithms, consisting of diverse collection of robotic manipulation tasks built on the MuJoCo simulator. ML10 includes ten distinct training tasks designed to assess generalization across five test task distributions.
Setting 2:
Setting 3: In this setting, we compare the performance of EMERALD and VariBAD on the HalfCheetah- Direction task, standard benchmark in renfrreenfor learning. Unlike Settings and 2, which test out-of-distribution generalization, this experiment evaluates in-distribution adaptation specifically, how well the agent adjusts to moving in opposite directions (forward versus backward).
We compare EMERALD PPO and EMERALD SAC with ten baselines:
PPO
Schulman al.
2017):
Standard Proximal policy optimization (PPO).
SAC (Haarnoja al.
2018 Standard Soft Actor Critie (SAC).
3. Stacked PPO
Lee et al.
2020 PPO variant that feeds fixed window of past transitions to the
policy.
4. PEARL Rakelly al.
2019 Infers context variable by maximizing expected return.
PPO-EP [hoou et al.
2019 : PPO version that concatenates an embedding from early interactions
to the state.
6. CaDM Lee et al.
2020 Augments forward backward dynamics model with a context encoder.
7. VariBAD Zintgraf al.
2021 : Uses an RNN to produce belief state that is concatenated with passed to both the policy and value functions.
the environment. state and
RMF-RL Ni et al. and hyperparameters.
2021
Recurrent meta-RL model-free approach with optimized architecture
8

---

Published in Transactions on Machine Learning Research (12/2025)
9. RNN-HN (Beck et al.
2023
Recurrent hypernetwork approach for meta-RL.
10. VI-HN Beck et al.
2023 Variational hypernctwork approach for meta-RI.
To identify the effect of the learning algorithm (PPO or SAC), we pair the most important competitors to a learning algorithm || Morcover, to ensure fairness, the ARM is pretrained on offiine data and the corresponding transitions are deducted from each method's overall interaction budget, which guarantees that all agents experience the same total number of transitions (sce Appendix M for the complete allocation budget). We further use the same hyperparmeter tuning process for all baselines (Appendix Ni.
(a) PPO-hased methods
CartPole (Lengths)
Pendulum (Lengths)
HalfCheetah (Volume)
Moderate
Extreme
Moderate
Extreme
Moderate
Extreme
PPO
198.2 + 0.9
187.8 + 4.7
-1113.2 + 69.1
-1356.8 + 48.0
807.7 + 553.6
574.0 + 645.6
Stacked PPO
197.8 +: 1.3
189.2 + 6.1
-475.7 1 228.1
-188.2 1 178.2
361.1 1: 141.7
5.7 = 208.1
PPO EP
196.3 + 4.0
184.5 + 9.7
-374.3
24.6
-256.7 + 26.4
895.3 + 445.1
674.2 + 686.8
CaDM (PPO)
197.9 3.0
193.0 3.5
-279.8
42.1
-426.4 1 227.0
1224.2 1 630.0
1021.1 676.6
VariBAD (PPO)
199.1 + 2.8
192.0 + 3.2
-318.5 + 56.6
-643.1 = 311.4
2964.0 + 179.5
1618.3 251.8
RMF-RL (PPO)
198.3 + 0.9
191.9 24
-505.6 + 42.5
-899.3 = 185.3
2303.5 = 146.3
1037.0 = 268.1
VI-HN (PPO)
197.7 = 2.7
194.5 5.3
421.1
56.8
-53133 = 121.3
1882.1 91.5
1167.9 + 185.2
RNN-HN (PPO)
198.7 + 2.1
193.7 4.1
-44.9
51.0
-97.88 = 117.2
1982.9
88.2
984.2 + 242.5
EMERALD (PPO)
199.1 1.1
199.3 + 3.5
-312.4 + 19.0
-313.7 + 101.5
2942.5 213.5
1401.4 + 421.3
(b) SAC-based methods
CartPole (Lengths)
Pendulum (Lengths)
HalfCheetah (Volume)
Moderate
Extreme
Moderate
Extreme
Moderate
Extreme
SAC
199.2 = 0.3
199.4 t 0.1 402.0 + 131.3
-1274.2 + 133.7
1998.3 + 142.1
177.4 + 321.3
PEARL (SAC)
198.0 1.4
187.5 + 10.9
-645.3 + 320.7
-1136.6 + 251.0
642.1 + 488.3
462.1 + 534.5
VariBAD (SAC)
194.7 4.5
189.6 + 1.7
-1001.3 + 75.1
-1193.8 = 94.1
1755.6 + 194.2
1424.9 + 258.3
RMF-RL (SAC)
198.4 2.1
197.2 3.1
-769.3 + 198.1
-1058.5
238.1
4419.1 + 219.3
2142.2 323.6
VI-HN (SAC)
199.2 3.6
196.2 6.4
-657.4 + 128.4
-1098.6
202.3
4365.4 + 307.9
2744.3 + 439.4
RNN-HN (SAC)
196.3 = 2.3
194.7 + 3.0
-738.4 4: 271.9
-991.3 + 298.1
2837.7 + 218.0
2539.3 + 260.8
EMERALD (SAC)
194.6 = 2.7
196.2 + 4.9 -198.3 + 23.3
-749.4 + 340.9
5101.1 = 151.8
3271.8 + 180.2
Table 1: Performance comparison means over 8 seeds (Mean Return + Std.) for CartPole, Pendulum, and HalfChoctah (Volume). The best statistically significant (Welch's t-test < 0.01) results in each group are boldfaced and quantify the rolling average over the last 100 timesteps.
5.2 Results Benchmark Experiments
Results for Setting 1: Table Ib shows that EMERALD (SAC) surpasses most SAC-based baselines; the corresponding learning curves are displayed in Figures 4a 4b
In the PPO-based setting, we achieve per- formance on par with VariBAD (PPO). On the Pendulum task, EMERALD (PPO) does not outperform PPO-EP in the extreme regime, suggesting a potential limitation when the environment exhibits strong peri- odic dynamics. However, the difference between FMERALD (PPO) and PPO-EP is not large. Notably, both
B.B. For the hypernetwork-bueed approaches (VI-HN and RNN-HN) well VariBAD, could not find any official SAC implementations. In prior work these methods typically paired with on-policy algorithms such PPO, and empirical results with off-policy algorithms like SAC rarely reported. For completeness of comparison, we implemented SAC variants of these baselines ourselves using the existing PPO-hased source code.
Apppendix O contains analysis wherehy treat the learning algorithm hyperparameter.
9

---

Published in Transactions on Machine Learning Research (12/2025)
fendikmn Pecerste- SAC Methodh
Hattheetah SAC Merccds
300
200
Aoceagn bettum
190
120
1085
103
284
2.0C
E3C
038
1P4
ECC
118
(a) Pendulum
(b) HalfCheetah (Volume)
Figure 4: Learning Curves for two out-of-distribution Experiments from the Moderate Regime (rolling averages over 8 independent runs and adjusted for ARM budget allocation)
EMERALD (SAC) and EMERALD (PPO) perform well overall, underscoring the benefit of modularity: one can boost performance by combining the ARM with a different learning algorithm. Interestingly, the perfor- mance of the non-meta RL learners appears to be predictive of performance with EMERALD: for instance, SAC and EMERALD (SAC) both outperform other methods on the HalfChectah tasks. Furthermore, while VariBAD does achieve higher performance when paired with PPO, in the SAC-based setting EMERALD (SAC) achieves much higher performance, suggesting that our approach is more naturally paired with SAC than VariBAD.
(a) PPO-based methods
(b) SAC-based methods
Method
HalfChectah (Direction)
Method
HalfChoctah (Direction)
PPO
-219.7 = 75.2
SAC
-17.99 + 58.2
VariBAD (PPO)
2112.5 + 193.6
VariBAD (SAC)
2806.3 + 118.7
EMERALD (PPO)
2259.9 + 317.1
EMERALD (SAC)
3343.2 + 291.1
Table 2: Performance over 8 seeds (Mean Return + Std.) with random seeds.
Across both Meta- World ML10 and ML1, EMERALD variants generally achieve strong and stable performance compared to cxisting meta-reiinorrcmment learning basclines. On ML10, EMERALD (SAC) altains the highest average performance both training and lesting settings, indicating effective adaptation across multiple tasks. EMERALD (PPO) performs slightly below its SAC counterpart but still matches or exceeds other basclines such as PEARL and RMF-RL on most metrics. Overall, these results suggest that EMERALD's framework contributes to improved consistency and generalization, with SAC providing a performance edge in both efficiency and asymptotic returns. Interestingly, when considering the other baselines, we observe that performance drops significantly when using SAC. For example, VI-HN (PPO) reaches a training score of 41.8%, whereas its SAC counterpart achieves barely half of that. We ob- serve similar trend on the SAC-based versions on ML.1 where for the hypernetworks (RNN-HN (SAC) and VI-HN (SAC)) as well as VariBAD, underperform compared to their PPO-based counterparts. We believe this may be due to the fact that architectures rely on an on-policy, trajectory-consisteni data distribution, while SAC's off- off-policy replay buffer breaks this temporal coherence and destabilizes both latent adaptation and hypernetwork weight generation (sce Appendix N for more extensive discussion). EMERALD, on the other hand, shows a relatively stable performance, regardless of the chosen policy.
Results for Setting 2:
Results for Setting 3: The results are reported in Table 2 and show that the PPO version our approach performs on par with VariBAD (PPO). The improvement over regular PPO further suggests that training with augmented abstract states provides an additional boost in performance in this experimental setting. Similar to setting 1, we observe that EMERALD SAC outperorms VariBAD (SAC).
10

---

Published in Transactions on Machine Learning Research (12/2025)
(a) PPO-based methods
(b) SAC-based methods
Method
Avg (Train)
Avg (Test)
Method
Avg (Train)
Avg (Test)
RMF-RL (PPO)
16.7 + 1.0
13.8 + 0.4
RMF-RL (SAC)
48.5 + 1.4
10.7 + 0.9
VariBAD (PPO)
52.4 + 2.9
16.1 + 1.2
VariBAD (SAC)
16.6 + 0.7
13.1 = 1.1
VI-HN (PPO)
44.3 + 1.9
14.2 + 1.1
VI-HN (SAC)
20.8 + 0.4
14.0 + 0.6
RNN-HN (PPO)
14.7 0.8
12.3 + 0.3
RNN-HN (SAC)
0.0 0.0
0.0 = 0.0
EMERALD (PPO)
46.7 + 1.9
15.3 + 0.9
EMERALD (SAC)
53.9 0.6
17.6 1.0
Performance comparison on Meta- World ML10. Results are reported mean percentages + standard error over 8 seeds.
Table 3:
(a) PPO-based methods
Method
door-open
basketball
window-open
pick-place
button-topdown
RMF-RL (PPO)
0.0 + 0.0
0.0 = 0.0
34 + 1.0
0.0 + 0.0
0.0 = 0.0
VariBAD (PPO)
91 + 2.4
0.0 = 0.0
100 0.0
28 + 2.0
98 + 1.0
VI-HN (PPO)
0.0 = 0.0
0.0 = 0.0
100
33 - 0.3
0.0 = 0.0
RNN-HN (PPO)
0.0 0.0
0.0 0.0
95 4
0.0 0.0
0.0 0.0
EMERALD (PPO)
92 + 0.6
4.0 + 0.9
65 3.8
0.0 0.0
99 1.0
(b) SAC-based methods
Method
door-open
basketball
window-open
pick-place
button-topdown
RMF-RL (SAC)
99 + 0.3
40 = 0.6
0.6
0.0 0.0
98 + 0.7
VariBAD (SAC)
0.0 + 0.0
0.0 0.0
0.0 + 0.0
0.0 + 0.0
0.0 + 0.0
VI-HN (SAC)
0.0 + 0.0
0.0 = 0.0
0.0 + 0.0
0.0 + 0.0
0.0 + 0.0
RNN-HN (SAC)
0.0 + 0.0
0.0 = 0.0
0.0 + 0.0
0.0 + 0.0
0.0 - 0.0
EMERALD (SAC)
100 + 0.0
2.0 = 0.5
64 + 0.4
0.0 + 0.0
99 + 0.6
Table 4: Performance comparison on Meta-Worts World ML1 tasks. Results are reported as mean percentages standard error over 8 seeds
5.3 Ablation Studies
Effect of Environment Diversity on Performance We ask whether training on a greater diversily of tasks is more beneficial than training with more data drawn from a single environment. This is an in- distribution setting, as train and test tasks are the same. We allocate total budget of 3M transitions for HalfCheetah (Volume) and 200k for CartPole. In Configuration 1 the entire budget is spent task, whereas in Configuration 2 the budget is split evenly across multiple tasks. Table summarizes the outcomes. Configuration 2 yields higher overall performance, indicating that exposure to a wider set of environments outweighs simply scaling up data for a single task, demonstrating that increased task-diversity offers greater sample efficiency.
Model Effect on Policy Performance We study how performance varies with (i) the number of training epochs, (ii) the amount of offline data, and (iii) the presence of task-aware encodings. Figure 5 reports the results (each marker is the mean of 8 seeds). In the CartPole setting with 10k transitions, observe slightly higher performance of the task-aware variant. Similarly, for CartPole with 100k transitions, the lask-aware variant oulperforms the lask-agnostic one. In the HallCheetah (Volume) setting, the same pattern emerges but more pronounced: both the task-aware 100k and 10k-transition regimes converge carlier. Morcover, the performance of the task-agnostic (10k) variant effectively zero, which suggests that removing task encodings (i.c., using task-agnostic abstract states) sharply degrades performance, particularly HalfCheetah, demonstrating the effectiveness of using state-task encoder.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Environment
Config
# Training Tasks
Samples per Task
Mean Return + Std
CartPole
1
200k
91.4 + 37.5
CartPole
2
40k
187.0 + 19.8
HallCheetah
1
3M
749.2 + 209.3
HalfCheetah
2
600k
1218.0 + 338.7
Table 5: EMERALD In-Distribution Performance on Training Tasks (Mean Return + Std. over 5 seeds). Mean difference statistically significant (Welch's t-test p < 0.01)
Cari Performence
HallChweiahh Performence
1471
EELTBO CaTTee LC PAAM
Har77 wma Avers)
1600
802
420
200
32
Sperts
Spects
Figure Average Policy Performance (8 scods) with stdev. on CartPolc and HalfChoetah (Volume) for several Training Iterations, starting from model trained with epoch up to 100 epochs. Dashed lines are task-aware iterations, solid lines display aask-agnostic versions (no context encoding).
Latent Trajectory Analysis Figure 6 visualizes policy behavior in both training and test environments. For each environment, we record the first 50 transitions and project them into a three-dimensional principal component space. The resulting training and test trajectories exhibit similar shapes but occupy distinct regions, suggesting that EMERALD learns behaviors that transfer across tasks while remaining sensitive to task identity.
6 Discussion & Conclusion
We introduced EMERALD, a peetaa-reinforrmmee--eeeiiii method that leverages augmented, contexl-aware abstract states. Across benchmarks, our approach either outperforms other model-free context-basse base- lines performs par with statc-of-the art approaches. In particular, identified the combination of EMERALD with SAC hand, when using PPO the performance of EMERAL.D compared to other PPO-based baselines is less pro- nounced. Still, the fact that EMERALD easily pairs with both on-policy and off-policy methods, shows its versatility. Further, relative to its non-meta-RL counterparts, we show that task-aware representation models yield substantial gains.
natural pairing, performing generally better than competitors. On the other
Beyond improved performance, we provided justification for the representational and architectural advan- tages of approach: by disentangling task-relevant features early in the pipeline, simplify training, improve robustness, and gain compatibility with broad class of RL algorithms simple encoder that produces task-aware abstractions. Despite these strengths, EMERALD has limitations. First, the optimal structure of the latent space (e.g., dimensionalitty) can be enviromment-dependee difficult to determine a priorí. Second, as the number of training tasks grows, the latent space may need expand to capture task diversity, potentially inercasing computational cost. In such cases, hybrid ap- proaches that decouple task inference from state abstraction may offer better scalability. Identifying where this trade-off becomes critical is important direction for future work.
procedure yields a
and
12

---

Published in Transactions on Machine Learning Research (12/2025)
Lntent Pejectores CartPale
Latent Trojectoies HultChentah
Labent Trajectortes Penduuuum
Envrovvnt
Dio
50
Figure 6: Visualization of the Latent Space (PCA mapping) of the EMERALD ARM with Training and Test Environments of the first 50 transitions. Left: Cartpole. Middle: HalfCheetah (Volume). Right: Pendulum. In CartPole and HalfCheetah (Volume) the out-of-distribution trajectories are and 6, while in Pendulum 11 and 12.
A natural exlension of EMERALD is to integrate it into a model-based framework. Whereas the present study focuses on using a pretrained latent model within a modular, model-free pipeline, one could instead exploit the latent dynamics directly to train a policy, thereby opening the door to direct comparisons with established model-based meta-RL methods.
Acknowledgments
We thank the reviewers and the editor for their substantial time, valuable and extensive discussions, and the care they dedicated to improving this work. This research was supported by the EU Horizon 2020 project grant No. 96534 (ICARF4OLD).
References
Jacob Beck, Risto Vuorio, Zheng Xiong, and Shimon Whiteson. Recurrent hypernetworks are surprisingly strong in meta-rl. Advances in Neural Information Processing Systems, 36:62121-6211 2023.
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, Shimon Whiteson, ct al. tutorial on meta-rciiforcement learning. Foundations and Trends in Machine Learning, 18(2-3): 224 384, 2025.
Nicolò Botteghi, Mannes Poel, and Christoph Brune. Unsupervised representation learning in deep rein- forcement learning: A review. arXiv preprint arXix:2208. 14226, 2022.
Mustapha Bounoua, Giulio Franzese, and Pietro Michiardi. Learning to match unpaired data with minimum entropy coupling. arXie preprint arxvv:2503 08501, 2025.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. John Wiley & Sons, 1999.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Tn Proceedings of the 34th International Conference on Machine Learning, pp. 1126-1135
PMLR, 2017.
Vincent Frangois-Lavet Yoshua Bengio, Doina Precup, and Joelle Pineau. Combined reinforcement learning via abstract representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3582 3589, 2019.
Robert M. Gray. Entropy and Information Theory. Springer Science & Business Media, 2011.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning, pp. 1861-1870. PMIR, 2018.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mchta, and João G. M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1 (127))11-18- 18, 2022.
Jan Humplik, Alexandre Galashov, Lconard Hasenclover, Pedro A. Ortega, Yee Whye Tch, and Nicolas Heess. Meta reinlorcement learning as task inference. ar preprinl 19055005.66422 2019.
Kimin Lee, Younggyo Sco, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware dynamics model for generalization in model-based reinforcement learning- Tn Proceedings of the 37th International Conference Machine Learning (ICML), volume 119, pp. 5757 5766. PMLR, 2020.
Bonnic Li, Vincent Frangois-Lavet, Thang Doan, and Joclle Pincau. Domain adversarial reinforcement learning. Xix preprint (Xx::2102.000090 2021.
David J. C. MacKay. Information Theory, Inference and Learning Algorithms. Cambridge University Press, 2003.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt in dynamic, real-world environments through meta-reinforreemnt learning. ar preprint Xix::11003. 11947, 2018.
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free rl can be a strong baseline for many pomdps. arXiv preprint (xXi::2100000022 2021.
Tianwci Ni, Benjamin Eysenbach, Erfan Scyedsalchi, Michel Ma, Clement Gchring, Aditya Mahajan, and Pierre-Luc Bacon. Bridging state and history representations: Understanding self-predictive rl. arXic preprint arXte:2401.08890 2024.
Gaurav Pandey and Ambedkar Dukkipati. Variational methods for conditional multimodal deep learning. In Proceedings of the 2017 International Joint Conference Newrol Networks (IJCNN), pp. 308 315. IEEE, 2017.
Antonin Raflin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Re- search, 22(268):1 8, 2021.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta- reinforcement learning via probabilistic context variables. In Proceedings of the 36th International Con- ference Machine Learning, pp. 1331-5340. PMLR, 2019.
Zohar Rimon, Tom Jurgenson, Orr Krupnik, Gilad Adler, and Aviv Tamar. Mamba: an effective world model approach for meta-reinforceemet learning. arXiv preprint 240::440000853 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arXin:1700 nne:1777.0694) 06344, 2017.
Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science, 411 (20-33)22296-2920 -7111 2010.
Rolf AN Starre, Marco Loog, and Frans A Oliehoek. Model-based reinforcement learning with state abstrac- tion: A survey. In Benelux Conference on Arlificial Intelligence, pp. 133 148. Springer, 2022.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. Tn Proceedings of the 2012 TEEE/RSE R/SSJ International Conference on Intelligent Robots and Systems (TROS), pp. 5026 5033. IEEE, 2012.
Geoffrey Van Driessel and Vincent François- Component transfer learning for deep rl based on abstract representations ions. XrXio preprint arXim:2111 11.1222,, 2021.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pp. 1094 1100. PMLR, 2020.
Amy Zhang, Harsh Satija, and Joelle Pineau. Decoupling dynamics and reward for transfer learning (2018).
arXiv preprint Xrxi:: 10689, 2018.
Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta, Environment probing interaction policies. In JCLR, 2019.
Luisa. Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Tgl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmaan,, and Shimon Whiteson. Varibad: Variational bayes-adaptive deep rl via meta-learning. Journal of Machine Learning Reseavch, 22((89):1 39, 2021.
15

---

Published in Transactions on Machine Learning Research (12/2025)
A Notations
S: State space.
A: Action space.
T; The i-th task, defined as an MDP (S, A, P..
Pi(84+1 81,a)) a)): Transition kernel for task Te.
a)) Reward function for task Tz.
e [0, 1): Discount factor.
p(T): Distribution over possible tasks.
p(Tuain); p(Tret): Training and test distributions (subsets of p(T)).
T p(T): Sampling the i-th task from the task distribution.
D; = [Si,k Ci,k, 11,611 F= Dataset of transitions for task Ti with my transitions.
D - Dj: Combined datasot across N tasks.
0: Parameters of the abstract representation model or policy, depending on context.
X: Abstract (latent) state space.
Pr(xe+1 (xa, ux): Transition kernel in the latent space for task Tr.
क T:-- + A: A policy mapping from abstract states to actions.
Tet A parameterized policy with parameters 6.
2); State encoder mapping raw state 8, and context z to abstract state It E N.
((haz): Context encoder that produces a task embedding z E 2 from trajectory history.
Context (task embedding) space.
L(f0): Ti): Loss of the ARM on task T.
C(0; T): Loss of the ARM a set of tasks T. If clear from context, write C()) to denote the expected loss over p(T).
Ž(; 7):
Empirical loss of the ARM over a set of tasks T.
F: Function class.
+ Parameter space.
H(X): Entropy of random variable X.
16

---

Published in Transactions on Machine Learning Research (12/2025)
B
Definitions
Definition 1 We define task a Markov Decision Process (MDP) via the following 5-tuple
T. = (s.A,P
where:
S denotes the state space,
A denotes the action space,
Pi(4++1 St, az) is the transition kernel for the task T3 ~ p(T),
(s, is the reward function, i.e. ri : A
7 E [0, is the discount factor.
Definition 2 We define four parameterized model components:
(()))200)) + It. F++1 = + (II, as: 0r), Å. = p(xs, as; 0p), = @(re 0²),
where: . S + X' contertual encoder mapping high-dimensional states S S R" and task embeddings S R& to abstract states X C 12,, T x A X is a transition model predicting latent state deltas, p : X x A + R predicts rewards in abstract space, and Φ (S x A x RR x S Z is contert encoder mapping past transitions to task embedding, where parameters O = (0.r, Or, 0, 0z)
We draw N i.i.d. training lasks To ~ p(T). Each task To provides dataset {18,,e Ci,t, 81,4 r(,)) } Denote the combined data by
D = {To 8i,, aes, 81441, riss) i== -i= 1,... N; m,
Definition 3 For each task Te, define the transition and reward losses:
Ciransition (0; Ta) = E( [1|6 (11411)411:0)) (s);0s))) + T (n;
Creward(0; T:) Bl |1re (1)(sr))))
For a scalar X 0, the lotal task-specific loss
C(0: Ti) Ltransttion Tt) + X Arewannd Tı).
We collect the parameters 0 (9 07,8
To simplify notation, we denote the expected loss over all tasks as L(0) rother than L((B;)).
Definition 4 Given Markov Decision Process T, (S.A,P,, 7), a policy is mapping To S A [0, 1]. We have nt Ti 8t), which defines the probability of selecting action when in slate 8t- A deterministic policy is special case where the mapping reduces to TT :S + A, meaning that for each state there exists a single action a = (st). A stochaslic policy assigns a probabilily distribution over actions, meaning that for any state St, the probabilities over actions sum to 1.
We assume there a distribution T over possible tasks in- dexed by Te. Each lask: To define stochastic mapping from (st,0t) 0z) to (81+1,7e).
Assumption (Existence of Distribution)
17

---

Published in Transactions on Machine Learning Research (12/2025)
Definition 5 Given Markou Decision Process (MDP) = (S,A PaTrar)) and policy n, define the value function of policy mapping V :S + R such that:
(st) = E, Σ (s,, ax) 100000000000000 10====000 =
Here, the expectation is laken over the trajectory induced by following T( 8x); where the state transilions follow the transition dynamics P..
Definition 6 Given a Markov Decision Process (MDP) T: AS,A, ?2) and policy n, defone the state-action value function (Q-function) of policy π mapping QT A R such that:
(st) = E,T
Σ 11=0
s0 0,
Here, the expectation is taken over the trajectory where the initial action is a in state s, and subsequent actions are selected according to (a, 8c), while state transitions follow P..
18

---

Published in Transactions on Machine Learning Research (12/2025)
C Propositions and Lemmas with Proofs
Proposition 1 (Entanglement Yields Lower Entropy) Let Ti and T2 be two tasks such that for some state-action pair (st, at), the transitions differ: Pi(s+41 0z) + P2(s141 st, 01). Let 2 = ((ha,a) E Z be a deterministäc lask embedding computed from transition history. Consider two model variants: (1) a task-state encoder, where 2, = ()s,,2), 2), the transition prediction is given by Itt1 ss St + T(2,, at) and the estimaled reward fr. - p(I,, ue); and (2) a slate encoder, where It. = ()t(),,, ht-1 Rs It. + 0),2) and the estimated reword fr p² (2,,a,, 2). Then, assuming v, p and т well as 4², p and T are deterministic and trained to perfectly fit the transition and reward losses, the total entropy of the representations satisfies:
((,()(,))) I (((()(())).))))
That is, the lask-state version yields lower joint entropy due lo resolving lask ambiguily in the represenlation, resulting simpler transitions and reward functions.
Proof 2 Let s and 2 be random variables representing the state and task, respectively. Let v : S x 2 + X be a task-aware encoder, and 4/" : - X' a task-agnostic encoder.
Assume there erists a deterministic function g such that
- - (8))
that is, the task-aware encoding can be obtoined by transforming the joint variable 2)), We refer to the left-hand side as the early-conteat encoding (direct mapping to the abstract state-task space) and to the right-hand side as the late-content encoding (post-hoc adjustment within abstract state space).
Since g a deterministic function applied to (v* ()), the chain rule for entropy gives:
(s))))s), = H(g(V (s), + H((+ 100)) (s).
Because conditional entropy is nonnegative, we have
(I(( s (s)," 2).
Substituting 2), = (()((())++))))))))))(()) we obtain
II((((,))) S H(v H(v z).
Equality holds if and only if g is injective (i.e., informationn-prssruunn) on the support of (s)*()), 2). Other- wise, any many-to-on to-eeto-oneeeeo mapping g strictly reduces entropy.
Hence, the early (lask-slate) representation carmol have higher entropy than the late (slate + lusk) repre- sentation; it resolves task ambiguulyy directly in the encoded space, yielding lower representational complerity and simpler transitionreeaar dynamics. D
Remark 1 ( Conditions for strict inequality) In practice, our archilecture will oflen have striclly lower entropy than downstream task-aliasing, because it operates directly in the state task space, avoiding the need for dounstream readjustment. To illustrate this, consider the following.
We have
[H((b(s, 2)) < H(b
Equality holds if and only if 1b(8, z) and (10* (s); 2) are one-lo-one, that is, if ((, can be perfectly recovered from (s), and vice versa.
As an exomple, consider two tasks Ti and Tz that differ in their transition dynomics from the some state- action pair (81, Right):
Ti
81 To 83.
81
82;
19

---

Published in Transactions on Machine Learning Research (12/2025)
If the encoder 4 collapses the two follow-up states into the same abstract state, equality still hold provided that. the tosk embedding con perfectly disambiguate this collapse.
However, we get striet inequality once x induces a many-to-one mapping (state aliasing) and canmot fully restore the lost information (e.g., due to overfitting on the training environments). In this case, the early task-aware representation 16(8, 2) carries strictly less eatropy, reflecting more compact and disambiguated encoding of the task-conditioned dynamics.
Remark 2 (On Collapse and the Role of the Assumption and Regularizer) The entropy inequal- ity follows from subadditivity. However, it can be trivially satisfied i collapses. However, the combination of the next-state encoder and the reward predictor will prevent this from happening. We additionally ensure that the represenlation encodes nontrivial information about the state. Practically, we enforce this via regularizer:
[(0) = exp (-Callbbs*,=:;00 - 1888 0z;
where st and s- randomly sampled states. This encourages separation in latent space and mitigates representational collapse.
20

---

Published in Transactions on Machine Learning Research (12/2025)
D Appendix D: Policy Algorithm
Algorithm 3 EMERALD Policy Learning
Require: Trained ARM Se, tasks T; p(T), learning rates On, ov, batch size B, number of gradient updates Rupdates: number of environment steps per rollout Prollout: policy parameters 0, and value/ critic parameters 8y, per-task replay buffers D, + 0, context horizon H
while not converged do for each task T, do
Policy learning loop
> Data collection
Reset temporary buffer: B <- 0 4: Reset contezt window: c² + 0
5:
for step = to Prollout do
6:
Get ct from recent transitions; sample 8~ ~ o(2 d(;0))
Obscrve state 8y; encode 2t (8),, 2pp)
Sample action de Ta, (- (-1
Execute a, in Tú observe (8)+1:
10:
Encode next state r2+1 = (88441,
11:
Store (It, rt, xf41) in B
12:
Update context: e +- U [(8). Ot, Tr, 8411) until lell H
13:
end for
14:
Update replay buffer: D DB end for
15:
16:
for update- to Rupdates do
Optimization step
17:
for each task T, do
18:
Sample minibatch b'= {(2,, Ti, $1-1)) Dj
19:
Compute task- specific policy loss L poolie Compute optional value loss Ln end ford, + 0 - ae Vo, Σ, -policy
e.g. PPO, SAC, REINFORCE.
20:
21:
22: Update value network: Oy < Ov ov Vav E, Lialue
23: end for
24:
end while
25: return Trained policy Te, and Vey
21

---

Published in Transactions on Machine Learning Research (12/2025)
E Policy Algorithm Details
The goal of reinforcement learning is to learn a policy that - on the basis of the current state takes the action that maximizes the cumulative reward. In our setting, moreover, the "raw" input states are mapped to the learned latent space. Formally, the optimal policy satisfies:
arg max V (1)( AsES,
where V (1( E [E (-r(e) (2e, ax) rn = 2]
PPO PPO a policy gradient method that improves stability by constraining policy updates. maximizes the clipped surrogate objective:
I (0x) = (nin(r((
where re(0,)
0x) Ta ( [+))) is the probability ratio and A, the advantage function estimated via
Generalized Advantage Estimation (GAE). The value function Vo(vh(se;; 0x)) is updated using:
Lv(o)= = E, (Va((((;;0)) - yargett
The final objective combines policy loss, value loss, and entropy regularization:
L(0,, L(0x) +c
where (mx)) encourages exploration.
SAC SAC is an off-py actor-critic method that optimizes not only for high expected returns but also for high-entropy policies, encouraging broader exploration.
The maximum-entropy RL objective is:
a Eg Es ~ps (1(0) aa (
0=
where H denotes the entropy of the policy, and a is temperature parameter that balances exploration vs. exploitation. In practice, a can be fixed or automatically tuned.
SAC maintains two Q-functions, Qa and Qo, to reduce positive bias the target updates. Each Qa, is Icarned by minimizing the mcan-squared Bellman error:
Lo, (0.) ( (Qo. (b(8)))))a)) vi)a].
where D is the replay buffer, and
Y/ = (e) ur) +
E. Emticp min Qoee (V(sc+ Op)s ac+1) log v (ac#1 (0)) 1i1
Here, Qotarx are larget networks (periodically updated copies of Qa,), and na is the current policy.
The policy To is updated by minimizing the Kullback Leibler divergence between To and exponential of the Q-function. In practice, the policy loss is often written as:
Tog (6) = EENDD (a a log (at ((8))0))) - Qo. (wis) (v(s):0)) 0)).
22

---

Published in Transactions on Machine Learning Research (12/2025)
Because there are two Q-functions, the minimum of Qo, and Qo is typically used in practice to reduce overestimation bias.
To automatically adjust the trade-off between exploration and exploitation, SAC can learn the temperature a. The corresponding loss is:
I(a)= ED,DD arnit [ log To (ss;; H.
where H is target entropy. Decrcasing reduces the entropy bomus and focuses more on reward maxi- mization, while increasing encourages broader exploration.
SAC alternates between:
1. Minimizing the Q-function losses LQ, (61) and (Q2)
Minimizing the policy loss (Ø).
3. (Optionally) Adjusting by minimizing L(a).
Thus, the SAC objective be summarized
2
M(a) (if a is learned),
LSAC Io, T+(
1o
(otherwise).
This encourages policies that achieve high returns while maintaining sufficient exploration through entropy maximization, all in the learned latent space (z))
23

---

Published in Transactions on Machine Learning Research (12/2025)
F
Comparative Table Methodologies
Feature
EMERALD
CaDM
PEARL
Context Conditioning
Implicit
Explicit
Explicit
Latent Encoding
State-space
Context
Context
Encoder model needed
No
Yes
Yes
Policy Input
Task-aware abstract state space &'
S and
S and 2
Encoder model needed
No
Yes
Yes
Model Based or Model Free
Hybrid
Hybrid
Model-free
Table 6: Feature Comparison Between Different Methods. X' denotes compressed representation of S.
24

---

Published in Transactions on Machine Learning Research (12/2025)
G Abstract Representation Model Architecture
We implement an abstract representation model designed for meta-reinforeement learning across distri- bution of (asks T/ ~ p(7). The model maps observations E S and actions a E A to latent abstract representations 3t E X, and infers task embeddings 2; E 2. Fach component of the model is parametrized by submodules of
State Encoder v0, S X The encoder v maps observations 8, and task embeddings 2; to abstract states Zt. It includes skip connections to promote identity preservation and stabilize representation learning.
Transition Function To, X A The transition model predicts the next abstract state 2++1 given the current abstract state and action de- It is implemented deep MLP and is context-agnostic, relying on the encoded state to contain relevant task information.
Reward Predictor Pe, :X x A + R The reward model estimates the immediate reward ri based on the abstract state and action.
Task Encoder Φ9g; higt + Z An LSTM-basod encoder & that produces task embodding 2, Z from the interaction history hitt (Sp, du, f1, $1,41, ,t)...) This embedding captures task-specific dynamics for each Tr
Terminal Classifier Bo X' A + {0,1} A binary classifier that predicts episode termination given the abstract state and action. Note used in current setup, but available in the implementation.
Discount Decoder An auxiliary module that estimates the discount factor Y E [0, 1] from (xt, a.). This can be used during value prediction in planning.
Inference Procedure each time step I, the model operates as follows:
1. Compute the task embedding: = @(h4,z)
Encode current and next states: 2; = (i(( Ie+1 (14411,2))
3. Predict the next abstract state: le+1 = T(I,, az)
4. Predict the reward: f, p( 0t)
5. Predict terminal status: de 3(I,,
The model is trained end-to-end using supervised objectives on transition, reward, and termination predic- tion. Training minimizes the expected loss across the task distribution:
BT--p(T [C,(0)]
25

---

Published in Transactions on Machine Learning Research (12/2025)
H
Additional Experiments
Structured Representation Learning for Structurally Similar MDPs
Suatiing 3iaiss and Woxt staues bcater Hutt with kanakiors taleds
Suating btaies loxt sta scacel Hot with kanaiorr btate Laleds
0.40
Sext Statrs
0477
048
0236
0222
0350
677
0790
Figure 7: Reconstruction of an "arrow" from two Figure 8: Reconstruction of a maze-like structure MDPs.
from three different structural MDPs
To evaluate EMERALD's ability to produce meaningful and interpretable latent structures of abstract representation models- we adapt the visual experimental setups of Van Dricssel & François- 22021) and Frangois-Lavel et. al. (2019) to a setting with multiple underlying MDPs (and thus multiple environments). Our objective is to assess how well EMERALD can learn a shared abstract representation across distinct environments.
property
We design two simple, interpretable MDP structures. In the first setting, we sample 300 transitions from two deterministic MDPs:
: + S2 S3, To S + S4 S2 S' - Sa-
The transitions embedded into 2D latent space. As shown in Figure EMERALD's ARM effectively captures the shared structure between the two environments in this space.
In the sccond setting, we consider three distinct 5x5 maze-like environments. For cach mazc, we collect 10,000 transitions under a random policy, generating random walks. The maze layouts are generated randomly, with a fixed start state at (0, 0) and goal state at (4,4). 4). visual overview of the layouts is provided in Figure g the resulting 2D latent representation (Figure S EMERALD successfully captures shared structural elements across the three mazes: similar states in different environments are closcly aligned, illustrating the utility of the learned abstract representation.
S
G
G
G
Figure 9: Three Maze Environments
26

---

Published in Transactions on Machine Learning Research (12/2025)
Effect of Input Policy Type Performance
For fairness, we pretrain the model with random policy. However, to illustrate the effect when the oflline data is of better quality (i.e. follows a non-random, but rather expert policy), we conducted small ablation study with the PPO version of our model. The results are shown in Figuro
A Improvement HalfCheetah (Vol.)
A Improvement HalfCheetah (Dir.)
EMERALD PPO
214.6 + 129.1
431.4 + 161.9
Table 7: Performance improvement (A Mean Return Std.) of EMERALD PPO relative to basclines on HalfCheetah Volume and Direction tasks.
27

---

Published in Transactions on Machine Learning Research (12/2025)
Additional Meta-Word Suite Experiments
Table 8: PEARL (SAC) baseline on Meta-World ML1 tasks.
Method
door-open
basketball
window-open
pick-place
button-topdown
PEARL (SAC)
0.0 + 0.0
0.0 + 0.0
36 = 1.1
0.0 = 0.0
43 + 2.5
Table 9: PEARL (SAC) baseline on Meta- World ML10.
Method
Average (Train)
Average (Test)
PEARL (SAC)
23.1 + 1.5
13.4 1.5
28

---

Published in Transactions on Machine Learning Research (12/2025)
Abstract Model Hyperparameters
Table 10: Training Model Parameters
Parameter
HalfCheetah (Vol)
HalfCheetah (Dir)
Pendulum (Length) CartPole (Length)
Weight Decay
0.00001
0.00001
0.00001
0.00001
Hidden Dim
128
128
32
32
Latent Dim
10
10
Model Learning Rate
0.0001
0.0001
0.0001
0.0001
Hidden Dim P
128
128
Output Dim P
1
1
Context Dim
4
4
12
Environment ID
HalfCheetah 5
HalfCheetah
Pendulum
CartPole
# Envs
2
11
Table 11: Training Agent Parameters
Parameter
HalfCheetah (Vol) HalfCheetah (Dir)
Pendulum (Length)
CartPole (Length)
Policy Network LR
0.0001
0.0001
0.0001
0.0003
Policy Hidden Dim
64
64
64
64
Gamma
0.95
0.95
0.99
0.99
Reward Weight
1
1
Loss Weight
1
Regularization Weight
0.01
0.01
0.01
0.01
Train Epochs
300
300
3000
50
Max Iterations
1000
1000
1000
200
History Length
100
100
100
100
Max Episode Steps
1000
1000
200
200
Policy Iterations Learn
5,000,000
5,000,000
2,000,00
2,000,00
Samples per Task (ARM Training)
100,000
100,000
4,,500
10,000
Batch Size
256
256
256
256
29

---

Published in Transactions on Machine Learning Research (12/2025)
J Training Details for the Different Environments
General Details We follow the experimental setup of Lee et al. (020), reproducing their results with the same environment modifications using the code base available at https:///ithub. http//a....oooooggggoooo/aaa/aal For each environment, we train on N training configurations and evaluate on several out-of-distribution (OOD) test environments. Results are averaged over runs with different random seeds. The setup is illustrated in Figure I0] While the conceptual design is the same, we modified the implementation to be compatible with OpenAl Gymnasium v5, which has built-in support for MuJoCo-based environments.
We implemented the EMERALD abstract representation model in PyTorrc (Python 3.9). RL agents were implemented using Stable Baselines3 (SB3) (Raffin et al., 2021) and CleanRL (Huang et al. 2022). Exper-
iments were conducted on an Apple M4 processor with 10 CPU cores and 10 GPU cores. All results are reported as the mean over runs with different random seeds.
30

---

Published in Transactions on Machine Learning Research (12/2025)
K Implementational Considerations
OOD Comparative Experiments We sample offline data using a random policy to ensure that our model is not positively biased by policy optimization. This setup arguably places our model at a disadvantage compared to methods that benefit from task- or goal-directed data collection. We sample at most 1M training data points, divided by the number maximum steps per episode and the number of available training environments. Full training details are provided in Table | For basclines, we re-ran the publicly available code from https://giii https///gitubbb..coouoggggo/o/// yonggyoouggooeo///// CCDM without any modifications.
ID Comparative Experiments We compare in-distribution performance against existing methods. For VariBAD and PEARL, we report values as recorded in the original papers (Zintraa et al. 2021
Task-Diversity Experiment For each environment, we train models (ARMs) under two configurations, averaging results over five runs, while keeping the total number environment interactions fixed: 200K for CartPole and 3M for HalfCheetah. In Configuration 1, each model is trained on a single task using the full sample budget. In Configuration 2, the same total number of samples is distributed across five tasks i.e., 200K / for CartPole and 3M/ / for HalfCheetah rerduuccetrrrrducinngg rreuucing per-task data while increasing task diversity.
Ablation Study We evaluate the model under four different configurations:
1. 100K samples with a context encoder on CartPole (Volume).
10K samples without a context encoder on CartPole.
3. 100K samples with a context encoder on HalfCheetah (Volume).
4. 10K samples without context encoder CartPole.
We trained each configuration for up to 100 epochs to analyze the effect of model training on downstream per- formance. After pretraining, we evaluate by fine-tuning the policy on a single larget environment: CartPole with pole length 1, or HalfCheetah (Volume) with scaling set to 1. We fine-tune for 10K steps in CartPole and 100K steps in HallCheetah. We deliberately keep fine-tuning short to avoid masking the effects of model pretraining since longer training might allow some seeds recover good performance regardless of initialization, thus obscuring the contribution of the model.
Latent Trajectory Analysis For each environment, we extract the first 50 latent transitions from both training and test. episodes. In CartPole and HalfCheetah (Volume), this yields 7 distinct clusters (5 from training environments and 2 from test environments), while in Pendulum (Length), we obtain 13 clusters (11 training, 2 test). By design, the initial training transitions are collected from largely untrained policies, resulting in greater variability within the corresponding latent clusters. In contrast, test. clusters reflect more stable bchavior, as they are derivod from trained policies. To visualize the latent space structure, we apply principal component analysis (PCA) to reduce the dimensionality and plot the trajectories in 3D.
This form of latent space analysis has been used in prior work to examine representation quality and clustering behavior in RL and meta-RL. RL models (c.g., Rakelly et al.) 2019 Zintgraf et al. 2021
31

---

Published in Transactions on Machine Learning Research (12/2025)
L Environment- Varritttnns
We follow the general structure of Lee et al. 2020
The CartPole task involves balancing pole moving cart by applying discrete forces to the cart.
CartPole
Observation: (Iz, 2t, t2, Bs), where x is the cart's position and 0 is the angle of the pole from the vertical.
Action:
{0, 1], where pushes the cart left and pushes it right.
Reward:
ft 1
reward of 1 is given the cart remains within =2.4 and the pole within =14;; otherwise Push force and pole length / are modifiable.
Modifications:
Pendulum
The Pendulum lask aims to swing rod upright using continuous torque.
Observation: (cos 0, sin 0, 0), where 0 e [-x, n), with f 0 being upright. E 2.0,, representing continuous torque.
Action:
Reward:
= - (0) + 0.16f + 0.001a²)
Penalizes deviation from the upright, angular velocity, and torque magnitude.
Modifications: Pendulum mass m and length can be changed.
HalfCheetah (Volume) The HalfCheetah is a planar robot designed to learn fast and energy-effiiient running.
Observation:
20-dimensional vector including joint positions/velocities,, root joint state (exclud- ing x), and torso center-of-mass velocity.
Action:
E [-1.0, 1.0 torques applied at six joints.
Reward:
Aorso 0.05/|az||
Rewards forward velocity and penalizes control elfort.
Modifications: Scale the mass of every rigid link by a fixed factor m and the damping of every joint by a fixed factor
HalfCheetah (Direction) HalfCheetah is a planar robot with 9 links and 6 actuators, designed to move along the x-axis. The agent is trained to move either forward or backward depending on the task direction.
Observation: 17-dimensional vector containing positional and velocity information of the joints and torso.
Action: [[,,1]6, representing the torques applied at each joint.
32

---

Published in Transactions on Machine Learning Research (12/2025)
Extreme
Moderate
Training
Moderate
Extreme
Extreme Moderate
Training
Moderate Extreme
Figure 10: Illustration of the Experimental Setup. We train on N intermediate training environments. At test time, we test the model's performance in a test regime that is moderately different and one that differs extremely from the training environment.
Reward:
d d, 0.05||aa|1
where d E {-1,1} specifies the target direction (backward forward). The reward encourages movement in the desired direction and penalizes large control inputs.
Modifications:
1. Scale the mass of each link by a factor 2. Scale joint damping by a factor d
33

---

Published in Transactions on Machine Learning Research (12/2025)
Episode
Length
200
200
1000
1000
Table 12: Environment Modifications used for Comparative our Experiments (following Lee et al (2200)
lE (0.15, 0.20, 0.80, 0.85)
E {0.20, 0.40, 1600, 1,00, 1.80)
le (0.20, 0.40. 60, 1.80)
m. E (0.20, 0.30, 1.70, 1.70, 1.80)
d (0.40. 0.50, 1.50, 1.60} d (0.20, 0.30, 30.00,1..1.. 1.70, 1.80)
f e (2.0, 17, 18.0)
Test (Extreme)
d 1.0)
lE (0.25, 0.30, 0.70, 0.755
m {0.50, 0.70, 1.30, 1.50}
1e (0.50, 0.70, 1.30, 1.50)
m E (0.40, 0.50, 1.50, 1.60)
f 33,0,3.5, 16.5, 17.0)
Test (Moderate)
de d (-10, 1.0)
1.00, 1.05, 10, 1.15, 1020 1.25)
m (0.75, 0.80, 0.85, 0.90,0000000. 0.95,
1.0, 1.05, 1.10, 1.15, 20, 25)
m E (0.75, (.875,0.85, 0.85, 1.00, 1.15, 1.25)
d E (0.75, 0.85, 1.00, 1.15, 1.25]
8.0, 8.0, 9.0, 9.0,10.0 10.0
E (0.75, 0.80, 0. 85, 0.90, 0.95,
f 6.00 7.0, 11.0, 12.0, 13.0, 14.0, 15.0)
lE (0.40, 0.45, 0.50, 0.55,0660
e {-100
Training
HalfCheetah (Dircetion)
HalfCheetah (Volume)
Environment
Penduluu
CartPole
34

---

Published in Transactions on Machine Learning Research (12/2025)
M
Budget Allocation
Environment
Total Number of Transitions
ARM Training
Policy Training
HalfChectah
5M
500K
4.5M
Pendulum
500K
~50K
450K
Cartpole
500K
50K
450K
Table 13: Sample breakdown across training phases for each environment.
35

---

Published in Transactions on Machine Learning Research (12/2025)
N Additional Considerations regarding Baselines
We aimed to reproduce the basclines as faithfully as possible by relying directly on the authors' publicly available implementations. For each method, we used the original source code for the networks and policies whenever feasible. To ensure fairness in comparison, we applied the same hyperparameter tuning protocol to both our method and the baselines. Specifically, for every method we selected the pest-performing discount factor X 0.9, 0.95, 0.99 and learning rate 10=3,10~4,10-
Hyperparamcter Tuning:
Observed Discrepancies: We also verified the extent to which our experimental results deviate from existing literature. For Setting 1, our findings largely align with prior work (most notably Lee et al. (2020)). The same holds for the HalfCheetah (Direction) experiment in 2 although we were only able to validate the VariBAD (PPO) results using Zintgraf et al. (2021) and could not find any reported performance for VariBAD (SAC) in this setting. In Scttings 2 and 3. the PPO-based results are generally consistent with the values reported in the original papers (e.g., Beck et al.
(023);; Ni et al. (2021)). A notable exception is the VI-HN (PPO) result from Beck et al. (2023), which in our experiments performs substantially better than RNN-HN. Finally, most hypernetwork-bassll SAC approaches perform significantly worse than their PPO counterparts in the MetaWorld settings. We could not find any prior litcrature demonstrating strong SAC performance in these settings, which may indicate that others have altempted SAC without success.
We further highlight the following implementational challenges: (1) It was not always possible to run the source code as is; in several cases we had to modify or debug the implementations to make them functional. This was particularly true for RMF-RL Ni et al. (2021) and RNN-HN Beck et al.. (2023). (2) Another difficulty and potential source of variation stems from the need adapt the baselines to Gymnasium environment version (1.1.1). Most original implementations rely earlier versions of Gym and/or MuJoCo. Nevertheless, our VI-HN (PPO) and RNN-HN (PPO) results closely match those reported in Beek et al. (2023) (see Section 8). (3) Finally, we maintain strict distinction between PPO- based and SAC-based methods. However, many official repositories provide complete implementation for only one of these families (usually only the on-policy PPO). For example, the original VariBAD codebase lhttps://githul http://ww..ommm///tngg/a/a/////e (aarriggrrraaaaaaaaaaaaalaa includes PPO and A2C implementations but no SAC version. For this reason, we implemented most of the SAC variants ourselves. While we managed to do so within our available time and resources, we emphasize that integrating SAC into PPO-oriented codebases could likely be improved, though doing would require methodological design changes beyond the scope of this work. Still, we did test whether SAC-based approaches such as VI-HN (SAC) worked in single-setting problems and found that we obtained reasonable results there, indicating that there are almost certainly no large bugs in the code and that breakdowns only occur when the method needs to generalize to the meta-Rl RL, setting.
Additional Considerations:
36

---

Published in Transactions on Machine Learning Research (12/2025)
O
Averaged Performance for SAC and PPO
The table below show the avrage performance over both PPO and SAC.
CartPole (Lengths)
Pendulum (Lengths)
HalfChectah (Volume)
Moderate
Extreme
Moderate
Extreme
Moderate
Extreme
VariBAD (Avg.)
196.9
190.8
-659.9
-918.5
2359.8
1521.6
RMF-RL (Avg.)
198.35
194.55
-637.45
-978.9
3361.3
1589.6
VI-HN (Avg.)
198.45
195.35
-539.25
-814.95
3123.75
1956.1
RNN-HN (Avg.)
197.5
194.2
-591.65
-744.55
2410.3
1761.75
EMERALD (Avg.)
196.85
197.75
-255.35
-531.55
4021.8
2336.6
Table 14: Averaged performance between PPO and SAC versions (Mean Return).
37

---

Published in Transactions on Machine Learning Research (12/2025)
P Learning Curves
Performance on Cart Train
CartPule Train Methaes
CartPole Methods
200
275
75
F70
DAC
50
(PPO)
SUEA, (SAC
25
Vi-1t (MNC)
VSH (ACI
STOI
50K
100K
150K
200K
250K Timestops
300K
300K
AD0K
<30K
300C
30K
100K
150K
200K
250K
300K
350K
COOK
450C
500K
Time stess
Figure 11: CartPole: Training (rolling averages over independent runs)
Performance on CartPole Moderate
CartPole Moderate PPC Matheds
CartPole Mccierate SAC Methods
175
150
125
200
Stecked ITO
Virbdo (9O1
KAYDAD I540
SHEROLD 1962
Timestaps
Timesteps
Figure 12: CartPole: Moderate (rolling averages over 8 independent runs)
Performance on CartPole
CartPole Extreme Methods
CartPole Extreme SAC Methods
175
300
75
CaD9 BPC)
POAIL 19455
COHAAD ISAD
25
Nn. Hi. 09011
1NC -N 1966)
DISSSOLD 1990
(36))
53K
100K
150k
200K
2506
300K
360K
400K
4S0K
SD0C
50K
.00K
150K
2006
250K
300K
350K
<0OK
450c
500K
Timestegs
Figure 13: CartPole: Extreme (rolling averages 8 independent runs)
38

---

Published in Transactions on Machine Learning Research (12/2025)
Performance on Pendulum Train
Pendulum Train PPO Methods
Pendulum Train SAC Methods
-400
-000
-600
-000
-800
-100
-1000
Sacoe 790
-1700
-1200
VentKD (PAC)
B0T-8L 4L, 770
1140
VerDAD
SFE- (SK)
VolRt
ENDRADD (TF)
DMDRALO (SAC)
250C Timesteps
Timesteps
Figure 14: Pendulum: Training (rolling averages over independent runs)
Performance Penduluu Moderate
-00
400
-000
-1000
-1000
CacM (VC)
EP
-1200
SAC
RAAL ISAC
-1200
1400
Vi-6 (PPO
HT (ACC)
630 (MS)
UBC PMA
-1600
12
50K
100K
150K
200K
250C
SIIK
150K
ADOK
450K
500
50K
100K
150K
200C
2500
BOOK
Y50K
4B0K
4N/IC
500K
Timestops
Timestops
Figure 15: Pendulum: Moderate (rolling averages over 8 independent runs)
Performance on Pendulum Extreme
Pendulum Extreme PPO Methods
Pendulum Extreme SAC Methods
400
800
-1200
Sucod
-1200
-3400
Kedkk (Mc)
KardAD 1540
100 1900
Sac
-1400
-300
DNSAALD (9-0)
DMGRALD SUC)
Timesttps
Timesteps
Figure 16: Pendulum: Extreme (rolling averages over 8 independent runs)
39

---

Published in Transactions on Machine Learning Research (12/2025)
HalfCheetah
HelfCheesan Train PPO Methods
HalfCheetah Train Methods
3000
2500
5000
2000
1500
4000
CaBN
POAIL I540
K41060 (990)
CHHRAD IA4O
Hi 1PPC)
ASC 1M62
(993)
0-
(U3C)
0.5M
2.3M
201
2.5M
301
3.591
400
0.5M
14
1.3M
255
2.5M
391
3.5M
4.59
Timesteps
Timesteps
Figure 17: HalfCheetah (Volume): Training (rolling averages over 8 indeeenneent runs)
Performance HalfCheetah Moderate
HallCheetah Methods
HallCheetah Metnods
3000
2000
2000
3000
2000
2000
800
DAC
C60M (5K))
300
1000
397-8L 9901
TUTA, 15R3
vith 1760
VHB (SC)
2Mh-
0.3M
1.5M
2.5M
3M
3.5M
4M
4.3M
3H
0.5M
1M
1.51
2.5M
3.M
4.5M
5M
Timesteps
Timestops
Figure 18: HalfCheetah (Volume): Moderate (rolling averages over independent runs)
Performance on HalfCheetah Extreme
HallCheetah Extreme
PPO Methods
HallChestah Extreme
SAC Methods
3000
1000
2500
1300
500
Sacort RP0
VerBAO (PCC)
VertAD 1540
BOY RL (90)
SUPIL 350
IMPRAID (SAC)
Timestops
Timestops
Figure 19: HalfChoctah (Volumo): Extreme (rolling averages over 8 independent runs)
40

---

Published in Transactions on Machine Learning Research (12/2025)
Q
Latent Space Visualizations
(a) CartPole (PCA)
(b) Pendulum (PCA)
(c) Half Cheetah (PCA)
(d) CartPole (t-SNE)
(e) Pendulum (t-SNE)
(f) Half Cheetah (t-SNE)
Figure 20: Latent space visualizations across environments using PCA (top row) and t-SNE (bottom row).
41


Reviews:
[{"review_id": "Zm6MqQ97Dz", "reviewer": "Reviewer_cNgU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary of Contributions\n\nThis paper proposes EMERALD, a meta-reinforcement-learning method that introduces task-aware abstract representations to improve generalization across tasks. The approach integrates task inference directly into an Abstract Representation Model (ARM) composed of a state encoder, transition model, reward predictor, and task encoder. By conditioning the state encoder on task embeddings, the model produces latent representations that are both compact and context-sensitive. I particularly appreciated the ARM framework, which cleanly separates representation learning from policy optimization — a design that feels general, reusable, and valuable beyond EMERALD itself. The authors provide theoretical justification showing that task-aware and entangled representations lead to lower loss and entropy, and demonstrate empirically that EMERALD achieves competitive results on meta-RL benchmarks.\n\n\n## Strengths\n\t•\tThe methodology is coherent and well grounded. The motivation for early task-conditioning is clear, and the architecture aligns naturally with the theory.\n\t•\tThe ARM framework makes intuitive sense and provides a flexible foundation for future extensions or alternative implementations.\n\t•\tThe paper is clearly written and emphasizes modularity, making the approach compatible with standard RL algorithms such as PPO and SAC.\n\n### Weaknesses\n\t•\tThe evaluation setup feels somewhat dated. The experiments largely follow Lee et al. (2020), focusing on CartPole, Pendulum, and HalfCheetah variants — benchmarks that are no longer particularly informative in 2025.\n\t•\tThe empirical evidence is limited in scope; while the results are positive, it is difficult to view success on these simple environments as a strong test of generalization.\n\t•\tThe set of baselines is incomplete", "claims_and_evidence_yn": "No", "claims_explanation": "The main limitation lies in the evaluation setup, which is narrow and dated. The experimental protocol reproduces Lee et al. (2020) and relies on simple continuous-control tasks such as CartPole, Pendulum, and HalfCheetah. These benchmarks are not representative of the complexity expected in contemporary meta-RL research, and results on them cannot be taken as strong evidence of generalization in 2025.\n\nIn addition, the set of baselines is incomplete. The most recent baseline included in the paper is from 2021, which leaves out several years of progress in meta-RL and representation learning. This significantly weakens the empirical comparison, as it is unclear how EMERALD would perform against more modern architectures or methods.\n\nFinally, the absence of a recurrent model baseline is a major omission. Prior work (e.g., Ni, Eysenbach & Salakhutdinov, 2021) has shown that a simple RNN-based policy can be surprisingly strong — often competitive with or superior to more complex meta-RL methods. Including such a baseline would provide a much fairer and more informative evaluation of EMERALD’s true benefits.\n\nOverall, while the methodology is sound, the experimental evidence does not convincingly substantiate the paper’s claims.", "audience_yn": "Yes", "audience_explanation": "While meta-RL has become somewhat less central in recent years, there remains an active subset of researchers focused on understanding task generalization and representation learning across tasks.", "requested_changes": "## 1.\tModernize the evaluation setup.\nThe current experiments replicate Lee et al. (2020) and focus on simple benchmarks (CartPole, Pendulum, HalfCheetah), which are no longer informative or convincing. The authors should include more challenging and diverse tasks that better reflect the current state of meta-RL evaluation — for instance, Meta-World, DMControl Suite variations, or other modern continuous-control benchmarks. \n\nGiven recent trends, it could also be valuable to demonstrate the approach in LLM-based reinforcement learning. This would make the empirical section more relevant and impactful to today’s audience.\n\n## 2.\tUpdate and expand the baselines.\nThe most recent baseline currently included is from 2021. The comparison should be extended to include more recent meta-RL and representation-learning methods. In particular, the authors should add the recurrent model baseline from Ni, Eysenbach & Salakhutdinov (2021), which shows that a simple RNN trained end-to-end can be highly competitive — sometimes outperforming specialized meta-RL approaches. This would provide a much fairer and more informative test of EMERALD’s claimed benefits.\n\nAlso, the notion of task-awareness is somewhat misapplied in this work. In standard terminology, task-aware typically refers to settings where the task identity is explicitly provided to the model, rather than inferred from interaction. In EMERALD, however, the agent still operates in a task-agnostic regime — it must infer the task from experience rather than observe it directly. The terminology should therefore be revised or clarified to avoid confusion, as the current phrasing overstates the degree of task observability in the proposed setup.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lxouwC2Teg", "reviewer": "Reviewer_aQ3c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a task-inference method for meta-RL that makes use of trajectory reconstruction to learn the latent representation. In order to improve out-of-distribution generalization, the method proposes that 1) states and tasks be integrated in the model architecture before other inputs (e.g. actions) and that 2) the reconstruction loss predicts transitions over these \"abstract\" states (similar to a belief state in a POMDP). The paper adds an additional objective to prevent representation collapse and trains the encoder upfront on purely random data. The method is motivated by two theoretical claims and is evaluated on a MuJoCo task, Cartpole, and Pendulum. The key strengths of the paper are its fair comparisons (e.g. starting their method accounting for the data collection and using a random policy), its theoretical motivation, and the number of baselines. I have some concerns about the choice of baselines, interpretation and potential inaccuracy of the theoretical results, and reliance on a random policy, which are elaborated in the rest of the review.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are mostly accurate, with notable concerns listed here.\n1. Proposition 1 seems to motivate the fact that sufficient statistics for the meta-RL problem must have a non-zero amount of information about history. It is already known that a history-dependent policy is necessary for such problems (Beck et al., 2025), and this observation does not seem to add any motivation for this method over the baselines. Is there another point to this proposition?\n2. Proof 2 may have a mistake. I am not entirely certain, and would like clarification. A) In proof two, you cannot directly apply the subadditivity inequality H(X,Z) < H(X)+H(Z) because you are comparing H(\\psi(s,z)) to H(\\psi*(s)) + H(z), not H(s,z) to H(s) + H(z). It seems that the point of Lemma 1 and 2 is that \\psi preserves entropy (if one-to-one, Lemma 2), whereas \\psi*, as a stochastic one-to-many mapping, may increase entropy. But unless I am missing something, Lemma 1 only applies when the input variables are the same. In this case, aren’t the inputs to \\psi and \\psi* different? Is your point that H(\\psi(s,z)) = H(s,z) (since ψ is one-to-one) and that H(\\psi*(s)) can exceed H(s) because ψ* is stochastic? If so, then the argument that a one-to-many mapping has “higher entropy” is only valid for conditional entropy H[Y|X]: stochastic mappings increase the uncertainty of the output given the input. However, this does not imply that the marginal entropy H(Y) is larger; adding noise can increase or decrease H(Y) depending on how the noise interacts with the input distribution. Since your proof seems to compare marginals like H(\\psi(s,z)) and H(\\psi*(s)), I’m not sure the conditional-entropy argument applies directly. Could you clarify exactly where conditional vs. marginal entropy is being used, and how you are applying Lemma 1 and 2? B) Moreover, isn’t the more relevant comparison H(\\psi(s,z)) to H(\\psi*(s), z), since in the architecture you concatenate the task representation with z? Comparing to H(\\psi*(s)) + H(z) only gives a loose upper bound, whereas the joint entropy H(\\psi*(s), z) directly reflects the representation actually used. If this is the intended argument, could you clarify why you introduce the looser sum bound rather than comparing the joint entropies?\n3. The baselines all seem to be other task-inference methods. It would really add a lot to compare to methods trained end-to-end on the policy objective, since those have recently been shown to achieve comparable or better performance -- Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs (Ni et al., 2022) -- and some do so by encoding the state in a way dependent on the task -- Recurrent Hypernetworks are Surprisingly Strong in Meta-RL (Beck et al. 2023). Within the task-inference literature itself, it would be nice to compare to methods that use multi-task (pre-)training to learn only the relevant information from the task — e.g., Learning adaptive exploration strategies in dynamic environments through informed policy regularization (Kamienny et al., 2020) and Belief Agent (Humplik et al., 2019). \n4. It is unclear which methods, other than Vanilla SAC, use SAC, and it is unclear what is meant by \"Vanilla PPO\" and \"Vanilla SAC\". PPO and SAC, to me, only specify the outer-loop. Do these methods use an RNN? In what ways are they similar to, and different from, EMERALD?\n\nLess critical, but other issues that detract from a convincing paper:\n5. Less critical, but a separate ablation for the architectural design (encoding state and task before the action) and the latent state reconstruction loss (as opposed to on the MDP state), would help to disentangle the contributions of each.\n6. I am not convinced that using random policy data to train the task-inference module will generalize to other domains, and it also may account for the difference in results from VariBAD. For example, even in some toy T-Maze environments, a significant amount of exploration is needed before seeing a transition that would distinguish the tasks. More difficult baselines (e.g., Meta-World or manually designed) would help make this more convincing. At the same time, I wonder whether training on a random policy, on the domains where it is effective, improves transfer, and it would be nice to see a version of VariBAD that likewise trains on only offline data.", "audience_yn": "Yes", "audience_explanation": "Meta-RL is a popular field and these results are certainly relevant. The improvement on the HalfCheetah domain is substantial (although I do wonder how much of the improvement is coming from SAC vs the EMERALD method, and baselines could be stronger). Overall, I would certainly say that some individuals in TMLR would be interested.", "requested_changes": "Critical for acceptance:\n1. As mentioned above, proof 2 may have a mistake. If I am not incorrect about this mistake, then it would require correction. Note: I only checked proof 2, and not proof 1, in the appendix.\n\nBorderline:\n2. Could you clarify the motivation for proof 1 over the known fact that history is required to know the task in meta-RL? \n3. Could you clarify Vanilla PPO and SAC?\n4. Could you add end-to-end baselines (if this is not accounted for in Vanilla PPO and SAC)\n\nWould strengthen the paper:\n4. Task-inference baselines with multi-task pre-training.\n5. Separate ablations for the architecture and objectives, as mentioned above.\n6. More challenging environments.\n7. VariBAD trained on random offline data.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "49uY1a7Ivl", "reviewer": "Reviewer_99cF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new framework for meta-reinforcement learning (meta-RL), which consists in learning a task-aware abstract state representation that encodes information both about the state and about the task at hand. This is achieved by conditioning the abstract state representation not only on the current state but also on the MDP trajectory, since the latter conveys critical information about the task. In order to ensure that the discovered abstract representation is meaningful, the authors introduce several loss terms that capture various aspects of the MDP, such as the transition and reward functions, while ensuring that the learned representations are non-trivial (i.e., avoiding representation collapse). The framework is generic enough to be used with any off-the-shelf RL algorithm (such as PPO or SAC). The authors conduct an experimental study with standard meta-RL benchmarks to demonstrate the performance of their framework.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper proposes an interesting framework, but I have several concerns regarding a number of points:\n- Is the framework totally model-free? The authors implicitly learn the transition and reward functions, as part of training the abstract state representation model. In Algorithm 2, the authors opt for policy learning, but given the framework also learns a state transition model as well as a reward model, the approach could in principle also be combined with model-based RL methods? I was unclear as to how the authors position their framework, but I feel it has strong elements of model-based approaches (especially as far as Algorithm 1 is concerned).\n- Even though I understand where the authors where the authors are trying to get at with Proposition 1,  found the exposition quite confusing and possibly not so rigorous. First, the authors assume a task embedding $z$ of the form $z=\\phi(h_{1:t})$ in the statement of Proposition 1. Under such an assumption, where the embedding depends on $h_{1:t}$, we expect that the mutual information between the input $h_{1:t}$ and the embedding $z$ will be strictly positive, unless we are in some weird pathological case. Actually, $h_{1:t}$ contains state $s_t$, and at the very least we expect $z$ to depend on $s_t$. So, I do not see how the mutual information term in Proposition 1 could be zero. Even if $z$ does not depend on the history leading up to $s_t$, it should at least depend on $s_t$ (which is contained in $h_{1:t}$). So, unless we are in some strange pathological scenario, the mutual information is expected to be positive. But even putting that aside, the proof is more of a sketch. A rigorous proof, for instance, would assume some embedding for case 1, and would show how modifying this embedding for the pair $(s_t,a_t)$ with different next state $Ss_{t+1}$ would result in a strictly lower loss. And I'm not even sure a separate proposition is needed. Perhaps the authors could just provide some intuitive arguments as to why conditioning on the task history is meaningful, without introducing extra propositions.\n- I also found the proof in Lemma 2 confusing. In part (2) of Proof 3, the authors claim that the encoder cannot differentiate between tasks, leading to a collapsed latent representation. But then they go on to claim that the mapping $s_t\\to x_t$ is one-to-many (not many-to-one). So, I think they need the opposite claim, i.e., that the mapping may actually map a given state $s_t$ to many different $x_t$ (the opposite of collapse. Again, I feel the proofs are not sufficiently rigorous and mostly resemble proof sketches.\n- In Proposition 2, could the authors detail their derivation? $\\psi(s_t,z)$ essentially corresponds to one random variable (that depends on two other random variables). But the term $\\psi(s_t,z)$ itself is not two random variables. Could the authors then explain how they applied the general entropy inequality?\n- Regarding the experimental evaluation, I'm not sure I was able to fully comprehend the results. First, it seems that there is so much variability depending on whether SAC or PPO is employed. I understand that some difference is to be expected, but it was very extreme. This was true in both the vanilla variants as well as the EMERALD variants. I'm not sure if the authors had any idea on that, but it would be nice to shed light on this. The reason why I believe this to be important is that what the authors ideally need to show is that the superior results are due to their framework, not due to the employed RL algorithm. To be more specific, if we look for example at HalfCheetah in Table 1, VariBAD in fact performs almost identically to EMERALD PPO in the moderate setting, and in fact outperforms EMERALD PPO in the extreme setting. It is of course true that EMERALD SAC beats VariBAD by a large margin in both settings. But, given the very large variability, it is not at the end of the day clear what causes the performance lift: is it the new framework or the RL algorithm? Notice that similar observations are true for CartPole and Pendulum.\n- To make things worse, in Table 2 Emerald PPO performs similar to VariBAD, but this time EMERALD SAC exhibits very bad performance. What causes such a huge degradation from Table 1 to Table 2 for EMERALD SAC in the HalfCheetah environment? I understand one is out-of-distribution testing and the other is in-distribution setting, but the swing is too wild. I was unable to make sense of these results.\n- I was unclear what Table 3 was trying to measure. Is the measured performance on only the tasks that the model was trained on? Focus for example on CartPole.  Config 1 has 1 training task, while Config 2 has 5. But is the mean return measured on 1 and 5 tasks, respectively, or on 5 tasks for both configurations? If it is the latter, then of course having more tasks should boost performance. \n- I was unable to follow the arguments in Section 5.3 regarding Figure 5. The authors state that \"In the CartPole setting with 10k transitions, returns plateau after roughly 10 epochs\". Looking at Figure 5, how is that the case? Performance keeps increasing as we increase the number of epochs beyond 10 or even 60. I have no idea how Figure 5 shows a plateau. Similarly, the authors claim \"Although the 100k-transition regime converges earlier, it does not exceed the final performance of the 10k model\". But in HalfCheetah in Figure 5, task-aware performance in the 100k-transition regime is certainly higher than than in the 10k regime. It is as if the authors were referring to an entirely different figure.", "audience_yn": "Yes", "audience_explanation": "Meta-RL is a significant research area with several real-world applications. Furthermore, the paper explains how it is related to the existing RL literature and how it tries to advance it. In that sense, I feel the work is definitely relevant to a subset of the TMLR audience.", "requested_changes": "I'd appreciate it if the authors could address all points raised above with proper clarifications. I would also have some additional questions:\n- Have the authors tried more diverse tasks? It is not clear whether the proposed framework could handle for instance situations where the task distribution is much more diverse.\n- How did the authors ensure that the performance comparison was fair? The authors mention in the Appendix that \"For VariBAD and PEARL, we report values as recorded in the original papers\". Was this the case for both OOD and ID settings? I totally understand the difficulty of implementing all methods and conducting all experiments, but then one must also ensure that the comparison is fair and meaningful.\n- What was the purpose of the in-distribution experiments? In practice, aren't these algorithms expected to be used predominantly in out-of-distribution settings? Do we even need meta-RL for the in-distribution setting? One might claim that conventional RL can already deal with this case, assuming we have trained on all tasks?", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)