Efficient Cross-Episode Meta-RL
Authors: Gresa Shala
André Biedenkapp
Pierre Krack
Florian Walter
Josif Grabocka
Venue: ICLR 2025 Poster (2025)
Keywords: meta-reinforcement learning
transformers
in-context learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: We introduce Efficient Cross-Episodic Transformers (ECET), a new algorithm for online Meta-Reinforcement Learning that addresses the challenge of enabling reinforcement learning agents to perform effectively in previously unseen tasks. We demonstrate how past episodes serve as a rich source of in-context information, which our model effectively distills and applies to new contexts. Our learned algorithm is capable of outperforming the previous state-of-the-art and provides more efficient meta-training while significantly improving generalization capabilities. Experimental results, obtained across various simulated tasks of the MuJoCo, Meta-World and ManiSkill benchmarks, indicate a significant improvement in learning efficiency and adaptability compared to the state-of-the-art. Our approach enhances the agent's ability to generalize from limited data and paves the way for more robust and versatile AI systems.

Published as a conference paper at ICLR 2025
EFFICIENT CROSS-EPISODE META-RL
Gresa Shala', André Biedenkapp', Pierre Krack², Florian Walter? & Josif Grabocka²
'University of Freiburg, Germany University of Technology Nuremberg shalagecs uni-freiburg... de
ABSTRACT
We introduce Efficient Cross-Episodes Transformers (ECET), a new algorithm for online Meta-Reinforcemee Learning that addresses the challenge of enabling reinforcement learning agents to perform effectively in previously unseen tasks. We demonstrate how past episodes serve rich source of in-context information, which our model effectively distills and applies to new contexts. Our learned algorithm is capable of outperforming the previous state-of-the-ard and provides more efficient meta-trane weta-training while significantly improving generalization capabili- ties. Experimental results, obtained across various simulated tasks of the MuJoCo, Meta-World World and ManiSkill benchmarks, indicate a significant improvement in learning efficiency and adaptability compared to the state-of-the-art Our approach enhances the agent's 's ability to generalize from limited data and paves the way for more robust and versatile Al systems.
Meta-
INTRODUCTION
Reinforcement learning (RL) (RL;,Sutton SSutton & Barto 2018) has made rapid progress in recent years and
has shown success in complex real-world benchmarks (Bellemare et al. 2020 Degrave al. 2022
Kaufmann et al. 2023) that go beyond typical (video) game playing benchmarks (Mnih et al. 2015a
Schrittwieser et al. 2020)) Despite these high-profile success stories, RL has not yet found widespread
adoption in many woaldworld applications. This is partly due to the high computational demands (Cobbe et al. [2000] Shala al.) [002]] 2023a) and the brittleness of RL algorithms (Henderson et al. 2018 8018|1 Engstrom et al. 2020 Andrychowicz et al. [202)), which is further amplified by the
difficulty of optimizing the hyperparameters of RL algorithms Faarkrr-Holder et al., 2022] [Fimer]
et al. 2023 Mohan et al., (023). More importantly, RL training pipelines are generally not built with
generalization in mind (Kirk et al. 2023).
The lack of generalization capabilities is crucial factor that limits the applicability of RL in domains where there are no perfect simulators. The classical RL training pipeline involves training, validating, and testing in the exact same task. However, in reality, RL agent's 's sensors might slightly drift over time or tasks might exhibit drastically different features during different stages an episode (such as, e.g., times of the day). If the training task was not set up with such changes in mind, an RL policy will not have the opportunity to learn generalizable behavior. Consequently, small variations in the task that might not have been observed during training can lead severe failure cases. For example, a policy that was trained to steer a robot in a highly controlled lab environment might fail the instant is supposed to leave the familiar lab environment.
Hospedales et al. 2022) is a subfield of machine learning that aims take advantage of previous experience as efficiently as possible so that learned behavior can be transferred to similar problems. One promising approach within meta-lcarning is to include meta- features directly in the training procedure (Benjamins al., [023 [eukman et al. 2023 Shala et Arango et al.) 2024), enabling policies capable of zero-shot generalization based on observed
Meta-learning [Vanschoren] 2019
2023b;
meta-features. These meta-features are either hand-crafted (Benjamins et al. 2023 Beukman et al.)
2023
Arango et al. 2024) or learned (Shala et al. 2023b In the Meta-RL setting, it might not always be possible to have access to meaningful crant-d meta-features to learn generalizable policies. That is why, most work meta-features for Meta-RL RL focuses learned mnetuffettress (Duan et al. 2016; Rakelly et al. 2019; Zintgraf et al., 2220;; Melo 2022 Grigsby et al.) 2024a).
These meta-features generally learned through the use of recurrent neural networks (RNNs)

---

Published as a conference paper at ICLR 2025
Meta-Training Tasks
Push
Reach
Window
Diawer Crose
Sweep
Peg Insert
Bascefoell
Figure I: Illustration of the setting of the ML10 Benchmark in Meta-World. We evaluate ECET in each task and collect these frames. On the left the tasks ECET is trained on, whereas the right we show 3 frames (beginning, middle and end of the episode) from evaluating on of the 5 test tasks. Though these tasks are present in the training set, ECET manages generalize and successfully close the door and sweep the block into the hole.
DDan al. 2016 Zintgraf et al. 2020), amortized inference (Rakelly al. 2019) transformers [M22 Grigsby et al. 2024a). For instance, Raskely et (2019) infer a posterior over meta- features by sampling transitions from the sequencing of these transitions. Thus, the meta-features generated have limited information on the task dynamics. On the other hand, RNN- and transformer-base methods process entire sequences of recent transitions spanning few episodes to infer meta-features. This enables such methods to also incorporate dynamics information the meta-features. Due to the vanishing gradient problem and the limited memory capacity of RNNs, the meta- feetures generated by RNN-based methods often emphasize characteristics of recent transitions, focusing primarily on intra-episode experiences. Meta-RL approach that generates meta-
task without explicitly considering the order of
Similarly, Melo (2022) propose TrMRL, transformer-be features by taking as input a sequence only T most recent transitions. Grigsby al. (2024a) with AMAGO further increase the amount of task information processed to generate meta-features by utilizing a transformer architecture capable of processing longer sequences of transitions spanning multiple episodes (i.e., cross-episode experiences). While exhibiting impressive performance in Meta-RL,, both TrMRL and AMAGO present substantial computational overhead with a runtime complexity depending quadratically on the complete sequence length processed.
In this paper, we present novel method for learning meta-features through efficient cross-episode Meta-RL using novel transformer architecture. Our approach enhances the adaptability of RL agents across diverse sets of tasks by leveraging both intra-episode and cross-episode experiences, providing a more comprehensive learning process. Our learned algorithm capable of outperforming the previous state-of sfate-of-the-asd the-art art and providing more efficient meta-training in terms of timesteps, while significantly improving generalization capabilities. To foster reproducibility, we provide our code at htt.op/s/tthhno..cceeeeeaaaaaaap
2 RELATED WORK
Reinforeifrr Learning (Meta-RL)) emerged as a framework aimed at making RL algorithms more sample-efficient by leveraging knowledge learned from previous tasks speed up learning for the current task. Meta-RL optimizes a bilevel optimization problem during the meta-training phase. The inner loop typically consists of traditional RL algorithms, while the outer loop represents meta-learning algorithms that exploit information from the inner loop to improve performance across multiple tasks. Once the meta-training phase is complete, only the inner loop algorithm is used in
Throughout paper, we use term cross-episode describe interactions across multiple episodes, which is synonymous with inter-episode. This choice aligns with the terminology used in the title for consistency For recent survey, see Beck 2023)
2

---

Published as a conference paper at ICLR 2025
the adaptation phase. Due to the meta-training phase, the inner loop algorithm can achieve good performance using fewer interactions with the new task compared to starting from scratch.
Finn et al. (2017) proposed the Model-Agnostie Meta-Learning (MAML) algorithm, which can be used with any policy gradient algorithm as the inner loop to learn an initialization for the policy to speed up learning on
test task. Duan et al. (206)) introduced RL. approach that learns reinforcement learning algorithm using RNNs, which share the hidden state across episodes from the same task. RL² uses a policy gradient algorithm as the meta-learning algorithm in the outer loop, with learning during the adaptation phase occurring only within the RNN's dynamics, without gradient adaptation. Another approach, PEARL. proposed by Rakelly et al. (2019). uses an off- policcy RL algorithm, Soft Actor Critic (SAC) (Haarnoja et al. 2018). in the outer loop. and augments the state with stochastic meta-features of the task in the inner loop. An MI.P generates the parameters of the posterior distribution of task meta-features.
Zintgraf et al. (2020) proposed VariBAD, a variant of RL. 2 for Bayes-adaptive deep RL, which uses VAEs to learn approximate posterior distribution over task features to augment the state input to the policy. VariBAD uses Proximal Policy Optimization (PPO) (PPO; |Schulman et al. 2017) as the outer loop algorithm, but it can be replaced by any policy gradient algorithm. RL (Bhatia cet al. 2023) builds on RL2 by incorporating action-value value estimates as additional input to the RNN, enhancing its capacity to capture task features.
Under the umbrella of meta-RL, there has also been work on augmenting the meta-training task set to improve generalization (Lee & Chung 2021 Rimon et al. 2022). These methods can be used with meta-RL algorithms based on task feta-reatures to improve generalization.
The transformer architecture (Vaswani al., 2017) has revolutionized the field of Natural Language Processing (NLP) and beyond. Its reliance on self-attention mechanisms enables it to process entire sequences of data simultaneously, enhancing efficiency and handling global dependencies in data compared to RNNs. Parisotto et al./ (2020) explored the application of transformer architecture in RL settings, addressing issues such as data efficiency and overfitting. Subsequent works, including the Decision Transformer (Chen et al. 202T treated RL as a sequence modeling problem, leveraging the scalability and simplicity of transformers to efficiently handle RL tasks, departure from conventional RL methods that typically rely on value function estimation or policy gradients. Panner et al. (2021) extended this concept, showcasing how transformers can unify various RL tasks into a single sequence modeling framework, promising improvements in offline RL algorithms. Extensions such as MGDT (Le et al., 2022) and Gato (Reed et al., (022) showcased the ficxibility of transformers in multi-task and mudti-modal RL, though they required expert demonstrations or fine-tuning for unseen tasks. Offline RL works use datasets of collected trajectories and a supervised prediction loss to train the transformer. This means that the learning process is based on fixed dataset of experiences and that the model does not interact with the task during training. Although effective for certain scenarios, this approach might not adapt well dynamic or unpredictable tasks where new experiences differ significantly from the training data.
Despite their promise, stabilizing transformer training in RL has proven challenging (Mishra et al. 2018 Parisotto et al. 20200 Melo 2022 Grigsby et al. (2024)). Recent work has proposed stabilizing training through changes in the architecture (Parisotto et al.] 2020) or the weight initialization schema
(Huang et al. 2020: Melo 2022). We follow Huang et al. (2020) and Melo (0022) and stabilize transformer optimization through weight initialization.
Numerous approaches exist in the literature for hierarchical transformers for offline RL (Shang & Ryoo |021 Correia Alexandre 2022, Mao et al. 2023 Zhang et al.] 2023). These works mostly focus on leveraging datasets of demonstrations to perform well in the same task-distribution as the demonstrations. In contrast, our focus lies in online RI. and generalization across tasks with varying dimensions (c.g., parametric and task variations). By leveraging the sequencing of transitions and cross-episode information, we aim to efficiently meta-learn task representations that adapt dynamically.
Meta-feature extraction lies at the heart of our method. Approaches like VariBAD (based on RNNs) and PEARL (based amortized inference) extract task-relevant information from transitions within a task, augmenting the state input with meta-features. These meta-features capture the nuances of task dynamics and improve policy performance. Expanding this, TrMRL (Melo) 2022) demonstrated that transformers can generalize better by using sequence of the most recent transitions,
3

---

Published as a conference paper at ICLR 2025
capturing dependencies through self- Similarly, AMAGO (Grisbyy et al. 2024a) showed that extracting information from longer sequences. spanning multiple episodes, sigmificantly enhances generalization. In AMAGO-2 (Grgssy et al.) 2024b). the authors evaluate their approach more extensively in multi-task RI. problems and showcase the generalization abilities to unseen parametric variations of tasks in the meta-training (e.g. harder levels).
Building on these foundations, our approach integrates intra-episode (within a single episode) and cross-episode (across multiple episodes) information to extract richer meta-features. Using transformers, we analyze sequences of transitions to capture complex, multi-level patterns that identify the task dynamics. This design enables in-context learning, allowing our model dynamically adjust its behavior based on the context provided by past experiences.
Our method achieves efficiency in terms of runtime complexity while capturing intricate patterns in dynamic and diverse task environments. By leveraging information from both intra- and cross- episode experiences, we improve generalization across tasks with varying dimensions. Compared to prior state-of-the-art methods, our approach demonstrates improved meta-training efficiency and generalization.
PRELIMINARIES
The Reinforcement Learning (RL) problem is typically formalized within the framework of Markov Decision Processes (MDPs), defined by tuple (5, A, P. 7) S represents a finite set of states the task. Each state E S encapsulates all the information that describes the situation at particular point in time. A denotes a set of actions that can executed in all states. P: Sx x A S - [0. 1] the state transition probability function that describes the dynamics of the MDP. More specifically, P(s'|s, a) gives the probability of transitioning to state when action a is taken in state s. R: S x A + R is the reward function, with R(s, a) giving the immediate reward received after taking action from state s. y E [0, 1] is the discount factor that determines the difference in importance between future rewards and current rewards. For an agent interacting with the MDP, lower values of ? mean that the agent will prioritize immediate rewards more strongly, while a higher value indicates a preference for long-tern gains.
The goal in RL is to find an optimal policy x' E II, with :: S + A. that maximizes the expected discounted cumulative reward. This can be formally defined as x" argmax=eu with
( E
7 RR(s, az) S
Here, Jar) denotes the expected cumulative reward when starting in 8o = 8. playing action d, in state 8t as dictated by policy т, i.e., Ot ~ ((s)). Most modern RL approaches parameterize the policy na. where the parameters @ are typically the weights of a neural network, and aim to find the parameters that satisfy the previous expression.
Meta-Reinforremeent Learning (Meta-RL): Given distribution of tasks p(T), where each task T (T) can be considered distinct MDP with its own state space 5", action space A", transition dynamics pr and reward function RT. the objective of a meta-RL technique is to learn reinforcement learning algorithm C. that directly outputs the parameters of a reinforcement learning policy. The learned algorithm L is parameterized by & such that L. T A and aims to learn the parameters so can generalize across set of tasks sampled from the distribution p(T) To this end, the policy interacts with a series of tasks т, drawn from p(T), (T), and aims to maximize its performance across this task distribution. This process involves two key phases:
Meta-Trainiin and Meta-Testing Phases: During meta-training, the agent learns a policy my and an update rule Ls (parameterized by 0 and Φ. respectively) that can quickly adapt new tasks, resulting in a bilevel optimization problem. This further involves learning a generalizable representation or prior knowledge that is transferable across tasks. The outer objective in this phase is to optimize the parameters & such that o" E argmax, G(Ø) with G(Ø) = E,-p(T) [Tro][Cs. t] whereas the inner objective involves finding the weights of the policy that maximize 0* E (J In the meta-testing (adaptation) phase, the agent encounters new tasks Tnew sampled from p(T). Using my-
4

---

Published as a conference paper at ICLR 2025
and Le,, the agent quickly adapts to each new task. The adaptation effectiveness is measured by the agent's performance on these new tasks after a limited number of learning steps or experiences.
METHOD
We illustrate proposed approach, which we dub ECET (Efficient Cross-Episode Transformers), in Figure 2] Our architecture consists of two distinct groups of transformer encoder blocks. The first is the Intra-Episodic Transformer (TET) that encapsulates the sequence of transitions inside an episode and allows us to understand the transition dynamics within each episode. The second group of encoder blocks is the Cross- Episodee Transformer (CET) that encodes the set of episodes and captures the transition dynamics across episodes.
Our experience on a task consists of E episodes, including the most recent episode. For efficiency reasons, we sample sequence of T transitions from each episode. For the current episode, we ensure that the sequence ends with the current state observation of the agent, such that can choose the most appropriate next action. The set of E episodes, each having T trannsitios, are stored as sequences of tuples xi = (sf, of r (e).)e.....
where dt {0, 1) is termination flag. The I-th transition from the sequence of the e-th episode denoted as Xi E S A R x (0, 1), while sequence of transitions from the e-th episode as (X6= Xi. .,............................ Xi) E (S A R x 10,1))
The Intra-Episodic Transformer (IET) computes representation that transforms sequence of T transitions from the e-th episode into a vector representation 2 E R as formalized below:
z" IET (Xi.....XF7 IET AS 13) + RR E
The IET employs encoder-only transformer architecture with positional encoding, that projects the sequence of transitions to a latent representation using multiple layers of transformer blocks. The network's representation of the sequence is the R latent embedding of the T-th transition in the last transformer block.
We then input the intra-episodic representations z" RR Ve E {I....,E) from all the episodes into the Cross -piisodic Transformer (CET), which captures the dynamics of sequences from different experience episodes. The CET network ultimately represents the experience on task into a vector Zusk formalized as:
rask = CET (x!,...,?"))
CET Fx RR.
Bakk E TR.
The CET network is also stack of transformer blocks. The representation of the episodic experience Zusk is the R output for the E-th element of the last transformer block in CET. Since episodes
z" RK Ve {1, {1...., E,..,,E) E} represent set, construct CET without positional encoding.
Finally, we condition the policy on the learned cross-episodic features and a linear transformation of the current state
== (4p(s), Ziask)
where the output of the policy, A, represents the parameters of the distribution Pa from which we sample the next action (A). We use a Gaussian distribution the model for the probability distribution of the actions, therefore, À is the mean and variance of the distribution.
Following the RL3 framework, we use PPO the outer-loop algorithm to train our pipeline. We use the PPO objective for the actor and Negative Log-Likelihood loss for the critic. We detail our Meta-Training and Meta-Testing Algorithms in Algorithm 2 and Algorithm resppettvelll
The hicrarchical nature of our novel transformer architecture improves the runtime complexity of computing the task representation. Suppose we would have opted for a standard hin-hheerarchical) transformer network (Melo) 2022) that is fed a single long sequence of all the transitions from all episodes (i.e. E many transitions). Such naive approach requires an attention matrix of complexity OEETT2). On the other hand, using the IET network only capture intra-episodic features requires an attention matrix of complexity O Capturing cross-episodic features Zusk using CET requires attention matrix of complexity O(E²). Thus, our two-tiered architecture of the cross-episodic transformer has a runtime complexity of O + T2), and seales to longer sequences and more episodes than standard non-hierarchical transformer.
5

---

Published as a conference paper at ICLR 2025
X
Xc-T-4
CET
(A)
xi
MLP
X--1
Episodes Experience
IET
X1-241
Figure 2: Illustration of ECET architecture. Transitions xi (5f, at, rf, .d.) represent which state of, action aj and reward rī were observed at the (-th transition of the e-th episode, but also termination flag dž that indicates if the e-th episode already terminated time By inputting the sequences independently through the Intra- adtra-Epioodic Transformer (IET), we generate feature vector for each episode E {I...., E}. We then pass these sequence representations through the Cross-Episodic Transformer (CET) to generate a task representation Zusk as input to the policy æ.
5 EXPERIMENTS
15
10
door-opan--2
drower close v2
butilenggeeen pww.-attppiooeedd
10
windoww o-pen--222222
20
-30
-20
30
-30
-20
20
-10
-5 10
15
15
Figure 3: T-SNE plots of the output embeddings for TrMRL (left), and our ECET (right) for five of the tasks in the training set of the ML10 benchmark of Meta- World. For better visibility, we only plot 5 tasks here but provide version with all 10 in Figure 14
5.1 EXPERIMENTAL PROTOCOL
MuJoCo We evaluate ECET and baseline methods MuJoCo (Todorov et al. 2012) lo- comotion tasks, which are widely used in the Meta-RL literature. Specifically, we consider the AntDir and HalfCheetahDir tasks, where the agent is required to move either forwards backwards. Additionally, we include the HalfCheetah Vel task, where the agent must adapt to running at different target veloeities. The maximum episode length for these tasks is 200 timesteps.
Meta-World
To assess the performance of our cross-episode transformer approach, use the Meta- World Benchmark (Yu et al. 2019). This benchmark designed to help evaluate the performance of meta-RL RL baselines in 50 different simulated robotic manipulation tasks. These tasks all share the same state S and action space A as they are all located on a tabletop environment with a Sawyer arm (see Appendix A.4 for details and illustrations of all tasks). What differs in each task is the

---

Published as a conference paper at ICLR 2025
behavior required to complete it successfully. For meta-learning, Seta-World includes three modes with varying degrees of difficulty (for details, see Appendix A.4):
MLL1 1 is designed to evaluate few-shot adaptation to parametric variations within one task.
MML0 evaluates few-shot adaptation to 5 unseen test tasks, after training on 10 tasks.
MLL45 is similar to, but more complex than ML10, with 45 training and 5 test tasks.
For MLI, the desired goal position is not provided as input, so the meta-RL algorithm needs to determine the location of the goal through trial and error. Similarly, for ML10 and ML45, task IDs are not provided as input, so the meta-RL algorithm needs identify tasks from experience. The maximum episode length for all MetaWorke tasks is 500 timesteps.
2023) to assess the performance ManiSkill We additionally use the ManiSkill Benchmark (Gu et al.
of our proposed method on tasks where the state representation is an image. This benchmark consists of robotic manipulation tasks to help evaluate the performance of approaches for embodied Al in terms of their generalization to parametric variations, as well variations in the object being manipulated. We evaluate on the StackCube and PickSingle tasks, located on a tabletop environment with Panda robot. What differs in each task the behavior required to complete successfully:
StackCube:
The goal is to pick up red cube and stack it on top of green cube and let go of the cube without it falling. The task is parameterized by randomly sampling the initial position of the cubes.
PickSingle YCB: The goal is to pick random object sampled from the YCB dataset (Calli et al, 2015) and move to a random goal position. This task designed evaluate few-shot adaptation to different objects.
We use the CNN architecture from Mnih et al. (2015b) to process RGB states for all methods. The maximum episode length for all ManiSkill tasks is 50 timesteps.
Performance metries For the MuJoCo benchmark, use the average return performance metric. The authors of Meta- World and ManiSkill propose the success rate as a performance evaluation metric. The distance of the task-relevant object from the goal position determines whether the agent is successful or not in an episode. Thus, the success rate the fraction of episodes where the agent manages place the task-relevant object inside a pre-set perimeter around the goal position.
Due to the fact that the tasks have different levels of difficulty, especially for the ML10 and ML45 modes in Meta-World, we additionally the average rank as performance metric for aggregating the performance across different tasks. We calculate the rank of methods over time for each task in the set based on their success rate, and then average over those ranks across tasks.
Baselines: We focus on investigating the performance of ECET compared to other online meta- reinforcement learning methods, namely MAML PPO (Finn et 2017), RL (Duan al. 2016), PEARL (Rakelly et al. 2019). VariBAD (Zintgraf et al., 2020), TrMRL (Melo, 2022) and AM- AGO (Grisbyy et al., (024a). Detailed descriptions of each are given in Appendix A.5 We conducted all experiments
compute cluster of NVIDIA A100 GPUs. We trained all methods for 10" steps in the MuloCo environments, 5 10" steps in Meta-World, and 2.5 x 107 steps in ManiSkill. We repeat each run for 5 different random seeds. When comparing the methods in the figures, plot the mean and standard error of the performance metries across seeds.
5.2 HYPOTHESES AND RESULTS
Hypothesis ECET captures more general task features compared to state-of-the-art meta- RL methods. This stems from ECET's ability to capture intra- as well as cross-episodic experiences. We compare the embeddings that ECET learns to those of TrMRL to investigate the extent to which they differentiate between tasks. We sample episodes from the training tasks in ML10 and plot the output the transformers for both ECET and TrMRL through dimensionality reduction. For the sake of visual clarity, in Figure 33 we show the results for 5 of these tasks. For the plot with all the 10 tasks, see Figure I4] Additionally, and inter-task embedding distances. From these plots it is visible that the learned embeddings
Figure [5 we compare the intra-
7

---

Published as a conference paper at ICLR 2025
HaliCheetah-Vel
HalfCheetah-Dir
1500
1200
-50
Average Return
1250
1000
-100
600
500
400
-150
250
200
2
2
2
Number of Episodes
Mumber of Episodes
Number of Episodes
VariBAD
Figure 4: Test performance in terms of average return across rollouts for ECET, TrMRL, PEARL, MAML PPO, RI.2 PPO and VariBAD on the HalfCheetah- Vel, HalfCheetah-Dic and Ant-Dir tasks of the MuJoCo benchmark, testing adaptation to parametric variations of these tasks. We show the corresponding meta-training performance plot in Figure
ML1-Push
1.0
0.8
Success Date
0.6
0.4
0.2
0.2
2
5
Meta-Leaming Timesteps
1e7
Meta-Learning Timesteps
1e7
ECET
PEARL
PPO
VariBAD
Figure 5: Meta-Train and Test performance in terms of average success rate for ECET, TrMRL, PEARL, MAML PPO, RL2 PPO and VariBAD on the ML1 benchmark for training(left) and test- ing(right) on parametric variations of the Push task. We show the corresponding plot with respect to time in Figure 19
whereas the average rank plot can be found in Figure [6
of ECET produce a clearer grouping for the same task, compared to embeddings of TrMRL. Furthermore,
grouping of tasks that are similar to each other, e.g., Window Open and Door Open. This shows the usefulness of cross-episodic experiences in task differentiation. For easier analysis, provide scripts for interactive versions of the T-SNE plots in code repository
com wwt.:/cc.ooaccccccccee i wng.iinnuuuueeee www.riinnnnnuurrneee.e..e. remberg ECET / blob/main plotly interactiie plot.
Hypothesis 2: ECET outperforms the state-of-the-art in online meta-RL algorithms in few-shot adaptation parametric variations of tasks. To evaluate adaptibility parametric variations of the task, we compare ECET and the baselines in the MuJoCo set of tasks, and in the ML1 mode of Kera-World (Push and Reach tasks). We evaluate the generalization on the parametrized MuJoCo tasks through doing 5 rollouts at test time, and calculating the return for each. We show the results Figure 4 ECET, TrMRI., and VariBAD show robust test performance across the three tasks. For the Meta- World MLI benchmark, we show the performance in terms of Success Rate in Figure 5 for Push. The figure contains results for meta-training (left) and test (right) performance in the case where we train variations of the Push task, and test the adaptability of the methods on set of unseen variations of the same task. terms of veeta-training performance (left), we can see that transformer- based methods are faster in differentiating between the variations of the task and thus achieve a maximum success rate of 1.0. In terms of test performance (right), ECET and TrMRL generalize successfully, reaching test success rate greater than 0.8. In addition, ECET shows
8

---

Published as a conference paper at ICLR 2025
ML10 Train Performance
Performance
0.8
0.8
0.6
0.6
0.4
0.2
167
167
ML10 Test Performance
ML45 Test Performance
0.4
0.3
0.2
0.2
0.0
3
Meta-Learming Timesteps
167
Meta-Learning Timesteps
1e7
ECET
AMAGO
TiMAL
PEARL
MAML PPO
RL2 PPO
VariBAD
Figure 6: Meta-Train and Test performance in terms of average success rate for ECET, AMAGO, TrMRL, PEARL, MAML PPO, RL2 PPO and VariBAD on the MLI0(left) and ML45(righh) bench- marks. Here, we training and testing disjoint sets of tasks. We show the corresponding plot with respect to time in Figure 21, whereas the average rank plot can be found in Figure
higher generalizability compared to TrMRL. RI.2 PPO struggles robustly generalize, exhibiting a higher variation in performance compared to the other methods. In Figure 13 in Appendix A.6 we show the Meta-Train and Test performance on variations of the Reach task. The task variability for Reach is not as difficult to learn, thus all the methods perform better both when meta-training and testing. To further assess generalization to parametric variation, we evaluate our method and the baselines in the StackCube task of ManiSkill. We show the meta-train and test performance in Figure (left column), where we notice a similar pattern on performances as with ML1, with ECET outperforming the baselines after 2.5 10? meta-training timesteps.
Hypothesis 3: ECET outperforms the state-of-the-art online meta-RL in out-of- distribution (OOD) tasks. We define out-of-distribution (OOD) performance to be the performance of meta-learned agent set of tasks that is disjoint from the tasks is meta-learned on. We show the performance of the methods terms of the average success rate for the meta- meta-training (top) and test (bottom) performance for MI.0 (left) and MIA5 (right) in Figure 6 In this setting also compare to AMAGO, as a recently proposed powerful transformer-be based method processing long sequences. ECET is the only method that achieves an average success rate greater than 0.8 in MLI0 and greater than 0.6 for ML45 in terms of meta-training performance with a final success rate roughly 20% higher than the baselines. The other methods fail learn policies that successfully identify the different tasks in the training set for the given budget of 5 x 10" meta-learning timesteps, and thus struggle solve them. In terms of OOD performance, the bottom row shows that generalization to distinct tasks is still challenge for meta-RI. ECET achieves average test success rate of approximately 0.35 for MI.10 and 0.4 for AI Similarly, Figure 7 (right column) where we show the performance effectively generalize on the task of picking different objects. ECET is the best performing method, achieving an average test success rate of 0.39. This further demonstrates that cross-episodic features can improve meta- Reta-RL performance without incurring the cost of gradient adaptation at test time.
PickSingle
notice that all baselines except for TrMRL struggle to
To focus comparing the methods based their robustness in relative performance to each other, in Appendix [.. we provide plots showing the average ranks the methods as a performance metric. These plots show that ECET robustly achieves the highest relative performance (and thus the

---

Published as a conference paper at ICLR 2025
Train Performance
PickSingleYCB Porformance
1.0
0.5
0.8
Success Rate
0.4
0.6
0.3
0.4
0.2
0.1
0.0
0.0
StackCube Test Performance
PickSingleYCB Test Performance
Success Rate
0.6
0.2
0.2
0.1
0.0
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
Meta-Leaming Timesteps
167
Meta-Learning Timesteps
1e7
ECET
TAMAL
PEARL
MAML PPO
RL2 PPO
VariBAD
Figure 7: Average success rate plots of Meta-Train and Test performance for all methods on the StackCube(left and time in Figure 22 whereas the average rank plot can be found in Figure 18
tasks. We show the corresponding plot with respect to
lowest rank) across the meta-learning benchmarks consider. For detailed illustration of per-task task performance, we refer to Appendix (.6. In Appendix (A.6.4) plots episodic return across 50 episodes at test time, show that ECET performs well even in scenarios when the agent has not yet collected the E episodes of experience.
5.3 ABLATIONS
We investigate the behavior and the impact of ECET's components on performance in Appendix A.6.5 We show the effect of varying the number of episodes (F.) in memory and the sequence length (7') of sampled transitions in Figure[88] Figure[2]] (top plot) shows that sampling without replacement results in more reliable and faster performance improvements, especially for smaller (E)) and (T). In terms of architectural variations, we investigated the effects of adding positional encoding in the Cross- Episode Transformer (CET) and simplifying ECET to flat architecture akin to TrMRL and AMAGO. The results in Figure 26jindicate that positional encoding in CET introduces misleading ordering. leading to suboptimal convergence, while the flat architecture struggles to capture comprehensive task representations.
CONCLUSION
We introduced Efficient Cross-Episodde Transformers (ECET) for Meta-Reinforcement Learning, powerful approach to online meta-RI.. Our experiments within the MuloCo, Meta-World and ManiSkill benchmarks highlight our method's capability for few-shot adaptation to tasks with parametric variations, well as its capacity to adjust completely unseen tasks that share the same state and action spaces as the training tasks. Despite outperforming state-of-the-art methods in online meta-RL by efficiently compressing cross-episodic knowledge, our method's generalization to tasks that completely different to the meta-training set remains limited our approach is tailored to exploiting similarities. Future work should consider using task augmentation improve generalization, modifying the adaptation phase, increasing model capacity to handle wider context in order to improve performance in this phase.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
Gresa Shala, André Biedenkapp and Josif Grabocka acknowledge funding by The Carl Zeiss Founda- tion through the rescarch network "Responsive and Scalable Learning for Robots Assisting Humans" (ReScaLe) of the University of Freiburg. The authors gratefully acknowledge the scientific sup- port and HPC resources provided by the Erlangen National High Performance Computing Center (NHR @ FAU) of the Friedrich-Alexander-- Alexander-Universitze Erlangen-Nürbbery (FAU). The hardware is funded by the German Research Foundation (DFG).
REFERENCES
M. Andrychowicz. A. Raichuk. P. Stanczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem. What matters for on-policy deep actor-critic methods? A large-scale study. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.
Sebastian Pineda Arango, Fabio Ferreira, Arlind Kadra, Frank Hutter, and Josif Grabocka. Quick- tune: Quickly learning which pretrained model to finetune and how. In The Twelfth International Conference on Learning Representations, 2024. URL
net/ forrmm?
id=tghl zdxI ra
J. Beck, R. Vuorio, E. Z. Liu, Z. Xiong, L. Zintgraf, C. Finn, and S. Whiteson. A survey of meta-reinforremmntt learning. arXiv preprint arXiv:2301.0802. 2023.
M. G. Bellemare, S. Candido, P. Samuel Castro, J. Gong, M. C. Machado, S. Moitra, S. S. Ponda, and
Z. Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77-822 2020.
C. Benjamins, T. Eimer, F. Schubert, A. Mohan, S. Döhler, A. Biedenkapp, B. Rosenhahn, F. Hutter, and M. Lindauer. Contextualize me the case for context in reinforcement learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
M. Beukman, D. Jarvis, R. Klein, S. James, and B. Rosman. Dynamics generalisation in reinforcement learning via adaptive context-aware policies. In shirth-seventh Conference on Neural Information Processing Systems (NeurIPS 2023), 2023.
Abhinav Bhatia, Samer Nashed, and Shlomo Zilberstein. RL$^3S:
Boosting meta reinforcement learning via RL inside RL In NeurlPS 2023 Workshop on Generalization in Planning, 2023.
URL /RR.httts:///ppneeeee http:///pnnneeeeeeeeeeeeeoereeee htt///nt//rriiee f 1d=
Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar, The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 International Conference on Advanced Robotics (ICAR), pp. 510-517, 2015. doi: 10.1109/ICAR. 2015.7251504.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, volume 34, pp. 15084-15097. Curran Associates, Inc., 2021.
K. Cobbe, C. Hesse, Hilton, and Schulman. Leveraging procedural generation benehmark rein- forcement learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp- 2056. PMLR, 13-18 Jul 2020.
André Correia and Luis A. Alexandre. Hierarchical decision transformer, 2022.
Degrave, Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdol- maleki, D. de las Casas, C. Donner, L. Fritz, C. Galperti, A. Huber, J. Keeling, M. Tsimpoukelli, Kay, A. Merle, Jean-M. Moret, Noury, F. Pesamosca, D. Pfau, O. Sauter, C. Sommariva, S. Coda, B. Duval, A. Fasoli, P. Kohli, K. Kavukcuoglu, D. Hassabis, and M. Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897)4144419, 2022.
11

---

Published as a conference paper at ICLR 2025
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL?: Fast reinforcement learning via slow reinforcement learning. CoRR, abs//611..00 1..7799 2016.
T. Eimer, M. Lindauer, and R. Raileanu. Hyperparameters in reinforcement learning and how to tune them. A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and Scarlett (eds.), International Conference on Machine Learning. ICML 2023, volume 202 of Proceedings of Machine Learning Research, pp. 9104-9149. PMLR, 2023.
L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry. Implementation matters in deep RL: case study on PPO and TRPO. 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net. 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1126-1135. PMLR, 2017. URL
1121 mwr http://p...pp.pp///o//oaaaaaaaa.
The garage contributors. Garage: A toolkit for reproducible reinforcement learning research. https
//github wtw.:tabb.commmwwwwww.t...ee wwtp:///wwwwoooo///////oo///o/re
2019.
Jake Grigsby, Linxi Fan, and Yuke Zhu. AMAGO: Scalable in-context reinforcement learning for adaptive agents. In The Twelfth International Conference on Learning Representations, 2024a.
URL httt:pss://ppeeeeteee
net/ / forum? Fdd
Jake Grigsby, Justin Sasek, Samyak Parajuli, Daniel Adebi, Amy Zhang, and Yuke Zhu. Amago-2: Breaking the multi-task barrier in rentrreenfrrrement learning with transformers, 2024b. URL
fitts:///ariv: org/ abs 2411 11188
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang. Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskil12: A unified benchmark for generalizable manipulation skills. The Eleventh International Conference on Learning Representations, 2023. URL htt.:: / //ppnreviee htte:/Eorumii___bbeeeee
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Leurning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018. URL
https: /ttps://pprceeet/ ngs amg.mmlrr... www.srssss/ee.. v 0 haarno 8b. htm.
Henderson, Islam, Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. In S. Mcllraith and K. Weinberger (eds.), Proceedings of the Thirty-Secont Conference on Artificial Intelligence (AAAI' AAAI Press, 2018
T. M. Hospedales, A. Antoniou, P. Micaelli, and A. J. Storkey. Meta-leaming in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5149-510) 2022.
Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs. Improving transformer optimization through better initialization. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4475-4483, PMLR, 13-18 Jul 2020.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, volume 34, pp. 1273- 1286. Curran Associates, Inc., 2021. URL Ei
pdf
E. Kaufmann, I.. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza. Champion- level drone racing using deep reinforcement learning. Nature, 620/7976);9829987..
R. Kirk, A. Zhang, E. Grefenstette, and Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. J. Artif. Intell. Res., 76:201-264, 2023.
12

---

Published as a conference paper at ICLR 2025
Kuang- Huei Lee, Ofir Nachum, Mengjiao (Sherry) Yang, Lisa Lee, Daniel Freeman, Sergio Guadar- rama, Ian Fischer, Winnie Xu, Eric Jang. Henryk Michalewski, and Igor Mordatch. Multi-game de- cision transformers. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27921-27936 Curran Asso- ciates, Inc., 2022. URL htR./httpps:/prrcceeeiiii neuri ps cc htte/pappr_filespppppeeeeee G222FEeeescc21f82233333377155333 Papeerrorrerrroeeeeeeeee wwnferenceeecc.eee p
Suyoung Lee and Sae- Young Chung. Improvinggeeneralizaainn generalization in meta-RI. RI. with imaginary tasks from latent dynamics mixture. A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL hattps:/ net/ wtw.-ooummmmiidnpe. SiNZi
Hangyu Mao, Rui Zhao, Hao Chen, Jianye Hao, Yiqun Chen, Dong Li, Junge Zhang, and Zhen Xiao. Transformer in transformer as backbone for deep reinforcement learning, 2023.
Luckeciano C Melo. Transformers are meta-reinforcemmnt learners. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning mlr
Research, pp. 15340-15359, PMLR, 17-23 Jul 2022. URL hattpp:://proeeeeeeee
press/vil 62 wwt.:///eoo2...a.c..m html
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations, 2018. URL https //openreview.oo net. forumm ?immmiiddB ZDU
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare. A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforce- ment learning. Nature, 518(7540):529-553 2015a.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg. and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(75400 February 2015b, ISSN 1476-4687. doi: 10. 10.1038/aturee42236 URL httpps://di..... owrooi.orgg/l 1038//attrrreeeeeeeee
A. Mohan, C. Benjamins, K. Wienecke, A. Dockhorn, and M. Lindauer. Autorl hyperparameter landscapes. In Second International Conference on Automated Machine Learming. PMLR, 2023.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphaël Lopez Kaufman, Aidan Clark, Seb Noury, Matthew Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7487-7498. PMLR, 13-18 Jul 2020. URL htttps://proceeeeeeeeeeeee html
mlr. www.spres//oe. http://111ppaarreeee
J. Parker-Holder, R. Rajan, X. Song, A. Biedenkapp, Y. Miao, T. Eimer, B. Zhang, V. Nguyen, R. Calandra, A. Faust, F. Hutter, and M. Lindauer. Automated reinforcement learning (autorl): survey and open problems. Journal of Artificial Intelligence Research (JAIR), 74:517-568, 2022.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reiinforcement learning via probabilistic context variables. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5331-5340. PMLR, 09-15 Jun 2019. URL press/v91/rakellyy
mlr
html
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions Machine Learning Research, 2022. ISSN 2835-8856 URL httpt::tt//tto /tttps::/oppnzevieeee net. wwt.nortttootoooeo. id=likkOkHjww Featured Certification,, Outstanding Certification.
13

---

Published as a conference paper at ICLR 2025
Zohar Rimon, Aviv Tamar, and Gilad Adler. Meta reinforcement learning with fi- nite training tasks - a density estimation approach. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Infor- mation Processing Systems, volume 35, pp. 13640-13653. Curran Associates, Inc., 2022. URL httpp::/pproceeteee httpa//sssssrpppppppppeeeeeeeeee http:/ppape_fflls/ppppppeepeee Papee-Coneeeeeeeeeeeeeeee www.erenneececcccceeeee pdf
neurips
J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, TT. Graepel, 'T. Lillicrap, and D. Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(783)):004-60)) 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
G. Shala, S. P. Arango, A. Biedenkapp. F. Hutter, and J. Grabocka. Autorl-bench 1.0. In Workshop on Meia-Learning (Metalearn Neurr/eeeeuPPP 22), 2022.
Gresa Shala, André Biedenkapp, Frank Hutter, and Josif Grabocka. Gray-box gaussian processes for automated reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023a. URL
net /ereet/foummt forum? idd
Gresa Shala, Thomas Elsken, Frank Hutter, and Josif Grabocka. Transfer NAS with meta-learned bayesian surrogates. In The Eleventh International Conference on Learning Representations, 2023b. URL httttp:::/ppereeeeeeeeeee net. wewpeerrrriieeeee/oorrraamm htt.ee/eoooommmddaaaaaaaaaa.
Jinghuan Shang and Michael S. Ryoo. Starformer: Transformer with state-action-rrewall representa- tions. CoRR, abs/2110.06206 2021. URL
06206
R. Sutton and A. G. Barto. Reinforcement learning: An introduction. Adaptive computation and machine learning. MIT Press, 2 edition, 2018 8.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-bate basel-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, IEEE, 2012. doi: 10.1 1..10099ROS.000 2. 2.6366109
J. Vanschoren. Meta-learning. In F. Hutter, Kotthoff, and J. Vanschoren (eds.), Automated Machine Learning Methods, Systems, Challenges, The Springer Series on Challenges in Machine Learning, pp. 35-61, 61. Springer, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL
wwwp::/wwceeeiiio....e......aeee neurips cc/paper files//aper/2017/ Ei wtleeeff5e2244444730 1 fbd053c -apee .pdf]
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019. URL httppss////xxv...//////o http://rggg//////aa/ao/oooss
10897
Bin Zhang, Hangyu Mao, Lijuan Li, Zhiwei Xu, Dapeng Li, Rui Zhao, and Guoliang Fan. Stackelberg decision transformer for asynchronous action coordination in multi-agent systems, 2023.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: very good method for bayes-adaptive deep via meta- learnin.. In International Conference on Learning Representations, 2020. URL htttps://operrevieee
met/forum Ad 1JJ
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 RL FRAMEWORK
200
Ex--1
Figure 8: Illustration of the RL framework, based on Figure from Beck et al. 22033 p. As illustrated, the RNN takes as input the current state 8t, the action(if there is any) that lead to the state and the reward incurred from reaching that state. The output of the RNN is then used as input to the policy m along with the current state.
A.2
AND META-TESTING ALGORITHMS
We include the detailed algorithm for Meta- Training in Algorithm 2 and the algorithm for Meta- Testing in Algorithm] 3 In order to be consistent in comparison to the baselines, use the garage contributors (2019) library implementations for the PPO algorithm, Gaussian Actor and Gaussian Critic.
Meta-Training
A.2.1 INITIAL SETUP AND HYPERPARAMETERS OF META-TRAINIA
We use as input distribution of meta-training tasks T and history H. FIFO queue that stores past transitions to capture the temporal dynamics. The hyperparameters that guide meta-training process include: the number of episodes E to maintain in the history for context, the sequence length of transitions to be processed by the Intra Intra-Episode Feature Extractor IET, the maximum number of training epochs mar_ _ee__es the sizes for both meta and mini- mini-batches (meta_ batch_atcchht _ize),, and the number of episodes collected per task in each training round.
A.2.2 PARAMETER DEFINITIONS
Actor Parameters include the parameters for embedding transitions ((1) and states (122), the parameters for our Intra- and Inter-Episode Feature Extractors (O,, θ2) within the hierarchical transformer framework, and the policy network (r) parameters (θz). Critic Parameters include
the parameters (V/) for value function (V), which assists in estimating the expected returns and guiding the actor's policy adjustments.
A.2.3 PROCEDURAL COMPONENTS
For clarity and conciseness of writing, we include some procedural components of our meta-training and meta-testing pipeline as procedures in Algorithm These include:
CollectEpisode: We gather one episode of interactions for a given task, utilizing the Actor (4,, ④)) for action decisions, and record these transitions within history H and an episode_burie _bffeerfffrrrrr which contains for each transition in the episode: the set of sampled transition sequences C used to generate the task representation .as well as the (state, action, reward, terminated) for processing.
CollectAidProcessEpiodee:: Using the mrisode_buffee data returned, we process each transition to compute advantages and returns, and add
15

---

Published as a conference paper at ICLR 2025
Algorithm 1 Subroutines for ECET
Distribution of tasks T, history (FIFO queue) store past transitions. Hyperparameters: number of episodes E' to keep in history. sequence length T of transitions to input in MET, mnar_eppohss, meta meta_batch__s batch_size, size, mini Semmi_batch_aee batch_bttaaaa size, number episodes k to collect for each task during training.
Input:
Actor Parameters
Transition((21) and state({p)) embedding parameters: & (11, P2
Intra-Episodic Transformer(EETT 11)). Cross-Episodic Transformer(CETT 02)), and
(o (s)) parameters: @ {θ1, θ2 θs)
Critic Parameters
Value function (VV::; parameters: V
procedure COLLECTTEPISODE(ausk;; +b e H)
buffocrccufffcccerr state - task.reset()
terminated +- False
while not terminated do
SAMPLETRANSITIONSS,,, state)
4- GENNBAATETAAKRRPPPSSSNAATTAOOOO (P1, 11, θ2)
f- (Ztakk: (2(stale); 0a)
action ~ X
next state, reward, terminated + task
Add (state, action, reward, terminated) to H Add (C, state, action, reward, terminated) to geisode_buffee
state next_state
return episode _uffer,_ H
procedure COOLEEETKADDROOSSSEEPPOOOCCC $, O, V H. D)
episode _uffeer H + CoLLECTEPISODE(ttss,, $ θ H))
for (C, state, action, reward, terminated) in episode buffer
Calculate advandage using the value function V:; M) and received reward. Calculate the return.
Add (C, state, actianciinnete reward, terminated, return) to D return D.
procedure TRA.TTTTTSTION carrent_stale)
Initialize episode transitions set C to empty.
for episode +- to E do if episode not current then
Sample sequence T transitions and add to else
Sample sequence latest transitions including current_state and add to
return C procedure
(p,, O1, θ2)
Initialize set Cseg to empty.
for i x- to do
T, Cli)
z seg + (T);
Add 2 to Cseg- Zaakk f- CET(Cssog) O2).
return Ztask
16

---

Published as a conference paper at ICLR 2025
Algorithm Meta-Training with ECET using Subroutines from Algorithm
Distribution of meta-traing meta-training tasks T, history H (FIFO queue) to store past transitions. Hyperparameters: number of episodes E keep in history. sequence length T of transitions to input in IET, -max_epochss mela seelabblchhsize, mini wrini_bllhh_siae number of episodes k to collect for each task during training.
Input:
Actor Parameters
Transition((;1) and state((p1) embedding parameters: == (41,)))))
Intra-Episodic Transformer(EET/ ④))), Cross-Episodic "ransformer(EET(( and (= )g)) parameters: + = { θ2 θ3)
Critic Parameters
Value function ((V(s (:; W) parameters: V
Procedures: COLLECTANDPROCESEPPSS
SAMPLETRANSITIONS
procedure (C state) J-m ii batch size + θ)
mmmnimbatchhhc. size Generate (zrash [==
using
GENRATETASKREREENNT
TION([C) [i=1
(P1, θ1 O2). batchbtthhssas size
/mini_batch &- ( Etaakk (P2(state) Jai main minii batch ; ;0)).
(Mactor, Jactor Jiml
return (Mactorr Tactor imini bize
li=1
procedure UPDATEMODELS(D. $ (.
hold Oold + $
for {(C, state, action, reward, terminated, advantage, return)) mini minnimmtthh_eeee batccbsizen in D do )
I==1
X ACTOR([(C, state); mini mmini_batch___o batch_size old Cold)
probald + Xald (factinn } miin batch_size).
À + ACTOR(((C, state), ) prob + X(faction:] Ma ymini
mmini_batch_sia
② θ).
baaihiibatch_iize
advantages +- {advantage, Jmmni cittii_bbtth_eeee sise
Calculate the PPO actor loss with respect to +; A:
T,acoor
4 =,, [minn(, @jadvaatagess,cipp// (b,0),1- 6, + c)advantages)
where ri ( θ is the probability ratio rz ()
prob
probata
peritiey Teritic +- V((state, ji=l returns + freturne] mmiin pmini_botchhate Calculate the NLL for the critic with respect to Y:
batch_eize)
L (ribii
1
Σ
N
log
exp
(returns - Meritse)
N
Ov,
2T
2ritii
1=1
Update b, O and V by minimizing Dottr and ceritic Initialize 4, θ V
for epoch 1 to max_ do
Empty meta-training dataset D.
Sample meta_batch, size tasks from T.
for each sampled lask do
Initialize H zeros. for episode +- to ls do
FIFO queue of E episodes
D, COLLFETAANNPPOOEEEEEHHOSSSSS 4, V.O. H, UPDATEMODELSSD, 4, e, V)
17

---

Published as a conference paper at ICLR 2025
Algorithm 3 Meta-Testing with ECET using Subroutines from Algorithm
Input:
Distribution of meta-testing tasks Tiest history H. (FIFO queue) to store past transitions,
nun_eval__ ceoal_takss, sum__vvll___ aval_episodes.
Parameters from meta-training
Transition(({)) and state({p1) embedding parameters: + (.1. Intra-Episooiic Transformer(IETTT O1)). )), Cross-Episodic Transformen((CT((..))) and po O2)) parameters: θ { θ2 θz).
Procedures: COLLECTEPISODE
Sample num _en___va_____ eval_tasks tasks from Trest-
for each sampled task do
Initialize H. zeros.
FIFO queue of E episodes
for episode < to num _piddess do
- -COLLECTEPSSODE((aa,, & O,
(C, state, action, reward, terminated, advantages, returns) to the meta-training dataset D. We return the updated history and meta-training dataset D.
Using the history H. we sample sequence of T'' transitions from each episode in H. except for the current episode, where we sample a sequence of T transitions that ends with the surrent_state We add these sequences in a set and return it.
SampleTransition::
Generate Repeeseerss:::::r We first transform each transition in the sequences T; using the transition embedding 91, and then input the embedding into the Intra-Episode Feature Extractor get z eeg + (ET (T,); O2) for each episode i. We store the zeeq output embeddings from IET in the set Coeg. We then input them in the Inter- Spiiode Feature Extractor to get Ztask + CET O2). Ztask is the task representation we return.
A.2.4 META-TRAINING
We describe the meta-training process in Algorithm 2 using the subroutines from Algorithm 1 We first initialize the Actor parameters (4), Q) and Critic parameters (V)).
Data Collection. For each epoch until reaching mar_epochss, we start by initializing an empty meta-training dataset 'D and sampling mela_ vaela_batch_size number of tasks т from T. For each sampled task, we first initialize the history queue H. zeros. This queue has a length of E episodes, to fit the hierarchical transformer's 's memory. For ls episodes we then update the meta-trann meta-training dataset DD and history H by calling the subroutine CollectAndPrrceestppiise
Model Update. Once we have collected data in DD for k episodes from each sampled task, we use to update the parameters. As we also show in the subroutine UpdateModels, we copy the current & and @ to ooid and O in order to be able to calculate the proba- bility ratios for the PPO objective for the Actor. We sample a mini_balch, (C, state, action, reward, terminated, advanlage, return) from D. Using the state values from the mini-batch, as well as the sampled transition sequences for each of these states, we get outputs from the actor using hold and gold, As we show in the Actor procedure, we use the mini-batch of sampled transition sequences C to generate a mini-batch of task representations Ztask as described in the GenerateTaskkRen seteeeetatteeeeeettttinnn subroutine. We embed the mini-batch of states using the state embedding 22, and concatenate them with the task representation Zrask for each state in the mini- batch. We then input the resulting mini-batch in the policy π to get the mini-batch of parameters of the action distribution. Since the action distribution is Gaussian, the policy π outputs the mean Mactor and standard deviation Oactor- Since we use aal and god, we name this mini-batch of distribution parameters ofd We then calculate the probability of the actions in the mini-batch given the distribution parameters Notd, and save the action probabilities in probold We do the same procedure for & and O resulting with the mini-batch of action probabilities prob. Using these action probabilities, as well as the advantages from the mini-batch, we calculate the PPO Actor loss with respect to & and O As our Value function, similarly to the policy is also Gaussian MLP, predicting the mean and standard deviation of Gaussian distribution for the value of each state input, to calculate the Critic loss, get the mini-batch of distribution parameters Peritic and Cerstic as output from the Value function V using the states in the minibatch as input. As the Critic loss, we calculate the Negative Log Likelihood loss of the returns in the mini-batch using Peritic and Ceritic-
of tuples
18

---

Published as a conference paper at ICLR 2025
We update T and @ by minimizing the PPO Actor Loss L L We update v by minimizing the NLL Critic Loss L critie
We repeat this process until we go over all tuples in the meta-training dataset D. Once D has been used for updates, we empty it, and return to sampling tasks and collecting episodes from them again, and updating the parameters based the new dataset, and so on, until mar..pohhs is reached.
A.2.5
We show the meta-test algorithm in Algorithm 3 using the CollectEpisode subroutine from Algo- rithm [ We use distribution of meta-testing tasks Trest and FIFO queue as history H. for the past episodes as input. We also set the num _eaeneeaaaaaaa tasks and num eval_ episodes parameters which determine the number of tasks sampled from Trest and the number of episodes performed in each task, respectively. We use the Actor parameters (4b )) which we got from meta-training. We first sampled num _eval_lasks lasks from Trest- For each sampled task, we initialize the history to zeros, and then collect numn_evad _vaa__elaeeeeeeeeeeeeeeeeeeeee _episdde__eeeeeeeeeee and update H. from each collected episode.
A.3 ECET HYPERPARAMETERS
Table 1: Hyperparameters Across Benchmarks
Benchmark Meta World
Hyperparameter
MuJoCo
Maniskill
Number of Transitions (T)
5
Number of Episodes (E)
2
25
Number of Layers for IET
2
Number of Heads for IET
16
4
Number of Layers for CET
2
Number of Heads for CET
16
4
Minibatch Size
256
32
Policy Learning Rate
3e-5
5e-5
Critic Learning Rate
3e-5
5e-5
A.4 BENCHMARK DETAILS
Train Tasks
Test Tasks
ML1
picoplise
pics place tegett $1,
Figure 9: Meta-world Tasks in the MI with training/test Split. Figure taken from (Yu et al. 2019
MI.1 provides one main task, with parametric variation in object and goal positions. MI.1 enables evaluation in the few-shot adaptation setting. In particular, adaptations to parametric variations within one task. Figure 9 givees an example of meta-training and testing for the pick in place task, where only the test-task target position different from those observed during training.
MI.10 provides different training tasks and others to test the adaptation performance of meta-learning methods. This mode evaluates few-shot adaptation to 5 test tasks, after training on 10 different tasks, see Figure I0, The test tasks are all related to the training tasks, but unseen operations have to be performed (such opening drawer at test time instead of closing it as learned in meta-training).
ML45 provides 45 highly different training tasks and 5 others to test the adaptation performance of meta-learmin meta-learning methods. This mode evaluates few-shot adaptation to new test tasks, after training on 45 different tasks. [T all 50 tasks.
19

---

Published as a conference paper at ICLR 2025
Train Tasks
Test Tasks
ML10
besketbol
button press
dil Turn
diawer dose
2ag Iree't
open
pull
place
sweep
Figure 10: Meta-world Tasks in the M10 protocol with training/test Split. Figure taken from al. 2019
Yu
Train Tasks
Test Tasks
cssembly
buttor topdowr
button wall
coffee button
topdown
cial turn
disossembee
door biose
ooor open
drower close
drawer open
"ouce" open
foucet close
hommer
DOX close
Fandle press
hondle press
handle pull
hardle pul
lever pull
peg insert
peg unplugg
plik out
pick place
door
val
sice
book
baok aide
reach wall
shelf place
90ccer
atck push
Stick
Sweep Into
Swoop
window
window close
hand Insert
Figure 11: Meta-world Tasks in the M45 protocol with training/test Split. Figure taken from Yu al.
2019
A.5 BASELINES
MAML PPO is the application of the MAML meta-learning algorithm for meta-RL (Finn al. 2017). The inner-loop algorithm is PPO, whereas the outer-loop algorithm is the MAML algorithm. We use the implementation provided the garage repo (garage contributors 2019).
RL uses an RNN in the inner loop described in Section A.1 and TRPO in the outer loop Duan al. 2016). We use implementation provided the garage repo (garage contributors: 201 which replaces TRPO with PPO.
PEARL is an off-policy meta-RI. algorithm, in contrast to the other methods we compare with Rakelly al.) 2019). We use the implementation provided in the garage repo (garage contribu
tors 2019).
VariBAD uses VAE to learn distribution of task features that increase the state space ZZintgraf ct al. 2020). In contrast to PEARL, VariBAD uses PPO in the outer loop, which is an on-policy meta- RL algorithm. We use the implementation provided by the authors.
TrMRL (Melo 2022) uses transformer in the inner loop of the RL² algorithm, similar to our proposed method. However, it takes sequence of the 5 most recent transitions input to the transformer. We use the implementation and pipeline as provided by the authors.
AMAGO (Grigsby al. 2024a) also uses transformer, but trains it using off-policy RL approach. It takes long sequences spanning multiple episodes as input to the transformer. We use the implementation provided by the authors.
20

---

Published as a conference paper at ICLR 2025
A.6 ADDITIONAL RESULTS
In this section, we show additional results for our evaluations on the MuJoCo, MetaWorld, and ManiSkill benchmarks.
HalfCheetah--el
HalfCheetah-
Ant-Dir
1500
-100
Average Return
1000
-200
-300
-400
1.0
Meta Learning Timesteps
Mcta: Meta-Lcarning Timestops 1e7
Meta: Leaming Timestops le7
Figure 12: Meta-Train performance in terms of average return of ECET, TrMRL, PEARL, MAML PPO, RI.2 PPO and VariBAD on the MuJoCo benchmark for training on parametric variations the HalfCheetah and Ant tasks.
ML1-Reach Train Performance
ML1-Reach Test Performance
1.0
1.0
0.8
0.8
Success Rate
0.6
0.6
0.4
0.4
Meta-ne Timestops
1e7
Meta-Learning Timestops
1e7
ECET
VariBAD
Figure 13: Meta-Train and Test performance terms of average success rate of ECET, TrMRL, PEARL, MAML PPO, RL2 PPO and VariBAD on the ML1 benchmark for training(left) and test- ing(right) on parametric variations of the Reach task.
A.6.1 VISUALIZATION OF EMBEDDINGS
We show the visualization of the embeddings learned by ECET and TrMRL for the ML10 Training tasks Figure [4] Furthermore, we show color maps the inter-task distance matrix of embeddings for TrMRL( and ECET(right) in Figure T5 There, it is more easily visible that the inter-task distance the embeddings much bigger for ECET than TrMRL, suggesting a better task differentiation.
A.6.2 AVERAGE RANKING PLOTS
To focus on comparing the methods based on their robustness in relative performance to each other, in Figures I6 to 18 provide plots showing the average ranks of the methods performance metric. These plots show that ECET robustly achieves the highest relative performance (and thus the lowest rank) across the meta-learning benchmarks consider.
21

---

Published as a conference paper at ICLR 2025
reach-u2
posho2
10
doon-sperv22
crawee-coose22
insert-
stndow-open-12
-20
-20
bastetball22
-20
20
10
10"
10
10
20
15
Figure 14: T-SNE plots of the output embeddings for TrMRL((lett and ECET(right) for the tasks in the training set of the ML10 benchmark of Meta-World
TrMRL
ECET
reach-v2
reach-v2
20
push-v2
push-v2
pick-place-v2
pick-place-v2
door-open-22
door-openwv2
15
drawer-clase-vv
drawer-clase-v2
button button-
press-topdown--2
Sutton-press-tort
press-topdowwn-2
peg-Insert-side-v2
peg-insert-side-22
window-open--22
window-open-v2
sweep-v2
sweep-v2
basketball-v2
basketball-v2
reach-v2
push-v2
pick-place-22
door-open-v2
Dower-close-v2
nutton-pressannt v
buttona peg inw.iit-eettee insert-side-e -idd--2
Window open-v2
sweep-v2
basketball-y2
reach-v2
push-v2
pick-place-22
door-open-v2
drawer-close-y close-22
button-press-tee stodswo-pd
peg arg-insert-ooe inssin--iidd--22 side-v2
window open-epee222
sweep-v2
basketball-v2
Figure 15: Color maps of the inter-task distance matrix of the embeddings for TrMRL,(eef) and ECET(right) for the tasks in the training set of the MI.10 benchmark of Heta-World.
22

---

Published as a conference paper at ICLR 2025
ML1-Push Train Performance
ML1-Reach Train Performance
Average Rank
5
2
5
1e7
1e7
ML1-Push Test Performance
ML1-Reach Test Performance
Average Rank
4-
Mota Leaming Timestops
1e7
Meta: Lone-Lnaming Timestops
1e7
ECET
Figure 16: Average ranking plots of Meta- Train and Test performance for all methods on the ML1 Push(left) and Reach(right). Lower ranks indicate better performance.
ML10 Train Performance
ML45 Train Performance
Averoge Rank
3
2-
5
1.07
107
ML10 Test Performance
ML45 Test Performance
6-
Averoge Bank
5
Timesteps
167
Meta-Leariin Timestaps
167
ECET
AMAGO
TAMAL
PEARL
MAML PPO
RL2
VariBAD
Figure 17: Average ranking plots of Meta-Train and Test performance for all methods on the ML.10(eft) and ML45(right) with disjoint train and test tasks. Lower ranks indicate better perfor- mance. The corresponding success rate plot can be found in Figure
23

---

Published as a conference paper at ICLR 2025
Performance
PickSingle Performance
Average Bank
StackCube Test Performance
PickSingleYCB Test Performance
Average Bank
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
Meta-Leaming Timesteps
167
Meta-Learming Timesteps
1e7
ECET
TAMAL
PEARL
MAML PPO
RL2 PPO
VeriBAD
Figure 18: Average ranking plots of Meta-Train and Test performance for methods on the StackCube(left) and PickSingleYCB(rigg)) tasks with disjoint train and test tasks. Lower ranks indicate better performance.
A.6.3 PERFORMANCE WITH RESPECT TO TIME
We show performance of the methods in terms of average success rate with respect to time in GPU hours in Figures 19 22 These plots show that most cases ECET achieves the best anytime performance across the meta-learning benchmarks we consider.
ML1-Push Train Performance
ML1-Push Test Performance
1.0
1.0
0.8
Success Bate
0.6
0.6
0.4
0.2
0.2
0.0
0.0
100
140
100
120
GPU Hours
GPU Hours
ECET
TIMRL
PEARL
MMML PPO
RL2 PPO
VariBAD
Figure 19: Meta-Train and Test performance in terms of average success rate with respect to time in GPU hours for all methods on the MLI Push task.
A.6.4 EVALUATION ACROSS EPISODES TEST TASKS
To assess the test performance of ECET, we perform rollouts for 50 episodes on test tasks and compare the episodic return for each episode to the baselines.
24

---

Published as a conference paper at ICLR 2025
ML1-Reach Train Performance
ML1-Reach Test Performance
0.3
0.8
Success Rate
0.6
0.6
0.4
0.4
0.2
0.2
20
40
60
80
100
120
140
0
20
40
60
80
100
120
140
GPU Hours
GPU Hours
ECET
TMMLL
PEARL
MAML PPO
RL2 PPO
VariBAD
Figure 20: Meta-Train and Test performance terms of average success rate with respect to time in GPU hours for all methods the MLI Reach task.
ML10 Train Performance
ML45 Train Performance
1.0
0.8
Success Rate
0.6
0.4
0.2
0.0
20
60
80
100
120
20
40
80
100
120
ML10
Performance
Performance
0.4
0.4
0.3
Success
0.2
0.1-
0.0
GPU Hours
GPU Hours
ECFT
THMRL
PEARL
MAML PPO
RL2 PPO
VariBAD
Figure 21: Meta-Train and Test performance in terms of average success with respect time in GPU hours for all methods on the ML10(left) and M1445(rghht with disjoint train and test tasks.
25

---

Published as a conference paper at ICLR 2025
StackCube Performance
PickSingleYCB Train Performance
1.0
0.5
0.8
0.6
Success
0.3
0.4
0.2
0.2
0.0
0.0
StackCube Test Performance
PickSingleyCC Test Performance
Success Rate
0.6
0.4
0.2
0.0
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
GPU Hours
GPU Hours
ECET
TAMAL
PEARL
MAML PPO
RL2 PPO
VariBAD
Figure 22: Meta-Train and Test performance in terms of average success rate with respect to time in GPU hours for all methods the StackCube(left) and PickSingleYCB(righ))) tasks with disjoint train and test tasks.
ML1 We sample 10 test tasks for Push-v2 and perform rollouts of 50 episodes with each of the resulting models from the meta-training using seeds for each method. For each episode plot the mean and standard error. Figure 23 shows the resulting plots, where it is visible that ECET generalizes well accross parametric varationns has not seen meta--raining, by achieving the highest episodic returns consistently accross episodes. This consistency since the first episode shows the independence of ECET to the experience available for the current task. This happens because during meta-training we make sure to train on transitions where we don't 't have the full history of 25 episodes available.
5000
4000
Ep sodic Retum
ECET
3000
RL2 PPO
MAML
2000
VariBAD
PEARL
1000
10
20
30
40
50
Figure 23:
Episodic return accross episodes on test tasks for all methods on the ML.1 Push Benchmark.
ML10 We sample 10 parametric variations for each of the tasks in the test of ML10 (Door Close, Sweep Into, Lever Pull, Shelf Place, Drawer Open) and perform rollouts 50 episodes with each of the resulting models from the meta-training using 5 seeds for each method. For each episode
26

---

Published as a conference paper at ICLR 2025
plot the mean and standard error. Figure 24 shows the resulting plots. It is visible that ECET generalizes well accross tasks it has not seen in meta-training, by achieving the highest episodic returns consistently accross episodes. However, for some tasks, there is more variance performance accross seeds, as well as accross episodes, which indicates more difficulty to adapt. Similarly as for the MI.1 benchmark, there is drop in performance for ECET in the first episodes where have less observations, further demonstrating the independence of ECET to the experience available for the current task.
door-close-22
sweep-into-v2
lever-pull-v2
4000
4000
2000
Episodic Retum
3000
Episodic Retum
3000
Epicodic Retum
2000
2000
1000
1000
1000
500
0
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Episode
shelf-place-v2
drawer-open-v2
1500
3500
1250
3000
Episodi: Return
1000
Epitade Return
2500
2000
750
ECET
1500
TIMRL
500
1000
MAML PPO
250
VariBAD
0
10
20
30
40
50
10
20
30
40
50
Episoda
Epicoda
Figure 24: Episodic return accross episodes each test task for all methods the ML10 Benchmark.
ML45 We sample 10 parametric variations for each of the tasks in the test set of ML10 (Box Close, Bin Picking, Door Unlock, Hand Insert, Door Lock) and perform rollouts of 50 episodes with each of the resulting models from the meta-training using seeds for each method. For each episode plot the mean and standard error. Figure 25 shows the resulting plots. Compared to the baselines, ECET manages to get higher episodic returns on tasks has not seen in meta- meta-training, similarly to the ML1 and ML10 benchmark. However, for ML45, the variance in performance is higher accross episodes, which indicates more difficulty adapt. For the Door Unlock task, is noticeable that TrMRL manages to perform similarly to ECET. The other baselines fail to reach high episodic return. Similarly for the ML1 benchmark, there is no drop in performance for ECET in the first episodes where we have less observations, further demonstrating the independence of ECET to the experience available for the current task.
A.6.5 ABLATIONS
To investigate the behavior of ECET and the impact of its components on performance, we ablate the number of episodes E' to keep in memory and the number of transitions 7 in the sampled sequence in
Figure 28 The configuration that performs best, and the one we use in our experiments is E 25,
T = 5. We ablate the sampling strategy (sampling one transition sequence per episode vs. sampling multiple sequences randomly from any episode) in the top plot in Figure 29 to show the usefulness of sampling one transition sequence per episode. In contrast to TrMRL, which only uses the output of the transformer as input to the policy т,, we concatenate a linear transformation of the current state input to the policy. We ablate the performance of ECET for these two settings of policy input in the middle plot in Figure 29 In the bottom plot, we ablate the number of encoder blocks for IET and CET. To differentiate the contributions of the context sampling strategy from the architecture in
27

---

Published as a conference paper at ICLR 2025
box-close-v2
bin-picking-v2
door-unlock-v2
1750
M MMm
2500
1500
2000
Episadic Retum
2000
MMN
Episadic Retum
1250
Episadic Retum
1500
1500
1000
1000
1000
0
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Episode
Episode
Episode
hand-insert-v2
door-lock-v2
3000
2500
Eptsodi: Retur
2500
Eplcodi:: Return
2000
2000
1500
ECET
15000
1000
RL2 PPD
MAML FPO
500
500
PEARL
20
30
40
30
40
Episade
Episode
Figure 25: Episodic return accross episodes on each test task for all methods on the ML45 Benchmark.
ECET, in Figure 26 we compare ECET to a variant using positional encoding for CET (ECET (pos)) and using flat architecture with only IET (ECET (flat)). This version processes context that is sampled exactly the same as in ECET, specifically sequences of length S are randomly sampled from E episodes and then put through the transformer. ECET (flat) yields inferior performance, underscoring the challenges non-hieran non-hierarchical approach faces in capturing comprehensive task representations. Finally, Figure 27 we also compare ECET TrMRL, and TrMRL (extended context) that processes sequences of the same length as the total number of transitions that ECET processes, namely sequences of length ExS. We extend the context of TrMRI investigate the effect that the context length has on the performance improvements we see with ECET. The low success rate of TrMRL (extended context) shows that extending the context length of TrMRL makes the meta-training less efficient, as much more training data is needed to extract useful task meta-features from these long sequences. Our benchmark configuration for these experiments stands at E = 25, T = 5,incorporates two encoder blocks for both (ET(with positional encoding) and CET(without positional encoding), and uses a linear transformation yo2 of the state as additional input to the policy π. We do all ablations in the MI.1 Benchmark, using parametric variations of the Push task.
Figure 26 shows the performance of our default configuration of the method, ECET, compared flat architecture
using positional encoding for СЕТ, ECET (pos), and the simplification akin to TrMRL by solely employing IET, ECET (flat). We show the average and standard error of the success rate accross Meta-Learning Timesteps. ECET (flat) performs worst, indicating the difficulty of flat architecture to capture a better representation of the task, instead of focusing on local charactersitics of the dynamics. ECET (pos) has a similar rate of improvement beginning, but ends up converging to suboptimal representation of the task, and thus its success rate does not improve above ~ 0.8 after 2.5 10 imesteps. This indicates that the postional encoding in CET induces ordering in the Zeey representations which is misleading, episodes don't 't naturally have a causal ordering. thus imposing an ordering on the representations we get from the sequences of transitions sampled ends up hurting performance.
ECET in the
Figure 27 shows the performance of our default configuration of the method, ECET, compared to TrMRL that processes sequences of length and TrMRL (extended context) that processes sequences of length 125. We extend the context TrMRL to investigate the effect that the context length has on the performance improvements
with ECET. The low success rate of TrMRI (extended
28

---

Published as a conference paper at ICLR 2025
ML1-Push Train Performance
Succass Rate
ECET
(pcs)
ECET (fltt)
Meta-Leaming Timestops
x10
Figure 26: Investigating the meta-training performance impact on ECET of the inter-e feature extractor CET and the positional encoding on the sequence embeddings Zseq before being input into CET in the MI.1 Push Benchmark. ECET shows the performance of the configuration we propose, with IET using positional encoding, and CET without positional encoding. ECET (pos) shows the performance of the configuration with IET and CET both using positional encoding. ECET (flat) shows the performance of the configuration with no hierarchy of transformers, only IET using positional encoding, taking the same information input ECET.
context) shows that extending the context length of TrMRL makes the meta-training less efficient, as much more training is needed to extract useful task meta-features from these long sequences. Thus, the ECET architecture of encoding shorter sequences from different episodes with IET, and then processing those encodings with CET is crucial in ensuring better anytime performance than TrMRL.
ML1-Push Train Performance
1.0-
0.8
Succecs Rate
0.6
0.4
0.2
ECET
0.0
(extended context)
Meta-Leaming Timesteps
x10*
Performance of ECET, TrMRL with its standard sequence length of 5, and TrMRL with extended context (processing sequences of length 125 to equal the context length of ECET).
Figure 27:
To investigate the performance sensitivity of ECET on the number of episodes (E) kept memory and the samples sequence length (T), we provide ablation results in Table
Figure 28 shows the performance ECET, when varying key parameters and structural elements: the number of episodes (E) in memory, the length of sampled transition sequences (?). The top plot compares the impact of varying E = (5, 10, 15, 20, 25, 50) when T 5. It is noticeable that the configuration does not manage to reach a success rate of compared to the other configurations. On the other side of the spectrum, decreasing E. to 5 results in slower improvement. The configuration that use, F 25, shows
increasing E' to 50 results in suboptimal performance.
29

---

Published as a conference paper at ICLR 2025
E
10
15
20
25
50
0.997
0.005
1.000
0.000
0.997
0.005
1.000
0.000
1.000
0.000
0.909
0.058
25
0.997
0.005
1.000 1: 0.000
1.000 1 0.000
0.993
0.009
0.924 1 0.053
1.000 1 0.000
50
0.963 0.039
0.960 1 0.050
0.993 1 0.005
0.827
0.098
0.990 1 0.014
0.893 1 0.041
Table ECET's final meta-training success-rate on MI.1 Push when varying the number of episodes (E) and the sequence length (T).
the fastest improvement and better performance accross the Meta-Learning Timesteps. The middle plot compares the impact varying = {5, 10, 15, 20, 25, 50} when T 25. noticeable that slower to improve than the configurations using T = 5, and that
all of the possible configurations because with higher sequence length, ECET needs much more data and training to arrive at good representation Zaeg and Itask- This claim is further supported by the bottom figure, showing slower improvement for configurations with = {5, 10, 15, 20, 25, 50} when 50.
ML1-Push Train Performance
1.0
0.8
Suceess Rate
0.6
ECET (E-5, T-5)
0.4
ECET (E=10,
ECET (E-15, T-5)
0.2
(E+20, T-5)
ECET (E+25, T+5)
ECET (E-50, T-5)
0.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.8
Succoss Rate
0.6
(E-5, T-25)
0.4
T=25)
ECET (E-15, T-25)
0.2
ECET (E=20, T=25)
ECET T-25)
0.0
ECET (E=50,
T-25)
0.2
0.4
0.6
1.0
1.2
Success Rate
0.6
ECET (E+S, T=50)
0.4
T-50)
ECET (E=15, T=50)
ECET (E-20, T-50)
ECET (E-50, T-50)
0.0
0.4 Meta-Leaming Timestegs
1.0
1.2
Figure 28: Investigating the meta-training performance impact of the sequence length T and number of episodes E for ECET in MLL Push. The top plot shows the performance of sequence lengths T with different episode lengths E E {5, 10, 15. 20, 25, 50). The middle plot shows the performance of 25 with E E (5. 10, 15. 20, 25, 50). For the bottom plot, 50.
30

---

Published as a conference paper at ICLR 2025
singular transition sequence per episode to a more randomized approach drawing multiple sequences from any episode. All of the shown configurations strategy as our default. It is noticeable that when sampling sequences from any randomly chosen episode, the performance improvement is slower. However, for the F 50, T configuration, the randomization of sampling helps ECET not get stuck in suboptimal regions, and thus it manages to reach a success rate of 1.0 compared the same configuration using the non-randomized strategy of sampling with respect to episodes, which we see in Figure 28 Despite this, all the configurations using the randomized strategy have slower improvement compared to our default configuration using the non-randomized strategy. The middle plot shows the performance comparison of augmenting the policy и input with a linear transformation of the state(ECET) versus relying exclusively on the transformer's output(ECETT)) We notice a significant improvement from getting the linear transformation the state as imput in the policy. The bottom plot shows the influence of varying the number of encoder blocks within the architecture. Although it has least parameters compared to the other configurations, ECET(2, 2) performs best, having the fastest rate of improvement and also best final performance.
Figure 29 top shows ablations of sampling strateyy-
the latter strategy, in Figure 28 and everywhere clse we use the former
A.6.6 RANK AND SUCCESS RATE PLOTS PER TASK
Through investigating the individual plots for each task in terms of average success rate in Figures 30 34 the difference in the level of difficulty for the tasks can be demonstrated, noticing that c.g. in Figure the Door Close task is casier to successfully perform after meta- meta-training, reaching success rates of approximately 0.6, compared to the Shelf Place task, for which all the methods fail to reach a success rate higher than 0.12. This is also the case for the performance per task for the test tasks in ML45, which we show in Figure 34 For the Bin Picking task, none the baselines can successfully finish even one episode, except for VariBAD with success rate of approximately 0.05 (ECET being more robust, reaching 0.3). On the other hand, for the Door Unlock task, RL2 PPO manages to reach success rates of approximately 0.4, and TrMRL reaches 0.6, being one of the few cases where ECET underperforms by reaching a success rate of 0.3.
31

---

Published as a conference paper at ICLR 2025
Train Performance
1.0
0.8
Success Rate
0.6
(E=5,
ECET (E-10, T-
ECET (E+15, T-
ECET (E-20,
ECET (E+25, T-5)
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
107
Succens Rata
0.4
0.2
ECET
0.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
167
Succede Rate
2)
ECET (2, 4)
0.0
ECE T (4, 2)
0.0
0.2
0.4
0.5
0.8
1.0
1.2
1.4
Meta-Leaming Timesteps
167
Figure 29: Investigating the meta-training performance impact of the sampling strategy for the sequence for ECET in ML1 Push. The top plot shows the performance of ECET when sampling E sequences from random episodes (i.e. more than one sequence per episode). The middle plot shows the performance impact of concatenating linear transformation of the state(ECET) using only the output of the transformer(ECETT)) as the input for the policy. The bottom plot shows the performance impact of the number of encoder blocks, comparing configurations of encoder blocks for each IET and CET, encoder blocks for IET and 4 for CET, and encoder blocks for IET and for CET.
32

---

Published as a conference paper at ICLR 2025
window-open--v2
peg-insert-inside-22
reach-v2
drawer-close-v2
1.0
1.0
1.0
1.0
Succiss Rate
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
4
1e7
1e7
1e7
1e7
basketball-v2
push-v2
sweep-v2
door-open-v2
1.0
1.0
1.0
1.0
Success Rate
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
MLATT
2
107
1e7
Timesteps
1.67
Timesteps
1+7
pick-place-v2
pentoo-prees--ooddoo
1.0
1.0
0.8
0.8
Success Rate
0.6
0.6
0.4
0.4
0.2
MnV
0.2
Timestops
1e7
Timestops
1e7
ECET
AMAGO
TrMRL
RL2 PFO
MAML PPO
VariBAD
PEARL
Figure 30: Success rate per train task for the ML10 benchmark for ECET, TrMRL, PEARL, RL2 PPO, MAML PPO, and VariBAD.
33

---

Published as a conference paper at ICLR 2025
door-close-v2
sweep-into-v2
lever pull-v2
0.6
0.6
0.5
0.20
Success Rate
0.4
0.3
0.2
0.05
0.1
1c7
1c7
Timesteps
107
shelf-place-v2
drawer-open-v2
0.175
0.5
Success Rate
0.3
WT MM Moloor
0.075
0.2
0.050
0.025
0.000
0.0
Timesteps
Timesteps
AMAGO
TYMRL
PPO
MAML PPO
VarIBAD
PEARI
Figure 31: Success rate per test task the ML10 benchmark for ECET, TrMRL, PEARL, RL2 PPO, MAML PPO, and VariBAD.
34

---

Published as a conference paper at ICLR 2025
drawer-close-v2
push-v2
shelf-place-v2
door-close-v2
0.8
Success Rate
0.75
0.6
0.2
0.75
0.50
0.50
0.1
0.2
0.00
0.0
0.0
0.00
1c7
1c7
1c7
1c7
push-wall-v2
plate-sllde--2 -/2/
handle- wwwpppeeessssse -ide---2
sweep-v2
1.00
0.8
0.6
Success Rate
0.6
0.75
0.6
0.4
0.50
0.4
0.2
0.0
0.0
0.00
0.0
le7
1c7
1e7
107
coffee-push-v2
window-open-vv
faucet-close-v2
faucet-open-v2
1.00
1.00
1.00
Success Rate
0.6
0.75
0.75
0.75
0.4
0.2
0.25
0.25
0.25
0.0
0.00
0.00
0.00
1e7
167
pick-place-v2
soccer-v2
button- prrton-press---e topdown-V
plate-slide back------b22222
1.00
1.00
1.00
Success Rate
0.75
0.4
0.50
0.25
0.2
0.25
0.25
3
lever-pull-vz
handle- pill-side-v2
stick-pull-ve
1.00
1.00
1.00
1.00
Success Rate
0.50
0.50
0.30
0.50
0.25
0.25
0.25
0.25
MAX
0.00
0.00
2
0
2
3
4
1e7
1e7
1e7
le7
coffee-button-
hammer-v2
handle-pull-v2
peg bog-insser-sidees
0.20
(WA
Success Rate
0.75
0.35
0.75
0.4
0.50
0.10
0.50
0.2
0.00
0.00
0.00
0.0
2
Timesteps 1e7
Timesteps 1e7
Timesteps
1e7
Timesteps
1e7
Figure 32: Success rate per train task for the ML45 benchmark for ECET, TrMRL, PEARL, RL2 PPO, MAML PPO, and VariBAD (Part 1).
35

---

Published as a conference paper at ICLR 2025
door-open-v2
disassemble-v2
assembly-v2
plate-slide llateslide-baccksidev
Success Rate
0.75
0.75
0.50
0.2
0.50
0.1
0.00
0.0
0.00
2
2
1c7
1c7
1c7
167
reach-v2
orawer-open-v2
dial-tuum-v2
pick-aut-of-halee--
1.00
1.00
1.00
Success Rate
0.75
0.75
0.75
0.3
0.50
0.50
0.2
0.25
0.1
0.00
0.00
0.00
0.0
1c7
le7
1c7
1e7
stick-push-v2
basketball-v2
push-back bach-bbak--2
coffee-pull-v2
0.3
Success Rate
0.2
0.6
0.6
0.2
0.4
0.1
0.2
AAW
0.0
0.0
0.0
187
sweep-inter -ieeppinto-v2
button-pres
towdown-wwlww12222222 propppproooooowwwwallll
handle- mandle-press-ve press-v2
reach-wall-v2
1.00
1.00
1.00
1.00
Success Rate
0.75
0.75
0.50
0.50
0.25
0.25
0.25
0.25
167
button- wwwnn-peess--ae wwwess-wwlllve wall-v2
peg unp weg--unpuugsiice uidside-2
plate-slide-side-w2
window-clee www.wwww-wooov...oo
1.00
1.00
1.00
Success Rate
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
3
0
2
1e7
Timesteps 1e7
Timesteps
1e7
Timestops
1e7
pick jick-place--ee wick-paacewwall-22
0.3
Success Rate
0.1
0.0
Timesteps 1e7
VariBAD
Figure 33: Success rate per train task for the ML45 benchmark for ECET, TrMRL, PEARL, RL2 PPO, MAML PPO, and VariBAD (Part 2).
36

---

Published as a conference paper at ICLR 2025
box-close-v2
bin-picking-v2
door-unlock-V
0.35
1.0
0.3
0.30
0.8
Success Rate
0.25
0.2
0.20
0.15
0.4
0.1
0.10
0.2
5.
1e7
127
Timesteps
107
hand-Insert-v2
door-lock-v2
0.8
0.4
0.6-
Success Rate
0.3
0.4
0.2
0.1
0.2
110
0.0
Timesteps
Timesteps
ECET
AMAGO
TrMRL
PPO
MAML PPO
VarIBAD
PEARI
Figure 34: Success rate per test task the ML45 benchmark for ECET, TrMRL, PEARL, RL2 PPO, MAML PPO, and VariBAD.
37

---

Published as a conference paper at ICLR 2025
window-openn-v2
peg-insert-inside-2
reach-v2
drawer-clase-v2
Average Rank
0
5
1
2
2
167
167
167
167
baskettb I1-v2
push-v2
sweep-v2
door-open-v2
Average Rank
5
1
2
Timesteps
Timesteps
1e7
pick-place-v2
button-
Potton-press-eee press topdow
Average Rank
Timesteps
Timestops
ECET
AMAGO
TrMRL
RL2 PFO
MAML PPO
VariBAD
PEARL
Figure 35: Average rank per train task for the ML10 benchmark for ECET, TrMRL, PEARL, RL2
PPO, MAML PPO, and VariBAD.
38

---

Published as a conference paper at ICLR 2025
door-close-v2
sweep-into-v2
lever-pull-y2
Average Rank
9
3-
167
1.7
Timesteps
147
shelf-place-v2
drawer-open-v2
my
5
Average Rank
Timestops
1e7
Timesteps
1e7
TrMRL
PPO
MAML PPO
VarIBAD
PEARI
Figure 36: Average rank per test task for the ML10 benchmark for ECET, TrMRL, PEARL, RL2 PPO, MAML PPO, and VariBAD.
39

---

Published as a conference paper at ICLR 2025
drawer-close-v2
push-v2
shelf-place-v2
door-close-v2
Average Rank
OXX
5
4
1n7
107
167
167
push-wall--y
plate-s serees-lideee -2
handle peiiee--rsss--- press-side-s- side-v2
sweep-v2
Average Rank
0-
2
1e7
faucet-close-v2 107
faucet-open-v2 1e7
coffee-push-ie
window- open-v2
Average Rank
pick-place-v2
Saccer-v2
button www.own-dww
plate-siie slide back
5
Average Rank
lever-pull-v2
button-press--eee
handle- cesele-ppul-siddeeee
stick pull-pp
Average Rank
button-2
hammer-v2
pull-v2
peg-insert-sider
Average Bank
Timestops
Timestops
Timesteps 1e7
Timmstopp
Figure 37: Average rank per train task for the ML45 benchmark for ECET, TrMRL, PEARL, RL2 PPO, MAML PPO, and VariBAD (Part
40

---

Published as a conference paper at ICLR 2025
door-open-v2
disassemble-v2
assembly-v2
plate-slidebbckksidde
Average Rank
4.
2
1
2
3
107
drawer-open-v2 107
167
107
reach-v2
dial-turn-v2
pick out- of of-hole-22
Average Rank
0-
4-
2
3
3
le7
basketball-v2 1e7
le7
1e7
Stick-push-v2
push bush-back-v2
coffee: pulleepplllv
Average Rank
2
4
1c7
167
reach-wall-v2 1c7
sweep-into-v2
button- www.ooo-dowwooooooo wall
handle- pereee--rees22
Average Rank
button- cutton-press-walt wall-v22
peg-unplug side-v2
Hate-slide-side-V>
window-close-v2
Average Rank
2
LMV LAAN
Timesteps
Timesteps
Timesteps
pick-place-wall-v2
Average Park
9.
Timestops
Figure 38: Average rank per train task for the ML45 benchmark for ECET, TrMRL, PEARL, RL2 PPO, MAML PPO, and VariBAD (Part 2).
41

---

Published as a conference paper at ICLR 2025
box-close-v2
bin wwn--pickin---
door-unlock-v2
Average Rank
5
167
1.7
Timesteps
147
hand-insert-v2
door-lock-v2
Average Rank
Timestops
1e7
Timesteps
1e7
TrMRL
PPO
MAML PPO
VarIBAD
PEARI
Figure 39; Average rank per test task for the ML45 benchmark for ECET, TrMRL, PEARL, RL2 PPO, MAML PPO, and VariBAD.
42


Reviews:
[{"review_id": "aIhsTkNHJm", "reviewer": "Reviewer_ixfQ", "summary": "The paper proposes new online Meta-RL algorithm that is able to combine intra- and cross-episodic information to better make decisions under uncertainty. Each intra-episode is encoded into a latent variable by a transformer model, which are then processed altogether by another transformer and a context-specific latent is formed. Authors claim SOTA results on Maniskill and challenging Meta-World ML45 benchmarks, surpassing previous SOTA methods.", "questions": "1. I would appreciate if authors include an additional baseline method on ML45, AMAGO [1], to ensure the result of the proposed method is indeed SOTA.\n2. To justify SOTA performance, I would also like to see the experiments on TrMLR with transformer or mamba architecture as an outer-loop memory model.\n3. I would also appreciate it if authors provide how many hyperparameters were swept for the baseline TrMLR method and which method for HPO was used.\n\nAlthough my initial recommendation is to reject the paper, I would be happy to revise my score as the authors address my questions. \n\n[1] Grigsby, Jake, Linxi Fan, and Yuke Zhu. \"Amago: Scalable in-context reinforcement learning for adaptive agents.”\n\n*After rebuttal update:* I changed my score to 6 since the questions were mostly addressed, see the comments for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper reports state of the art results on challenging tasks, including challenging environment Meta-World ML45.\n- The method is explained clearly and visually explained.", "weaknesses": "[W1] Authors compare their method with TrMLR, which uses RNN in the outer-loop, while authors of the proposed method use transformer. I believe that the difference in the capacity can explain the performance distinction between two methods. To my mind, it is early to claim SOTA results without trying to match architectures first.\n\n[W2] A strong baseline is missing from the paper: AMAGO [1]. The [1] authors do not report the results on ML45, stating it is left for the future work. However, since the main contribution of the paper is SOTA results, I believe AMAGO should be included in the baselines. Also, mind that AMAGO was introduced in ICLR”24.\n\n[W3] Authors do not state whether the baselines (especially TrMLR) were tuned to get the best performance out of them. Again, as the paper claims SOTA results, it is vital to have this information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dVOYP59rso", "reviewer": "Reviewer_iQfe", "summary": "This paper proposes a method for learning features for meta-reinforcement learning in latent space by training a two-level hierarchy of transformer networks. The lower layer is trained to represent trajectories experienced while doing a specific task (i.e. experiencing a single MDP) while the upper layer is trained on all tasks. This method is shown to have significantly better performance than several other meta-RL methods on a variety of meta-RL benchmarks, with the primary hypothesis for this being that the proposed architecture allows the networks to learn more general features that are informative across a variety of tasks. This hypothesis is supported by some experiments showing that embedded representations learned by the proposed method tend to cluster together by task more than representations learned by other methods.", "questions": "“Hypothesis 1: ECET captures more general task features through its ability to capture intra- as well as cross-episodic experiences.” \n\nI recognize claims about feature generality are difficult to substantiate, and that it is somewhat of a catch-22 to try to deal with these topics experimentally. I appreciate the authors for making an attempt. I am interested in their thoughts regarding essentially more rigorous versions of such experiments.\n\n- Do you have a definition for the generality of a task feature? How does one measure this? \n\n- You’ve shown that embedding vectors tend to cluster for some tasks. Do you think there are any possible confounders beyond generality of features?\n\n- Moreover, if one were to take a more systematic and focused approach to quantifying the difference between tasks so that embeddings generated from those tasks may be compared to the embedding distances --- similar to what is presented in figure 13 but with some notion of how similar two columns/rows ought to be --- do you suspect they would encounter anything surprising?\n\nIs there anything that makes such an experiment difficult to execute other than initially deciding how to measure difference between tasks? I agree these results seem promising w.r.t. this hypothesis, but there are definitely a few more logical steps to nail down before this can be determined without doubt (which would be an important result in it own right, of course).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Paper is overall clearly written and explained.\n\nRelated work positions the paper well.\n\nExperiments are well designed.\n\nI am a big fan of explicit hypothesis statements, and very much appreciate this paper for being clear about what it is investigating.\n\nThe idea is relatively simple and seems like a clear opportunity for more research.\n\nThe performance is impressive and the results are quite thorough.", "weaknesses": "There are few weaknesses in my opinion. Instead, I will give some minor feedback.\n\n- “By considering both intra-episode experiences (information within a single episode) and cross-episode experiences (information across multiple episodes), our method can learn richer meta-features for the environment.” The opposite of intra is inter, not cross. The title already contains 'cross', but consider using this term when explaining (I believe it already occurs once towards then end).\n\n- Consider citing Bhatia et al. (RL^3: Boosting Meta Reinforcement Learning via RL inside RL^2) which also uses transformers and learns state-augmentation features for meta-RL, albeit not through inter-episode data sharing.\n\n- Figure 2 bleeds into the margins.\n\n- It could be clearer which baselines (or other meta-RL methods) this method is compatible with and which are mutually exclusive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FAoaFZ6PFJ", "reviewer": "Reviewer_khht", "summary": "This paper proposes a new algorithm; Efficient Cross-Episodic Transformers (ECET) for online meta-reinforcement learning. Their method tackles the problem of enabling reinforcement learning agents to perform effectively in previously unseen tasks. In order to achieve that, they take past episodes and use them as in-context information, considering different task variations and single transitions by not only extracting information from within sequences of transitions but also incorporating information across sequences sampled from different episodes; thus, integrating insights from prior meta-RL research. Their main contribution is improving the generalization capabilities and providing more efficient meta training, outperforming previous state-of-the-art methods. They perform an empirical study in the Meta-World and Maniskill Benchmarks to conclude that their extended algorithm captures more general task features and outperforms the state-of-the-art in online meta-RL algorithms in few-shot adaptation to parametric variations of tasks and out-of-distribution tasks. \nOverall, this paper could be a significant algorithmic contribution, with the caveat of some clarifications on the theory and experiments. Given these clarifications in an author's response, I would be willing to increase the score.", "questions": "For the theory, there are a few steps that need clarification and further explanations on novelty. For novelty, it needs to be clarified if their method is being stated as a novel result. It looks like the method has already been shown in \"Transformers are Meta-Reinforcement Learners.” There is a statement that “TrMRL is the closest baseline to our proposed method (Melo, 2022). It uses a transformer in the inner loop of the RL2 algorithm. It takes a sequence of the 5 most recent transitions as input to the transformer. We use the implementation provided by the authors.” Is your algorithm 2 somehow an extension? Is your theorem 3 completely new?\n\nIn Appendix A.2.3 Procedural Components, you mentioned that “SampleTransitions: We sample the transition sequences of length T from each episode in H. For past episodes, we randomly” You randomly did what?\n\nThere is also one concept in the algorithm that I cannot verify. How did you apply the intra-episode and cross-episode simultaneously? More about this would be illuminating\n\nFor the experiments, the following should be addressed:\n- The central contribution is enhancing the generalizing capabilities. The authors concluded that their method is capable of adjusting to completely unseen tasks that share the same state and action spaces as the training tasks. Despite outperforming state-of-the-art methods in online meta-RL by efficiently compressing cross-episodic knowledge, their method’s generalization to tasks that are completely different from the meta-training set remains limited as it is tailored to exploiting similarities. I would like to see some discussions on this and I think it would be beneficial to demonstrate that empirically.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "For originality, the paper presents a novel method for learning meta-features through efficient cross-episode meta-RL, using a novel transformer architecture that enhances the adaptability of RL agents across diverse sets of tasks. For significance, their approach leverages intra-episode experiences and cross-episode experiences, providing a more comprehensive learning process and integrating ideas from previous meta-RL research. From a perspective of clarity and quality, their learned algorithm is capable of outperforming the previous state-of-the-art and providing more efficient meta-training while significantly improving generalization capabilities. In general, the proposed method enhances the agent’s ability to generalize from limited data supported with sufficient empirical experiments and paves the way for more robust and versatile AI systems.\n\nThe paper has some good points to highlight and summarize:\n- Improving generalization capabilities and providing efficient meta-training. \n- Outperforming previous state-of-the-art methods by capturing complex, multilevel patterns in the data.\n- Integrating multiple ideas from previous meta-RL research.", "weaknesses": "Based on my experience, while I truly appreciate the authors’ effort, I believe that their paper lacks novelty and can be considered as an improvement but not as a real contribution for several reasons. First of all, there is no central contribution beyond improving generalizing capabilities by introducing a new architecture and outperforming previous SOTA–which was supported in the empirical results in figures 4, 5, and 6— still, their method’s generalization to tasks that are completely different from the meta-training set remains limited as their approach is tailored to exploiting similarities as shown in hypothesis 3: ECET outperforms the state-of-the-art online meta-RL in out-of-distribution (OOD) tasks. This leads to Section 1, Related Work, it is still unclear how their method differs from other baselines in solving the same problem, specifically the TrMRL method Melo (2022), which is the closest to the authors’ proposed algorithm. In other words, the authors’ addition and value are still vague compared to previous research tackling the same research question. Having read Melo (2022), I see that their cross-episode experience equals the episodic memory system proposed in the TrMRL method. More discussion on these topics would be helpful. \n\n\nMinor comments:\n1. Page 6: Line 4, ‘ihas’ should be corrected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "85D5goV4cW", "reviewer": "Reviewer_URVA", "summary": "This paper presents an online Meta-RL method. The authors suggested that decomposing the encoder into an intra-episode stage (where each episode is encoded separately) and cross-episode stage (where the vectors from the previous step are encoded into a single representation of the context) should improve performance both in terms of few-shot adaptation and in-distribution and out-of-distribution generalization.\nThe authors validated their claim on the Meta-World and ManiSkill benchmarks and compared against popular Meta-RL benchmarks.", "questions": "1. In the title of the paper you suggested the method is \"Efficient\", but did not mention in what sense (not in the introduction at least). Do you mean in terms of time complexity (as discussed in lines 267-274)? If so, mention it in the introduction and add some results showing latency/FLOPs compared to \"standard\" (line 268) approaches.\n2. In line 211 you mentioned you sample a sequence of $T$ transitions from each episode.\\\n    a) How do you sample the transitions? Uniform sampling? With replacement?\\\n    b) What is the chosen value for the different benchmarks? Did you test the effect of this value on the runtime/performance?\\\n    c) DId you also sample the transitions when running the other baselines? \n3. Why did you choose to add positional encoding to the IET?\n4. What are the number of episodes, length of each episode, and $T$ in each benchmark?\n5. Did you use the same hyperparameters for all benchmarks? If not, how sensitive is your method to different hyperparameters?\n6. How did you choose the hyperparameters for the baselines? \n7. How many seeds did you use for the experiments in Section 5?  \n8. In Figure 5 the results of TrMRL don't match the ones in the TrMRL paper. Do you have an explanation for this? Did you run this baseline as done in the original paper (connecting to questions 2c and 6)?  \n9. How many learnable parameters does your architecture have compared to the other baselines (especially TrMRL)?  \n10. Typo line 273: ihas -> has", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The paper is well written, especially the Method and Experiments sections. The formulation and explanation of the algorithm can be well understood\n2. The results shown in the Experiments section look quite promising, including results over the widely used Meta-World benchmark \n3. The authors provided an open-sourced code to reproduce the results", "weaknesses": "1) My main concern regarding this work is the novelty of the proposed method. The overall algorithm is very similar to previous lines of work, especially RL2 and TrMRL with the main difference from TrMRL is the decomposition of the transformer into two stages. For this change to have a profound enough impact on the field I would expect at least one of the following: \n\n      a) A theoretical/analytical analysis of why this specific architecture should generalize/perform better than others. The authors gave high-level explanations (e.g lines 204-209), but I found these not rigorous enough. For example, there is no sufficient motivation for hypothesis 1 (line 368), regular recurrent architectures should also be able to capture intra and cross episodic experiences, why should we expect ECET do it better? \\\n      b) A more thorough empirical study showing the benefits of the chosen design, including more baselines, benchmarks, and an ablation study (more on that below)  \n\n2) The positioning of the paper concerning related work is not clear. For example, in lines 52-53/148-149 in the Introduction Section, the authors claim that previous methods used \"single transitions or sequences of transitions from the same episode\" to encode the context and state that incorporating cross-episodic information is one of the novelties of the proposed method. This claim is false as many previous works (e.g. VariBAD, TrMRL...) encoded transitions from multiple episodes into a single representation. \n\n3) The empirical study, which is shown in Section 5, is not sufficient to support the claims of the authors (mainly hypothesis 1 and 3):  \\\n    a) More benchmarks including ones from the original papers (e.g. VariBAD, RL2) are missing. It is not clear if the results will hold on benchmarks where sufficient hyperparameter tuning of the other baselines took place\n\n    b) An ablation study is missing, especially over the disentangled transformer architecture\n\n    c) Regarding hypothesis 3 - the authors did not compare to theoretical/empirical line of works in generalization in Meta-RL (e.g [1,2])\n\n[1] Lee et al. Improving generalization in meta-rl with imaginary tasks from latent dynamics mixture \\\n[2] Rimon et al. - Meta Reinforcement Learning with Finite Training Tasks -- a Density Estimation Approach", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)