Cross-Episodic Curriculum for Transformer Agents
Authors: Lucy Xiaoyang Shi
Yunfan Jiang
Jake Grigsby
Linxi Fan
Yuke Zhu
Venue: NeurIPS 2023 poster (2023)
Keywords: Transformers
In-context Learning
Reinforcement Learning
Robotics
TL;DR: Cross-Episodic Curriculum enhances Transformer agents’ learning efficiency and generalization by leveraging cross-episodic experiences, proving effective across RL and IL settings.

Abstract: We present a new algorithm, Cross-Episodic Curriculum (CEC), to boost the learning efficiency and generalization of Transformer agents. Central to CEC is the placement of cross-episodic experiences into a Transformer’s context, which forms the basis of a curriculum. By sequentially structuring online learning trials and mixed-quality demonstrations, CEC constructs curricula that encapsulate learning progression and proficiency increase across episodes. Such synergy combined with the potent pattern recognition capabilities of Transformer models delivers a powerful cross-episodic attention mechanism. The effectiveness of CEC is demonstrated under two representative scenarios: one involving multi-task reinforcement learning with discrete control, such as in DeepMind Lab, where the curriculum captures the learning progression in both individual and progressively complex settings; and the other involving imitation learning with mixed-quality data for continuous control, as seen in RoboMimic, where the curriculum captures the improvement in demonstrators' expertise. In all instances, policies resulting from CEC exhibit superior performance and strong generalization. Code is open-sourced on the project website https://cec-agent.github.io/ to facilitate research on Transformer agent learning.

Cross-Episodic Curriculum for Transformer Agents
Lucy Xiaoyang Shi'r, Yunfan Jiang" Jake Grigsby', Linxi "Jim" Fan Yuke 2hu
Stanford University
"The University Texas at Austin "Equal contribution "Equal advising
"NVIDIA Research
Abstract
We present new algorithm, Cross-Episodic Curriculum (CEC), to boost the learn- ing efficiency and generalization of Transformer agents. Central to CEC is the placement of cross-episodic experiences into a Transformer's context, which forms the basis of curriculum. By sequentially structuring online learning trials and mixed-quality demonstrations, CEC constructs curricula that encapsulate learning progression and proficiency increase across episodes. Such synergy combined with the potent pattern recognition capabilities of Transformer models delivers powerful cross-episodic attention mechanism. The effectiveness of CEC is demonstrated under two representative scenarios: one involving multi-task rein- forcement learning with discrete control, such in DeepMind Lab, where the curriculum captures the learning progression in both individual and progressively complex settings, and the other involving imitation learning with mixed-quality data for continuous control, as seen in RoboMimic, where the curriculum captures the improvement in demonstrators' expertise. In all instances, policies resulting from CEC exhibit superior performance and strong generalization. Code is open- sourced on the project website cec-agent, thub. io facilitate research on Transformer agent learning.
1 Introduction
The paradigm shift driven by foundation models [8] is revolutionizing the communities who study sequential decision-making problems [80), with innovations focusing control
2
45
38 []
planning 76 32 33, 78 [7), pre-trained visual representation [57] 50)
67
51 82
among others.
Despite the progress, the data-hungry nature makes the application of Transformer 77 agents extremely challenging in data-scarce domains like robotics
(52 53 19 38 9. This leads us to
the question: Can we maximize the utilization of limited data, regardless of their optimality and construction, to foster more efficient learning?
To this end, this paper introduces novel algorithm named Cross-Episodes Curriculum (CEC), method that explicitly harnesses the shifting distributions multiple experiences when organized into curriculum. The key insight is that sequential cross-episodic data manifest useful learning signals that do not easily appear in any separated training episodes [] As illustrated in Figure [ CBC realizes this through two stages: formulating curricular sequences capture (a) the policy improvement on single environments, (b) the learning progress on series of progressively harder environments, or (c) the increase of demonstrators' proficiency; and causally distilling policy improvements into the model weights of Transformer agents through cross-episodic attention. When policy is trained to predict actions at current time steps, it can trace back beyond ongoing trials and internalize improved behaviors encoded in curricular data, thereby achieving efficient learning
FFllowing the canonical definition Sutton Barto 73 we refer to sequences of agent-cnvironment interaction with clearly identified initial and terminal states as "episodes". We interchangeably use "episode", "trial", and "trajectory" in this work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

new Model Training with Cross-Episodic Attention
Curricular Data
Policy improvement single environments:
i
T
Context Window
T
Ongoing Episode
Learning progress series of progressively harder environments:
T
T
T
T
T
T
T
Increase of demonstrator's proficiency:
T
T
T
T
T
T
T
Figure Cross-episodic curriculum for Transformer agents. CEC involves two major steps: 1) Preparation of curricular data. We order multiple experiences such that they explicitly capture curricular patterns. For instance, they can be policy improvement in single environments, learning progress in a series of progressively harder environments, the increase of the demonstrator's expertise. 2) Model training with cross-episodis attention. When training the model to predict actions, it can trace back beyond the current episode and internalize the policy refinement for more efficient learning. Here each represents an episode (trajectory). à refers to actions predicted by the model. Colored triangles denote causal Transformer models.
and robust deployment when probed with visual or dynamics perturbations. Contrary prior works like Algorithm Distillation (AD, Laskin et al. [22 which, at test time, samples and retains a single task configuration across episodes for in-context refinement, our method, CEC, prioritizes zero-shot generalization across a distribution of test configurations. With CEC, agents are evaluated task configuration each episode, emphasizing adaptability to diverse tasks.
We investigate the effectiveness of CEC in enhancing sample efficiency and generalization with two representative case studies. They are: 1) Reinforcement Learning (RL) on DeepMind Lab (DM- Lab) (5), a 3D simulation encompassing visually diverse worlds, complicated environment dynamics, ego-centric pixel inputs, and joystick control; and Imitation Learning (IL. from mixed-quality human demonstrations on RoboMimic (53, framework designed to study robotic manipulation with proprioceptive and external camera observations and continuous control. Despite RL episodes being characterized by state-action-reward tuples and IL trajectories by state-action pairs, our method exclusively employs state-action pairs in its approach.
In challenging embodied navigation tasks, despite significant generalization gaps (Table [T. our method surpasses concurrent and competitive method Agentic Transformer (AT, Liu and Abbeel (47). It also significantly outperforms popular offline RL methods such as Decision Transformer (DT, Chen al. (13) and baselines trained expert data, with the same amount of parameters, architecture, and data size. It even exceeds RL oracles directly trained on test task distributions by 50% in zero-shot manner. CEC also yields robust embodied policies that are up 1.6x better than RL oracles when zero-shot probed with unseen environment dynamics. When learning continuous robotic control, CEC successfully solves two simulated manipulation tasks, matching Further ablation reveals that CBC with cross-episodic attention a generally effective recipe for learning Transformer agents, especially in applications where sequential data exhibit moderate and smooth progression.
and outperforming previous well-established baselines 5 25

---

2 Cross-Episodic Curriculum: Formalism and Implementations
In this section, we establish the foundation for cross-episodic curriculum method by first reviewing the preliminaries underlying our case studies, which encompass two representative scenarios in sequential decision-making Subsequently,
formally introduce the assembly of curricular data and the specifics of model optimization utilizing cross-episodic attention. Lastly, we delve into the practical implementation of CEC in the context of these two scenarios.
2.1 Preliminaries
Reinforcement learning. We consider the setting where source agents learn through trial and error in partially observable environments. Denoting states s E S and actions a E A, an agent interacts in a Partially Observable Markov Decision Process (POMDP) with the transition function p($ 88; at) x A + S. It observes E O emitted from observation function O and receives scalar reward from R(s, a) S
Under the episodic task setting, RL seeks to learn a parameterized policy xo(-|s) that maximizes the return over fixed length 7 of interaction steps: To arg maxoge Lt= 7'r,, where 7 [0, 1) is discount factor. Here we follow the canonical definition of an episode as a series of environment-agent interactions with length T. (so, do, ro..... ST-1, a7-1, ST-1,
(T-, where initial states so are sampled from initial state distribution 8q (s( and terminal states ST are reached once the clapsed timestep exceeds T. Additionally, we view all RL tasks considered in this work as goal-reaching prob- lems [39 26 and constrain all episodes to terminate upon task completion. It is worth noting that sim- ilar previous work [42,, training data are collected by source RL agents during their online learning. Nevertheless, once the dataset is obtained, our method is trained offfine in purely supervised manner.
Imitation learning. We consider IL settings with existing trajectories composed only of state-action pairs. Furthermore, we relax the assumption on demonstration optimality and allow them to be crowdsourced [10 12 [1]. Data collected by operators with varying expertise therefore unavoidable. Formally, we assume the access to a dataset DN := {π,..... -x consisting of N demonstrations, with each demonstrated trajectory т; (sp. ao
(s ................................ 8T-1: @T-1) naturally
identified as an episode. The goal of IL. specifically of behavior cloning (BC), is to learn a policy Ty that accurately models the distribution of behaviors. When viewed goal-reaching problems, BC policies can be evaluated by measuring the success ratio in completing tasks (26).
2.2 Curricular Data Assembly and Model Optimization
Meaningful learning signals emerge when multiple trajectories are organized and examined cross- episodically along a curriculum axis. This valuable information, which is not easily discernible in individual training episodes, may encompass aspects such as the improvement of an RL agent's navigation policy or the generally effective manipulation skills exhibited by operators with diverse proficiency levels. With powerful model architecture such as Transformer 16), such emergent and valuable learning signals can be baked into policy weights, thereby boosting performance in embodied tasks.
For given embodied task M, define its curriculum CM collection of trajectories T consisting of state-action pairs. A series of ordered levels [L1, Lz] partitions this collection such that Ulell..... 1) Li CM and (vt.je(L....L)..:)) [(.)) 0. More importantly, these ordered levels characterize a curriculum by encoding, for example, learning progress single environments, learning progress in a series of progressively harder environments, or the increase of the demonstrator's expertise.
With curriculum CM == { and its characteristics [C...., Ct), we construct a curricular sequence that spans multiple episodes and captures the essence of gradual improvement in the following way:
10)] where C~u ([I
and -(c) L..
IE (I.........)
The symbol @ denotes the concatenation operation. 4 ([KJ) denotes a uniform distribution over the discrete set (k E N, k < K). In practice we use values smaller than [.y] considering the memory consumption.

---

119
100
100
123
(a) Goal Maze
(b) Watermaze
Irreversible Path
(d) Lift
Can
Figure 2: We evaluate our method on five tasks that cover challenges such as exploration and planning over long horizons in RL settings, as well as object manipulation and continuous control in IL settings.
Figures are from Beattie et al. S and Mandlekar et al. [53
We subsequently learn causal policy that only depends on cross-episodic historical observations To ()))))) Note that this modeling strategy differs from previous work that views sequential decision-making
big sequence-modeling problem [13] 37 42 38). It instead resembles the
causal policy in Baker al. [4]. Nevertheless, we still follow the best practice [6, 60 22 to provide
previous action extra modality of observations in POMDP RL tasks.
We leverage the powerful attention mechanism of Transformer
75 enable cross-episodic atten- tion. Given observation series ((r) == {oo ..... o (S")) (shorthanded as O hereafter for brevity), Transformer projects it into query Q fQ(O), key K fx (O), and value V = (O) matrices, with each row being a D-dim vector. Attention operation performed to aggregate information:
QKT
Attention(Q. K, V) softmax
VD VV.
Depending on whether the input arguments for fo and f(K.V) are the same, attention operation can be further divided into self-attention and cross-attention. Since tasks considered in this work do not require additional conditioning for task specification, we follow previous work [3 to utilize self-attention to process observation series. Nevertheless, ours can be naturally extended to handle, for example, natural language or multi-modaa task prompts, following the cross-attention introduced in Jiang al. (38).
Finally, this Transformer policy is trained by simply minimizing the negative log fog-likelihood objective JNLL of labeled actions, conditioned on cross-episodic context:
T
T
TNLI = - log Ta (-IT)
Σ D1
log
(n)
Regarding the specific memory architecture, we follow Baker et al. [ to use Transformer-Xi [16] as our model backbone. Thus, during deployment, we keep hidden states propagating across test episodes to mimic the training settings.
4 Adaptive Agent Team et al.
2.3 Practical Implementations
We now discuss concrete instantiations of CEC for 1) RL with DMLab and 2) IL with RoboMimic. Detailed introductions to the benchmark and task selection are deferred to Sec. 3 We investigate the following three curricula, where the initial two pertain to RL, while the final one applies to IL:
cearning-progresst based curriculum. In the first instantiation, inspired by the literature on learning progress [54] 27 65 40), we view the progression of learning agents curriculum. Concretely, we train multi-tase PPO agents [70] [33 on tasks drawn from test distributions. We record their online interactions during training, which faithfully reflect the learning progress. Finally, we form the pearning-progress progress- based curriculum by sequentially concatenating episodes collected at different learning stages. Note that this procedure is different from Laskin et al. (4), where for each environment, the learning dynamics multiple single-task RL agents has to be logged. In contrast, we only track single multi-task agent per environment.
Task-difficulty-base curriculum. In the second instantiation, instead of taking snapshots of RL agents directly trained on test configurations, we collect learning progress series of easier

---

Table 1: Generalization gaps between training and testing for DMLab levels. Note that agents resulting from task bask-difficuly--bee based curricula are not trained on test configurations. Therefore, their performance should considered as zero-shot.
Level Name
Difficulty Purameter
Test Difficulty
Training Difficulty
Ours
Ours
BC
RL
Curriculum RI. (Oracle)
(Learning Progross)
(Task Difficalty)
Expert
(Orclee)
Goal Maze
Room Numbers
20 580
5-100-15
20
5-=10-15-22
Watermaze
Spawn Radius
580
150 300 :450
580
580
150
:300 1450 1580
Ineversible Path
Built-In Difficulty
3-33----7
but progressively harder tasks. For instance, in an embodied navigation task, the test configuration includes 20 rooms. Rather than logging source agents' learning progression in the 20-room maze, we record in a series of mazes with 5, 10, and 15 rooms. We then structure stored episodes first following learning progress and then the increase of layout complexity. This practice naturally creates task sosk-diffcutt--basee
curriculum, which resembles curriculum RL that is based on task difficulty [54] (88. We find it especially helpful for explloxplooatoon problems where the source RL agent does not make meaningful progress.
Expertise-based curriculum. For the setting of from mixed-quality demonstrations, we instantiate a curriculum based on demonstrators' expertise. This design choice is motivated by literature on learning from heterogeneous demonstrators 66 81], with the intuition that there little to learn from novices but lot from experts. To realize this idea, we leverage the Multi-Human dataset from RoboMimic (53). Since it contains demonstrations collected by human demonstrators with varying proficiency, we organize offline demonstration trajectories following the increase of expertise to construct the expertise-based curriculum.
Experimental Setup
In this section, elaborate the experimental setup of our studies. Our investigation spans two representative and distinct settings: 1) online reinforcement learning with 3D maze environments of DML La [5], and 2) imitation learning from mixed-quality human demonstrations of RoboMimic (53)
For each of them, we discuss task selection, baselines, and training and evaluation protocols. Teasers of these tasks are shown in Figure
3.1 Task Settings and Environments
DeepMind Lab (5] is a 3D learning environment with diverse tasks. Agents spawn in visually complex worlds, receive ego-centric (thus partially observable) RGB pixel inputs, and execute joystick actions. We consider three levels from this benchmark: Goal Maze, Watermaze (56). and Sky Maze with Irreversible Path. They challenge agents explore, memorize, and plan over a long horizon. Their goals are similar - to navigate in complicated mazes and find a randomly spawned goal, upon which sparse rewards will be released. Episodes start with randomly spawned agents and goals and terminate once goals are reached or elapsed steps have exceeded pre-defined horizons.
RoboMimic [53] is framework designed for studying robot manipulation and learning from demonstrations. Agents control robot arms with fixed bases, receive proprioceptive measurements and image observations from mounted cameras, and operate with continuous control. We evaluate two simulated tasks: "Lift" and "Can". In the "Lift" task, robots are tasked with picking up a small cube. In the "Can" task, robots required to pick up soda from a large bin and place into smaller target bin. Episodes start with randomly initialized object configuration and terminate upon successfully completing the task exceeding pre- pre-defined horizons.
3.2 Baselines
The primary goal of these case studies is to assess the effectiveness of our proposed cross-episodic curriculum in increasing the sample efficiency and boosting the generalization capability of Trans- former agents. Therefore, online RI. settings, we compare against source RI. agents which generate training data for our method and refer to them oracles. These include a) PPO agents directly
5

---

Goal Maze
Watermaze
Irreversible Path
80
60
Success Rate (%)
50
60
40
40
40
20
20
20
10
Ours
Ours
BC
(Task Difficulty), Auto
(Task Difficulty), Fixed
(Learning Progress)
w/ Expert Date
DT
DT
AT
(Mixed Difficulty)
(Single Difficultyl
(Mixed Difficulty)
(Single Difficulty)
Curriculum RL
(Oracle)
(Oracte)
Figure 3: Evaluation results on DMLab. Our CEC agents perform comparable to RL oracles and average outperform other baseline methods. On the hardest task Irreversible Path where the RL oracle and BC baseline completely fail, our agents outperform the curriculum RL oracle by 50% even in zero-shof manner. For methods, DTT, AT, and the BC w/ expert data baselines, independent evaluation runs, each consisting of 100 episodes for Goal Maze and Watermaze and 50 episodes for Irreversible Path due to longer episode length. We test RL oracles for 100 episodes. The error bars represent the standard deviations over 20 runs.
conduct 20
trained task distributions, denoted "RL (Oracle)" hereafter, and b) curriculum PPO agents that are gradually adapted from easier tasks to the test difficulty, which is referred to as "Curriculum RL (Oracle)". Furthermore, we compare against one concurrent and competitive method Agen- tic Transformer [47), denoted as "AT". It is closely related to our method, training Transformers on sequences of trajectory ascending sorted according to their rewards. We also compare against popular offline RL method Decision Transformer [3 denoted as "DT". Additionally, we include another behavior cloning agent that has the same model architecture as ours but trained on opti- mal data without cross-episodic attention. This baseline is denoted as "BC w/ Expert Data". For case study on from mixed-quality demonstrations, we adopt the most competing approach, BC-RNN, from Mandlekar et al. (53) as the main baseline. We also include comparisons against other offline RL methods 44 such as Satch-Constrained Q-learning (BCQ) (25 and Conservative
Q-Learning (CQL) 41.
3.3 Training and Evaluation
We follow the best practice train Transformer agents, including adopting AdamW optimizer learning rate warm-up and cosine annealing [48), etc. Training is performed on NVIDIA V100 GPUs. During evaluation, for agents resulting from our method, each run involves several test rollouts to fill the context. We keep hidden states of Transformer-XX [16] propagating across episodes. We run other baselines
49
oracles for 100 episodes to estimate their performances. For our methods on RL settings, we compute the maximum success rate averaged across sliding window over all test episodes to account for in-context improvement. The size of the sliding window equals one-quarter of the total test episodes. These values averaged over 20 runs to constitute the final reporting metric. For our methods on the IL setting, since all training data are successful trajectories, we follow Mandlekar et al. [53] report the maximum success rate achieved over the course of training, directly averaged over test episodes.
Experiments
We aim answer the following four research questions through comprehensive experiments.
1. To what extent former agents and boost their generalization capability?
cross-episodic curriculum increase the sample efficiency of Trans-
2. Is CEC consistently effective and generally applicable across distinct learning settings?
3. What are the major components that contribute the effectiveness of our method?

---

Ours (Task Difficulty)
Goal Maze (Unseen Mechanism)
ireversible Path (OOD. Difficulty)
Ours [Learning Progress)
DT (Mixed Difficulty)
Success Rate (%)
30
DT (Single Difficulty)
40
20
AT (Mixed Difficulty)
AT (Single Difficulty)
20
10
BC w/ Expert Date
RL (Oracle)
Curriculum (Oracle)
Goal Maze (Varying Env. Dyn.)
Watermaze (Varying Env. Dyn.)
Irveersible Path (Varying Env. Dyn.)
80
Success Rate (%)
30
40
10
0
Figure 4: Generalization results on DMLab. Top row: Evaluation results on Goal Maze with unseen maze mechanism and Irreversible Path with out-of-distribution difficulty levels. Bottom row: Evaluation results three levels with environment dynamies differing from training ones. CEC agents display robustness and generalization across various dimensions, outperforming curriculum RL oracles by up 1.6x. We follow the same evaluation protocol as in Figure The error bars represent the standard deviations over 20 runs.
4.1 Main Evaluations
We answer the first two questions above by comparing learned agents from method against 1) Reinforcement Learning (RI.) oracles in online RI. settings and 2) well-established baselines on learning from mixed-quality demonstrations in the Imitation Learning (II.) setting.
We first examine agents learned from learning-progres--oee challenging 3D maze environments. The first type of agent is denoted as "Ours (Learning Progress)", For the second type, to ensure that the evaluation also contains series of tasks with increasing difficulty, adopt two mechanisms that control the task sequencing [58]: 1) fixed sequencing where agents try each level of difficulty for a fixed amount of times regardless of their performance and 2) dynamic sequencing where agents automatically promoted to the next difficulty level if they consecutively succeed in the previous level for three times. We denote these two variants as "Ours (Task Difficulty), Fixed" and "Ours (Task Difficulty), Auto", respectively. Note that because the dilddiiffflt--bbeeec curriculum does not contain any training data on test configurations, these two settings are zero-shot evaluated test task distributions. We summarize these differences in Table We denote AT and DT trained on data consisting of mixture of task difficulties as "AT (Mixed Difficulty)" and "DT (Mixed Difficulty)". Note that these data are the same used train "Ours (Task Difficulty)". Similarly, we denote AT and DT directly trained on test difficulty as "AT (Single Difficulty)" and "DT (Single Difficulty)". These data are the same used train "Ours (Learning Progress)".
and task-difficulty-bor -based curricula in
Cross-episodic curriculum results in sample-efficient agents. As shown in Figure on two of three examined DMLab levels, CEC agents perform comparable RL oracles and outperform the BC baselines trained on expert data by at most 2.8x. On the hardest level Irreversible Path where agents have plan the route ahead and cannot backtrack, both the BC baseline and RL oracle fail. However, our agents succeed in proposing correct paths that lead to goals and significantly outperform the curriculum RL oracle by 50% even in cero-shot manner. Because CEC only requires environment interactions generated during the course of training of online source agents (the Lii--difficulty-basee eurriculum even contains fewer samples compared to the curriculum RL, as illustrated in Table [T. the comparable and even better performance demonstrates that our method yields highly sample-efficin embodied policies. On average, our method with fask-diffiully-baase curriculum performs the best during evaluation (Table A.5. confirming the benefit over the

---

Table Evaluation results on RoboMimic. Visuomotor policies trained with our expertise-based curriculum outperform the most competing cestory-depender behavior cloning baseline, as well as other offline RL algorithms. For our method on the Lift task, we conduct 5 independent runs each with 10 rollout episodes. On the Can task, we conduct 10 independent runs each with 5 rollout episodes due to the longer horizon required to complete the task. Standard deviations are included.
Task
Ours
BC-RNN 53
BCQ 25
CQL
Lift
100.0 + 0.0
100.0 0.0
93.3 = 0.9
11.3 = 9.3
Can
100.0 + 0.0
96.0 + 1.6
77.3 = 6.8
0.0 = 0.0
Table 3: Ablation on the importance of cross-episodic attentionn Transformer agents trained with the same curricular data but without cross-episodic attention degrade significantly during evaluation, suggesting its indispensable role in learning highly performant policies.
DMLab
RoboMimic
Goal Maze
Watermaze
Inreveriible Path
Lift
Can
Ours
65 2 + 6.7
50.9 + 6.6
38.2 + 7.0
100.0 + 0.0
100.0 + 0.0 99.3 1 0.9
Ours w/o Cross Cross-Episodic Attention
35.0 7.1
20.0 1 2.5
3.8 + 4.9
75.9 + 12.3
concurrent AT approach that leverages outperforms by significant margin, which suggests that our cross-episodic curriculum helps to squeeze learning signals that are useful for downstream decision-making.
experiences. When compared DT, it
Cross-episodic curriculum boosts the generalization capability. To further investigate whether CEC can improve generalization at test time, we construct settings with unseen maze mechanisms (randomly open/closed doors), out-of-distributio difficulty, and different environment dynamics. See the Appendix, Sec. C.2 Sec. for the exact setups. As demonstrated in Figure 4 CEC generally improves Transformer agents in learning robust policies that can generalize perturbations across various axes. On three settings where the BC w/ Expert Data baseline still manages to make progress, CEC agents are up 2x better. Compared outperform them under three out of five examined scenarios. It is notable that on Irreversible Path with out-o--dittribution difficulty, CEC agent 1.6x better than the curriculum RL oracle trained on the same data. These results highlight the benefit of learning with curricular contexts. On average, our method surpasses the concurrent AT baseline and achieves significantly better performance than other baselines (Table A.6). This empirically suggests that CEC helps to learn policies that are robust to environmental perturbations and can quickly generalize to new changes.
oracle curriculum RI. agents, our policies significantly
Cross-episodic curriculum is effective across a wide variety of learning scenarios. We now move beyond RL settings and study the effectiveness of the expertise-based curriculum in the IL. setting with mixed -ualityy demonstrations. This is a common scenario, especially in robotics, where demonstrations are collected by human operators with varying proficiency [52]. As presented in Table 2 visuomotor policies trained with the expertise-based curriculum are able to match and outperform the well-established baseline [53] on two simulated robotic manipulation tasks and achieve significantly better performance than agents learned from prevalent offline RL algorithms [25] 41) These results suggest that our cross-episodic curriculum is effective and broadly applicable across various problem settings, More importantly. it provides a promising approach to utilizing limited but sub-optimal data in data-scarce regimes such as robot learning.
4.2 Ablation Studies
In this section, we seek to answer the third research question to identify the components critical to the effectiveness of our approach. We focus on three parts: the importance of cross-episodic attention, the influence of curriculum granularity, and the effect of varying context length. Finally, we delve into the fourth question, identifying scenarios where CEC is expected to be helpful.
Importance of cross-episodic attention. The underlying hypothesis behind our method is that cross-episodic attention enables Transformer agents to distill policy improvement when mixed-optimality trajectories are viewed collectively. To test this, on DMLab levels and RoboMimic tasks, train the same Transformer agents with the same curricular data and training epochs but
8

---

without cross-episodic attention. We denote such agents as "Ours w/o Cross-Episodic Attention" in 3 Results demonstrate that the ablated variants experience dramatic performance degradation on four out of five examined tasks, which suggests that naively behaviorally cloning sub-optima data can be problematic and detrimental. Cross-episodic attention views curricular data collectively, facilitating the extraction of knowledge and patterns crucial for refining decision-making, thereby optimizing the use sub-optimal data.
Curriculum granularity. We perform this ablation
Relative Performance (%)
with the dask-diffieulty-busee
curriculum DMLab
100
levels, due to the ease of adjusting granularity. We treat the curricula listed in the column "Ours (Task Difficulty)" in Table I as "Fine", and gradually make them coarser study the impact. Note that we ensure the same amount of training data, See the Appendix, Sec. f.r how we define granularity levels "Medium" and "Coarse". We visualize the performance relative to the most fine-grained Figure 5 The monotonic degradation of policy perfor-
90
85
Goal
Watermaze
Ireversibbe Path
Hine
Medium Coarse
mance with respect to curriculum coarseness suggests that fine-grate fine-grained curricula are critical for Transformer agents to mostly benefit from cross-episodic training.
Curriculum Granularity
Figure 5: We compare the performance relative agents trained with the fine- grained curricula. Performance mono- tonically degrades as task-difficulty- based curricula become coarser.
Varying context length. Lastly, we study the effect of varying context length on DMLab and visualize in Figure 6 We normalize all performance values relative to those of "Ours (Task Difficulty), Auto" reported in Figure and unnecessarily long context windows are harmful. On two out of three levels, using shorter context decreases the performance even more. This finding coincides with Laskin et al. 42 that a sufficiently long Transformer context is necessary to retain cross- cross-episodic information. Furthermore, we also discover that an unnecessarily long context also harmful. We hypothesize that this is due the consequent training and optimization instability.
3 turns out that both too short
Curriculum selection based on task complexities and data sources. For RL tasks, we recommend starting with curriculum. However, if the task itself is too challenging, such that source algorithms barely make progress, we recommend the task-difficulty- based curriculum. In IL settings, we further investigate the performance of the learning-progress- based curriculum RoboMimic tasks considered in this work. Detailed setup and results are included in Appendix, Sec C To summa- rize, if human demonstrations are available, even if they recom-
Relative Performance (%)
300
500
1600
400
800
the saamiing-roggess-bbeee
100
60
40
20
generated to be heterogeneous in quality, mend using the expertise- based curriculum. However, in the absence of human demonstrations and only with access to machine-generated data (e.g., generated by RL agents), learning-progrest pet-orrrsoosseeeeeeeee based curriculum is recommended because it achieves non-trive con-trivial performance and signifi- cantly outperforms offline RL methods such as CQL [41).
Goal Maze Watermaze Irrevers. Path
Figure 6: Both short and unnecessarily long context windows decrease the per- formance. Numbers in the legend de- note context lengths. Performance val- ues are relative to those of "Ours (Task Difficulty). Auto" reported in Figure 3 "Irrevers. Path" stands for the task "Irre- versible Path".
Related Work
Sequential decision-make making with Transformer agents. There are many ongoing efforts to repli- cate the strong emergent properties demonstrated by Transformer models for sequential decision- making problems [8). Decision Transformer [33] and Trajectory Transformer [37] pioneered this thread by casting offline RL 44 as sequence modeling problems. Gato [68] learns massively multi- task agent that can prompted to complete embodied tasks. MineDojo [2] and VPT 4 utilize numerous YouTube videos for large-scale pre-training in the video game Minecraft. VIMA [38 and RT-1 [9 build Transformer agents trained scale for robotic manipulation tasks. BeT 7 and C-BeT 14 design novel techniques to learn from demonstrations with multiple modes with Trans-

---

formers. Our causal policy most resembles to VPT 4 But we focus on designing learning techniques that are generally effective across a wide spectrum of learning scenarios and application domains.
Cross-episodic learning. Cross-episodic learning is less-explored terrain despite that it has been discussed together with meta-RL [77] for long time. RL (18) uses recurrent neural networks for online meta-RL by optimizing multi-episodic value functions. Meta-Q-learning [2] instead learns multi-episodic value functions in an offline manner. Algorithm Distillation (AD) [42] and
Adaptive Agent (AdA) [ are two recent, inspiring methods in cross-episodic learning. Though at first glance our caamiing-proggess. based curriculum appears similar to AD, significant differences emerge. Unlike AD, which focuses on in-context improvements at test time and requires numerous single-task source agents for data generation, our approach improves data efficiency for Transformer agents by structuring data in curricula, requiring only single multi-task agent and allowing for diverse task instances during evaluations. Meanwhile, AdA, although using cross-episodic attention contrast,
with a Transformer backbone, is rooted online RL within a proprictary environment. we focus on offline behavior cloning in accessible, open-source environments, also extending to IL scenarios unexplored by other meta-learning techniques. Complementary to this, another recent study 43 provides theoretical insight into cross-ppisodic learning.
Curriculum learning. Curriculum learning represents training strategies that organize learning samples meaningful orders to facilitate learning 7 has been proven effective in numerous works that adaptively select simpler task (58] 74, 69
62
15 55
59
46)
auxiliary rewards (35 72]. Tasks
are also parameterized to form curricula by manipulating goals [24 30 [66, environment layouts (79
3 644, and reward functions 28 34 Inspired by this paradigm, our work harnesses the improving
nature of sequential experiences to boost learning efficiency and generalization for embodied tasks.
6 Conclusion
In this work, we introduce a new learning algorithm named Cross- Criss-Episodic Curriculum to enhance the sample efficiency of policy learning and generalization capability of Transformer agents. It leverages the shifting distributions of past learning experiences or human demonstrations when they viewed as curricula. Combined with cross-episodic attention, CEC yields embodied policies that attain high performance and robust generalization across distinct and representative RL and IL settings. CEC represents solid step toward sample-efficient policy learning and is promising for data-scarce problems and real-world domains.
Limitations and future work. The CEC algorithm relies the accurate formulation of curricular sequences that capture the improving nature of multiple experiences. However, defining these sequences accurately can be challenging, especially when dealing with complex environments or tasks. Incorrect or suboptimal formulations of these sequences could negatively impact the algorithm's effectiveness and the overall learning efficiency of the agents. A thorough exploration regarding the attainability of curricular data is elaborated upon in Appendix, Sec
In subsequent research, the applicability of CEC real-world tasks, especially where task difficulty remains ambiguous, merits investigation. A deeper assessment of a demonstrator 's proficiency trajectory from initial unfamiliarity to the establishment of muscle memory could offer valuable learning signal. Moreover, integrating real-time human feedback to dynamically adjust the curriculum poses intriguing challenge, potentially enabling CEC to efficiently operate in extended contexts, multi-agent environments, and tangible real-world tasks.
Acknowledgments and Disclosure of Funding
We thank Guanzhi Wang and Annie Xie for helpful discussions. We are grateful Yifeng Zhu, Zhenyu Jiang, Soroush Nasiriany, Huihan Liu, and Rutav Shah for constructive feedback on an early draft of this paper. We also thank the anonymous reviewers for offering us insightful suggestions and kind encouragement during the review period. This work was partially supported by research funds from Salesforce and JP Morgan.
10

---

References
[1] Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmice Michael Chang, Natalie Clay, Adrian Collister, Vib- havari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks- Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl. and Lei Zhang. duman-timeseen adaptation an open-ended task space. arXiv preprint arXiv: Arxiv- 200.-3301.0608, 2023.
[2] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz. Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huee Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv: Arxiv-2204.016%% 016601 2022.
[3] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations, 2020.
[4] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv: Arxiv-2206. 1795, 2022.
[5] Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney. Helen King. Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. arXiv preprint arXiv: Arxiv-1612.03801,, 2016.
[6] M. Beliaev, Andy Shih, S. Ermon, Dorsa Sadigh, and Ramtin Pedarsani. Imitation learning by estimating expertise of demonstrators, International Conference On Machine Learning, 2022.
[7] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning, 2009.
[8] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen I.,, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nic, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou. Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy I Lingg. On the opportunities and risks of foundation models. arxiv preprint arxiv:
2021.

---

[9] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan. Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofrr Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong. Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv: Arxiv-2212.066 2022.
[10] Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum, Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. arXiv preprint arXiv: 1rxiv-1904.06338 2019.
[11] Zhangjie Cao and Dorsa Sadigh. Learning from imperfect demonstrations from agents with varying dynamics. arXiv preprint arXiv: Arxiv-2/03.05910 2021.
[12] Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. arXiv preprint arXiv: Arxiv-2010. /723, 2020.
[13] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Rein- forcement learning via sequence modeling. In Marc' Aurelio Ranzato, Alina Beygelz- imer, Yann N. Dauphin, Percy Liang. and Jennifer Wortman Vaughan, editors, Ad- vances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 15094-15097, 2021. URL /tRp
nwwr httppp/cc/pppp///211//////// Abstract.
[14) Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data, arXiv preprint arXiv: Arxiv-2210.100 2022.
[15] Wojciech Czarnecki, Siddhant M. Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Nicolas Manfred Otto Heess. Simon Osindero, and Razvan Pascanu. Mix&match agent curricula for reinforcement learning. In International Conference on Machine Learning, 2018.
[16] Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: Arxiv-1901.02860, 2019.
[17] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. arXiv preprint arXiv: Arxiv-2303.03377, 2023.
[18] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI²: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: Arxiv- 161 1.02770, 2016
[19] Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danillidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv: Arxiv-2109, 1oxi=22109./3396 /3396, 2021.
[20] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukeuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv: Arxiv-1802.01561. 2018
[21] Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J. Smola. Meta-q-learning. arXiv preprint arXiv: Arxiv 2019.
12

---

[22] Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv: Arxiv-2006.08853 2022.
[23] Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. arXiv preprint arXiv: Arxiv-1509.06113 2015.
[24] Sébastien Forestier, Yoan Mollard, and Pierre Yves Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. ArXiv, abs/1708.02190, 2017.
[25] Scott Fujimoto, David Meger, and Doina Precup. Off-ppolicy deep reinforcement learning without exploration. arXiv preprint arXiv: Arxiv- (at0--88//.0200,, 2018.
[26] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning, arXiv preprint arXiv: Arxiv-1912.0660 2019.
[27] Alex Graves, Marc G. Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. arXiv preprint arXiv: Arxiv-1704.03000 030000 2017.
[28] Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-learning for reinforcement learning. ArXiv, abs/ 180..0666044 2018.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015. URL htttp//aaxivv org/abs//////eeeoo /512. 1512.03385 arXiv:1512.0338; [cs].
[30] David Held, Xinyang Geng, Carlos Florensa, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning. 2018.
[31] Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momehev, Danila Sinopalnikov, Piotr Stanezyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Vieillard, Seyed Kamyar Seyed Ghasemipour, Sertan Girgin, Olivier Pietquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang. Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez Colmenarejo. Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Aeme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979 2020. URL https: btttps:///rxxiee http//awxvv..o........./ abs 00979
Manu Orsini, Alexis Jacq, Johan Ferret, Nino
[32] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero- shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 911899111111111 1918-9977 PMLR, 2022. https:/ http::///oocceeen...mmm mlr prwss/rsssseeee http:/vv22/hnne w1w.//hang222aeee html
[33] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv: Arxiv- 2207.05608, 2022.
[34] Allan Jabri, Kyle Hsu, Ben Eysenbach, Abhishek Gupta, Alexei A. Efros, Sergey Levine, and Chelsea Finn. Unsupervised curricula for visual meta-reinforcemeet learning, In Advances in Neural Information Processing Systems, 2019.
[35] Max Jaderberg, Volodymyr Mnih, Wojeiech Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference Learning Representations, 2017.
13

---

[36] Max Jaderberg, Wojeiech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Gar- cia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel 7. Leibo, David Silver, Demis Hass- abis, Koray Kavukeuoglu, and Thore Graepel. Human-level performance in 3d multiplayer games with population- based reinforcement learning. Science, 3644444))4444448888888.55 2019. doi: http://ciiiiee..ogoo////aa/a////
10.1 (0.1126/scieccc.aau224), URL
aau6249
[37] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Mare' Aurelio Ranzato. Alina Beygelz- imer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Ad- vances Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 1273-1286, 2021. URL
099f6600444233664a550000 34600/444688003 2o0 wwwtratt.ttttttaammaaaaa html
[38] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv: Arxiv-2210.03094 2022.
[39] Leslie Pack Kaelbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence, 1993.
[40] Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton, Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, Oleg Klimov, and Jeff Clune. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint arXiv: Arxiv-2106. 14876, 2021.
[41] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv: 20070-2006604779,, 2020.
[42] Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steiger- wald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, maxime gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. In The Eleventh International Conference on Learning Representations, 2023. htttps:///ppnnttt
net/ fest/foorumiiddie fot.forrmmdddbya5MMPPP
[43] Jonathan N. Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. arXiv preprint arXiv: Arxiv-2306. 14892, 2023.
[44] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv: makiv-0005.0164. 2020.
[45] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv: Arxiv-2209.07553, 2022.
[46] Xingyu Lin, Harjatin Singh Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. In Advances in Neural Information Processing Systems, 2019.
[47] Hao Liu and Pieter Abbeel. Emergent agentic transformer from chain of hindsight experience. arXiv preprint arXiv: Arxiv-2305. 16554, 2023.
[48] Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. n net//orrumPdd-Sx488E.xx
[49] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICI. 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httpppppw..ppppooooos/ee/e//e/ea
14

---

[50] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training, arXiv preprint arXiv: Arxiv-22 200002200000030, 2022.
[51] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv: 2oxi-23333188440 2023.
[52] Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, and Li Fei-Fei. Roboturk: A crowdsoureing platform for robotic skill learning through imitation. arXiv preprint arXiv: Arxiv-1811.02790. 2018.
[53] Ajay Mandlckar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martín. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv: Arxiv- 2 2108.03298, 2021.
[54] Tambet Matissnn, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. arXiv preprint arXiv: Arxiv-1707.00/83 2017.
[55] Tambet Matüsen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. In IEEE transactions on neural networks and learning systems, 2019.
[56] Richard G.M. Morris. Spatial localization does not require the presence of local cues. Learning and Motivation, 12(2):239-20) 260, 1981. doi: 10. 1101//00239990000 (19000005. URL https //app. cempp.dimessiooo ons a att///taaail//ppiicccttttooo///
1028012961
[57] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv: vxxi-2220311261. 2022.
[58] S. Narvekar, J. Sinapov, and P. Stone. Autonomous task sequencing for customized curriculum design in reinforcement learning. In International Joint Conference on Artificial Intelligence, 2017.
[59] Sanmit Narvekar and Peter Stone. Learning curriculum policies for reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
[60] OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Jézefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique d. O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang, Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: Arxiv-1912 192.06001,.0 2019.
[61] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[62] B. Peng. J. MacGlashan, R. Loftin, M. Littman, D. Roberts, and Matthew E. Taylor. Curriculum design for machine learners in sequential decision tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 2:268 2776687777770 2018.
[63] Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, and Vladlen Koltun. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning, arXiv preprint arXiv: Arxiv-2006. 1/050, 2020.
15

---

[64] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yves Oudeyer. Teacher algorithms for curriculum learning of deep in continuously parameterized environments. Conference on Robot Learning. 2019.
[65] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yves Oudeyer. Teacher algorithms for curriculum learning of deep continuously parameterized environments. arXiv preprint arXiv: Arxiv-1910.07224 2019.
[66] Sébastien Racanière, Andrew Kyle Lampinen, Adam Santoro, David P. Reichent, Vlad Firoiu, and Timothy P. Lillicrap. Automated curriculum generation through setter-solver interactions. In International Conference on Learning Representations, 2020.
[67] Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. arXiv preprint arXiv: Arxiv- 2210.03109, 2022.
[68] Scott Reed, Konrad Zoina, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez. Yury Sulsky, Jackic Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. arXiv preprint arXiv: Arxiv-2205.06/ 2022.
[69] Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing-solving sparse reward tasks from scratch. In International Conference Machine Learning. 2018.
[70] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 7rxiv-1707.06344 2017.
[71] Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lemrel Pinto. Be- havior transformers: Cloning k modes with one stone. arXiv preprint arXiv: 1xxiv-2206.110 11251, 2022.
[72] William B Shen, Danfei Xu, Yuke Zhu, Leonidas Guibas, Li Fei-Fei, and Silvio Savarese. Situational fusion of visual representation for visual navigation. In Proceedings of the IEEE International Conference on Computer Vision, pages 2881-2890, 2019.
[73] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[74] M. Svetlik, Matteo Leonetti, J. Sinapov, Rishi Shah, Nick Walker, and P. Stone. Automatic curriculum graph generation for reinforcement learning agents. InAAAI Conference on Artificial Intelligence, 2017.
[75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, arXiv preprint arXiv: Arxiv- 1706.03766 2017.
[76] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305.16291 2023.
[77] Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: 1xxi0-1611.05763 2016.
[78] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv: Arxiv-2302.0756/ 2023.
[79] Jan Wöhlke, Felix Schmitt, and Herke van Hoof. A performance-based start state curriculum framework for reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 1503-1511, 2020.
16

---

[80] Sherry Yang, Ofrr Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Founda- tion models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv: Arxiv-2303.04 2023.
[81] Songyuan Zhang, ZHANGJIE CAO, Dorsa Sadigh, and Yanan Sui. Confidence-aware im- itation learning from demonstrations with varying optimality. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neu- ral Information Processing Systems, volume 34, pages 12340-12350. Curran Associates, Inc., 2021. URL http::////prrceeeingg.. neurips sfoeaatt44400llaaaaccr7reeeeeaa3
siaape_ffilespppprr////// pdf
[82] Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta, Dinesh Jayaraman, Yecheng Ja- Ma, and Luca Weihs. Universal visual decomposer: Long-horizon manipulation made easy, 2023.
[83] Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision- based manipulation with object proposal priors. arXiv preprint arXiv: Arxiv-2210. 1339, 2022.
17

---

A Model Architecture
In this section, we provide comprehensive details about the Transformer model architectures con- sidered in this work. We implement all models in Py Torch [61] and adapt the implementation of Transformer- XL from VPT
A.1 Observation Encoding
Experiments conducted on both DMLab and RoboMimic include RGB image observations. For models trained DMLab, use ConvNet [29] similar to the used in Espeholt et al. [20). For models trained on Robo.imic, we follow Mandlekar et al. [53] to use a ResNet-18 network [29]
followed by spatial- spatial-softmax layer [23]. We use independent and separate encoders for images taken from the wrist camera and frontal camera. Detailed model parameters are listed in Table A.1
Table A.I: Model hyperparameters for vision encoders.
Hy Hyperparameter
Value
DMLab
Image Size
72 x 96
Number of ConvNet Blocks
1
Channels per Block
[16, 32, 32] 256
Output Size
RoboMimic
Image Size
84 x 84
Random Crop Height
76
Random Crop Width
76
Number of Randomly Cropped Patches
1
ConvNet Backbone
ResNet-18 29
Output Size
64
Spatial-Soffie Softmax Number of Keypoints
32
-ootial-Softmax Temperature
1.0
Output Size
64
Since DMLab is highly partially observable, we follow previous work 20 to supply the model with previous action input. We learn 16-dim embedding vectors for all discrete actions.
To encode proprioceptive measurement RoboMimic, we follow Mandlekar al. (53] apply any learned encoding. Instead, these types of observation are concatenated with image features and passed altogether the following layers. Note that we do provide previous action inputs in RoboMimic, since we find doing so would incur significant overfitting.
A.2 Transformer Backbone
We use Transformer-Xi 16] our model backbone, adapted from Baker et al. 4 Transformer-X splits long sequences into shorter sub-sequences that reduce the computational cost attention while allowing the hidden states to be carried across the entire input by attending to previous keys and values. This feature critical for the long sequence inputs necessary for cross-episodic attention. Detailed model parameters are listed in Table A.2
A.3 Action Decoding
To decode joystick actions in DMLab tasks, we learn a 3-layer MLP whose output directly parameter- izes a categorical distribution. This action head has hidden dimension 128 with ReLU activations. The "Goal Maze" and "Irreversible Path" tasks have an action dimension 7f while "Watermaze" has 15 actions. decode continuous actions in RoboMimic, we learn 2- MI. that parameterizes Gaussian Mixture Model (GMM) with 5 modes that generates a 7-dimensional action. This network
18

---

Table A.2: Model hyperparameters for Transformer-Xe XL.
Hyperparameter
Value (DMLab)
Value (RoboMimic)
Hidden Size
256
400
Number of Layers
4
2
Number of Heads
8
Pointwise Ratio 4
4
has a hidden dimension of 400 with ReLU activations, During deployment, we employ the "low-noise evaluation" trick [3]]
Training Details and Hyperparameters
All experiments are conducted on cluster nodes with NVIDIA V100 GPUs. We utilize DDP (dis- tributed data parallel) to accelerate the training if necessary. Training hyperparameters are listed in Table A.3
Table A.3: Hyperparameters used during training.
Hyperparameter
Value (DMLab)
Value (RoboMimic)
Learning Rate
0.0005
0.0001
Warmup Steps
1000
0
LR Cosine Annealing Steps
100000
N/A
Weight Decay
0.0
0.0
C Experiment Details
C.1 DMLab Main Experiment
Our DMLab main experiment is conducted on three levels with task IDs
explore_goal www.ggollllcaioonss_1__ laarge,
rooms watermaze,
and skymaze irreversible path hard.
We use no action repeats during training and evaluation. For experiments with varying task difficulty, select difficulty parameters "room numbers". "spawn radius", and "built-in difficulty" for these three levels, respectively. We adopt environment wrappers and helper functions from Petrenko et al. to fiexibly and precisely maneuver task difficulties.
63
Due to different task horizons, we tune the context length of Transformer-X models and vary curricular trajectories accordingly. These differences are summarized in Table/A.4
RL oracles serve as source agents used generate training data for our methods and the "BC w/ Expert Data" baseline. They are trained with the PPO (70) implementation from Petrenko et al. [63] The "BC w/ Expert Data" baselines have the same model architecture, training hyperparameters, and amount of training data our method, but are trained solely on trajectories generated by the best performing RL oracles without cross-episodic attention.
C.2 DMLab Generalization
This series of experiments probe the zero-shot generalization capabilities of embodied agents in unseen maze configurations, out-of-distribution difficulty levels, and varying environment dynamics. For the task "Goal Maze w/ Unseen Mechanism". we the level with task ID
19

---

Table A.4: Experiment details on DMLab tasks. Columns "Epoch" denote the exact training epochs with best validation performance. We select these checkpoints for evaluation. For task -ask-diffeultt-- based curriculum, the column "Training Trajectories" with n x m entries means n trajectories per difficulty level (m levels in total). The column "Sampled Episodes" with [i,j] entries means we first determine the number of episodes per difficulty level by uniformly sampling an integer from [. (inclusively).
Task-Difficnlyy Curriculum
Learning. Progress Curriculum
Name
Length
Epoch
Training Trajectonis
Sampled Episedes
Epoch
Training Trajectories Sampled Episodes
Goal Mazo
500
84
x00
300
Watermaze
400
89
x
[1.5] 5]
80
300
[1.3]
Table A.5: Evaluation results on DMLab, averaged over three tasks (Figure
Ours (Task Difficulty,, Auto
Ours (Task Difficulty,. Fixed
Ours (1 flearning Progress)
DT (Mised Difficulty)
(Single Difficulty)
(Mxod Ditiiultt)
(Single Dithicalt)
Export Data
Curriculum
(Oracle)
(Orucle)
51.4
54.4
32.4
35.3
11.7
42.7
33.4
14.2
40.6
30.6
Table A.6: Generalization results on DMLab, averaged over five settings (Figure
Ours (Task
Ours (Learning
DTT (Mixed Difficulty)
DT (Single Difficulty)
AT (Mixed Difficulty)
AT (Single Difficulty)
BC w/ Expert
RL (Oracle)
Curriculum RL.
Difficltty)
Progress)
Data
(Oracle)
39.6
27.8
31.8
13.6
39.4
29.2
18.1
30,0
37.6
explore.obstructer www.trocteddddooo.oo stt.ruuctedggglllss. .large, which adds randomly opened and closed doors into the maze while ensuring valid path to the goal always exists. An example of an agent's ego-centric observation visualized in Figure I.I
The task "Irreversible Path (OOD. Difficulty)" corresponds to configurations with the built-in difficulty of 1 (agents are only trained on difficulty up to 0.9, as noted in Table IT. For tasks with varying environment dynamics, directly test agents with an action repeat of 2. This is different from the training setting with no action repeat.
C.3 RoboMimic Main Experiment
We leverage the Multi-Human (MH) dataset from Mandlekar et al. [53]. It consists of demonstrations collected by operators with varying proficiency. We construct the expertise-base based curriculum by following the order of "worse operators, okay operators, then better operators". We length 200 for both tasks. There are 90 trajectories per expertise level. To determine the number of trajectories per expertise level when constructing curricular data, uniformly sample an integer from [1, 5] (fnclusively). The "Lift" and "Can" tasks are solved after training for 33 epochs and 179 epochs, respectively. We control for the same number of training epochs in subsequent ablation studies.
context
C.4 Ablation Study on Curriculum Granularity
We perform this ablation with the task-difficulty-base curriculum on DMLab levels due to the ease of adjusting granularity. The definition of varying levels of curriculum coarseness is listed in Table A.7
Table A.7: Definitions of varying levels of curriculum coarseness.
Level Name
Difficulty Parameter
Test Difficulty
Fine
Medium
Coarse
Goal Maze
Room Numbers
20
5-110=15
5-+10
5-+15
Watermaze
Spawn Radius
580
150-3300-4400
150->300
150-+550
Irreversible Path
Built-In Difficulty
0.9
.1-.33x.5-7.7
.1-4.5-.7
.1 .1-333-.5
20

---

Figure A.I: A visualization of the task "Goal Maze (Unseen Mechanism)". includes doors that are randomly opened or closed.
Table A.8: Results show the performance of different curricula on two robotic manipulation tasks: Lift and Can. Standard deviations included.
Task
Bassed Curriculum Bearning-Progresss
Curriculum
CQL 41
Lift
100.0 + 0.0
32.0 + 17.0
2.7 + 0.9
Can
100.0 + 0.0
30.0 2.8
0.0 + 0.0
Average
100.0
31.0
1.4
C.5
Comparison of Curricula in RoboMinicc
In IL settings, we further explored the efficacy of various curricula. For the RoboMimic tasks examined, we employed peaming--rogresss-ee based curriculum, ensuring the total training trajectories matched those of the expertise- ppaeed curriculum (i.e., 270 trajectories per task). All other parameters remained consistent, with the training data derived from RoboMimic's 's machine-generate
dataset.
Table A.8 indicates that when heterogeneoos--uallte human demonstrations are accessible, the sepertise-base curriculum is preferable due to its superior performance over the dearning-progrest based approach. Conversely, without expert demonstrations and relying solely on machine-generalee data, the soaming-progress based curriculum is still commendable. It offers noteworthy results and surpasses offline RL methods like CQL [1], even though CQL is trained on the full RoboMimic dataset, encompassing 1500 trajectories for the Lift task and 3900 for the Can task.
D Feasibility of Obtaining Curricular Data
The challenge of accurately orchestrating a curriculum is non non-trivia trivial and hinges on various factors. In the present work, three curriculum designs are introduced and validated, each with its practical considerations and underlying assumptions, discussed herein.
Peerning-Progres--aae Based Curriculum. RL agents typically exhibit monotonic improvement over training epochs, thereby naturally producing incrementally better data. The curriculum here is devised through a series of checkpoints throughout the training duration, necessitating no supplementary assumptions for its formulation.
Task Bask-Difficulty-BBe Curriculum. In contexts where environmental difficulty is parameterizable, curricula can be structured through schedule, determined by the relevant difficulty parameter, as demonstrated within this work. In scenarios lacking parameterized difficulty, alternatives such as methods proposed by Kanitscheider et al. [40] may be employed. The application of our method to tasks where difficulty is not explicitly characterized presents an intriguing avenue for future research.
Expertise- Curriculum. A notable limitation resides in the requisite to estimate demon- strators' proficiency. While some IL benchmarks, e.g., RoboMimic [53), come pre-equipped with proficiency labels, a broader application of our method necessitates an approximation of proficiency. One plausible approach entails ranking trajectories via completion time. Furthermore.
21

---

demonstrator's proficiency is likely to organically improve from initial unfamiliarity with teleoperation systems or tasks, stage of executing data collection with muscle memory [52]
-fom
This progression potentially provides rich learning signal conducive for CEC application.
E
Broader Impact
Our Cross-Episodic Curriculum can significantly enhance Transformer agent learning but carries potential socictal impacts. The efficiency of our method depends on the curriculum's 's design. If the curriculum unintentionally reflects biases, could lead to the amplification of these biases in learned policies, potentially perpetuating unfair discriminatory outcomes in AI-driven decisions. Furthermore, the computational intensity of our approach at evaluation could contribute to increased energy usage, which has implications for the environmental footprint of Al applications.
22


Reviews:
[{"review_id": "kcRiFnf72D", "reviewer": "Reviewer_xTZL", "summary": "This work aims to study mechanisms of cross-episode attention to effectively learn to improve polices by training on contexts containing gradually improving trajectories.", "questions": "How does your RL approach in the DM labs differ from the general approach from Adaptive Agents Team et al?", "limitations": "The paper could include more discussion of the feasibility of getting data with the requisite ordering of poor data first, better data next, best data last.  It could also be useful for the paper to include some mention of the limitations of the imitation paradigm.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "On the whole, this paper is well written.  The topic of transformers in-context learning as an approach to planing and RL is of increasing importance. This work provides another important datapoint in that space for a technique which has scientific value to try.", "weaknesses": "The main weakness with this work is its lack of comparison to existing approaches in the space.  The empirical results are only compared against approaches that are not intending to take advantage of in-context learning.  This looses most of the scientific value of the work when several other prominent works have used the in-context learning capabilities of transformers in various ways to great success. It is known that in-context learning can help with these sorts of tasks, the question which this paper is in the position to answer, is if this way of applying in-context learning works better or worse than the others which have been tried. Even if this approach is importantly distinct from those, it is important to compare against the other natural approaches that have already been tried, such as Algorithmic Distillation or AdA, to understand if this approach to applying in-context learning works better or worse than other approaches.\n\nOn a similar note, the paper is often vague about how their approach is different from other approaches in the literature. Specifically the distinction between \"test time meta-learning\" and \"generalization across varying test configurations in each episode\" is difficult to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We address your concerns in detail below and will update our paper accordingly.\n\n> Lack of comparison to existing approaches\n\nThank you for pointing this out. We compare against two more relevant baselines, Agentic Transformer (AT [1]) and Decision Transformer (DT [2]). Due to the limit on reply length, please refer to the [global response](https://tinyurl.com/4bc2469z) for comparison and discussions.\n\n> Differences between Algorithm Distillation\n\nThanks for raising this question. First, we would like to highlight that we have different focus and scope. AD [3] focuses on learning a meta-RL agent that demonstrates in-context improvement during test time without gradient updates. Instead, we focus on how to improve the data efficiency for Transformer-based agents by explicitly formulating data in curricular sequences. To achieve this goal, we introduce and validate three solutions in RL and IL settings, which are underexplored in AD.\n\nSecond, we admit that our learning-progress-based curriculum bears a resemblance to AD at first glance, but they still differ in several angles, as detailed below.\n\n- On data generation. AD requires **N** different single-task source agents to generate data, which amounts to N different copies of model weights. This is evident in the second requirement and line 1 in Algorithm 1 in the AD paper. However, our learning-progress-based curriculum makes minimal assumptions about the diversity of source agents. In fact, we only require a single multi-task agent to generate training data. This makes our method more applicable since it requires less storage and time.\n\n- On evaluation. AD is supposed to only work with a single task instance at test time. This is evident in line 12 in Algorithm 1 in the AD paper. In other words, every time the environment is reset, the agent will be spawned at exactly the same initial location. Goal location and other environmental transitions also keep the same for all evaluation episodes. However, our agent is evaluated with new and different task instances every episode, meaning that the agent's initial location, goal location, and other factors change every test episode. Therefore, our test setting is **strictly harder** than AD’s.\n\nDue to the above two differences, we do not think our work is directly comparable to AD. We will still conceptually compare our learning-progress-based curriculum against AD and clarify the discussion in the camera-ready version.\n\n> Differences between Adaptive Agent Team et al. [4] \n\nWe would like to highlight the differences between AdA [4] from the following perspectives.\n\n- Scope and focus. Similar to how our work compares against AD, we focus on improving the data efficiency for Transformer-based agents by explicitly formulating data in curricular sequences. We introduce and validate three solutions toward this goal. Our method also works with imitation learning settings — robotic manipulation from human demonstrations — which meta-RL literature such as AdA does not explore.\n\n- Online data generation vs. offline data generation. AdA is trained by online RL (Muesli [5], specifically), where the agent is allowed to learn from experiences through trial-and-error. However, our cross-episodic curriculum method is offline behavior cloning instead, where training data are first generated and then fixed and used for learning.\n\n- Reproducibility and open-sourceness. AdA is developed and evaluated only on XLand [6], which has not been open-sourced and accessible by the community. Their algorithm implementation is also not open-sourced. By contrast, our method is developed and validated with open-sourced environments (DMLab [7] and RoboMimic [8]). We also open-source our algorithm implementation to contribute to the community. It is easy and straightforward for the community to reproduce our results, and further develop follow-up techniques on other domains.\n\n> More discussion of the feasibility of getting curricular data\n\nThanks for your suggestion. We discuss the feasibility of obtaining each curriculum one by one in the [global response](https://tinyurl.com/4bc2469z). We will move the discussion to the main text in the camera-ready version.\n\n**References**\n- [1] Liu and Abbeel, Emergent Agentic Transformer from Chain of Hindsight Experience, ICML 2023.\n- [2] Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling, NeurIPS 2021.\n- [3] Laskin et al., In-context Reinforcement Learning with Algorithm Distillation, ICLR 2022.\n- [4] Adaptive Agent Team, Human-Timescale Adaptation in an Open-Ended Task Space, ICML 2023.\n- [5] Hessel et al., Muesli: Combining Improvements in Policy Optimization, ICML 2022.\n- [6] Open-Ended Learning Team, Open-Ended Learning Leads to Generally Capable Agents, 2021.\n- [7] Beattie et al., DeepMind Lab, 2016.\n- [8] Mandlekar et al., What Matters in Learning from Offline Human Demonstrations for Robot Manipulation, CoRL 2021."}, {"review_id": "CDjsdvq05n", "reviewer": "Reviewer_hTvj", "summary": "This paper introduces a novel algorithm, referred to as Cross-Episodic Curriculum (CEC), which aims to improve the learning efficiency and generalization capabilities of Transformer agents in multi-task RL settings. The algorithm has been specifically developed to exploit the limited availability of sub-optimal data in environments with a scarcity of data, such as robot learning. The researchers examine the efficacy of Cooperative Evolutionary Computation (CEC) in two distinct and representative scenarios: online RL using 3D maze environments in DeepMind Lab, and imitation learning from human demonstrations of varying quality in RoboMimic. The findings indicate that visuomotor policies, which were trained using the expertise-based curriculum, exhibit the capability to surpass established baselines and achieve superior performance on simulated robotic manipulation tasks. Furthermore, these policies demonstrate significantly better performance compared to agents trained using offline RL algorithms. The researchers reach the conclusion that Cross-Entropy Method (CEC) offers a potentially effective strategy for leveraging restricted yet sub-optimal data in contexts characterized by a scarcity of data, such as robot learning.", "questions": "1. What distinguishes CEC from hierarchical RL?\n2. The experimental results presented in section 4 raise two inquiries: (1) What are the reasons behind the comparatively inferior performance of CEC in comparison to vanilla RL when applied to relatively uncomplicated tasks? (2) What factors contribute to the subpar performance of Learning progress?\n3. The fundamental concept of CEC, as I comprehend it, is to facilitate the exchange of experiences derived from diverse environments, tasks, and episodes in order to enhance the performance of RL. The aforementioned viewpoint aligns with the fundamental principles of various curriculum RL approaches. However, extensive research has demonstrated that measuring learning progress is a highly efficacious metric. In contrast, the findings presented in this article deviate from this prevailing notion. The authors do not expound upon the specific definition of learning progress, which may be a crucial factor and potentially the primary underlying cause of Q.3.\n4. The experimental control group setting described earlier in this paper is deemed insufficient. In addition, it is requested that the authors incorporate a comparative analysis of the curriculum pertaining to environmental change, e.g. PAIRED (Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design).\n", "limitations": "The authors do not discuss the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n1. The authors conduct an empirical assessment to examine the efficacy of CEC in two distinct and representative scenarios: online reinforcement learning using 3D maze environments from DeepMind Lab, and imitation learning from human demonstrations of varying quality in RoboMimic.\n\n2. Comparative analysis: The findings indicate that visuomotor policies, which were trained using the expertise-based curriculum, exhibit the capability to equal or surpass established baselines in simulated robotic manipulation tasks. Furthermore, these policies demonstrate significantly superior performance compared to agents trained using offline reinforcement learning algorithms.\n\n3. Extensive applicability: The researchers reach the conclusion that CEC (Contextual Embedding Clustering) presents a promising strategy for leveraging limited yet sub-optimal data in scenarios characterized by a scarcity of data, such as robot learning. Moreover, they assert that CEC is both effective and widely applicable in diverse problem contexts.\n\n4. The open-source code pertaining to the algorithm is provided as supplementary materials and will be publicly accessible to facilitate further research on the learning of Transformer agents.", "weaknesses": "1. Restricted comparison: The study presents a comparison between the proposed algorithm and a small number of established baselines. However, it does not encompass a comprehensive range of contemporary algorithms, thereby constraining the breadth of the comparison(e.g.[1], [2]). \n\n2. Inadequate consideration of limitations: The manuscript fails to adequately address the limitations associated with the proposed algorithm or the conducted experiments. This omission may impede the comprehension of the potential drawbacks and challenges that may arise when employing the algorithm in real-world scenarios.\n\n3. How does the proposed algorithm scale to larger and more complex environments? The paper only considers relatively small and simple environments, and it is unclear how well the algorithm would perform in larger and more complex environments, such as those encountered in real-world robotics applications.\n\n[1] Piotr Mirowsk, Razvan Pascanu, Fabio Viola, Learning to Navigate in Complex Environments.\n\n[2] Siyuan Li, Rui Wang, Minxue Tang, Chongjie Zhang, Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We address your concerns in detail below and will update our paper accordingly.\n\n> Restricted comparison\n\nThank you for your suggestions. Since [1,2] mainly focus on improving online RL with auxiliary tasks, these are useful to improve the source agents in our case. However, they are orthogonal to the main focus of our work, i.e., to improve the data efficiency of Transformer-based agents that learn in an offline manner. Therefore, we opt to compare against two more relevant baselines and will include a conceptual comparison against [1,2] in the final version.\n\nWe compare against Agentic Transformer (AT [3]) and Decision Transformer (DT [4]). Please refer to the [global response](https://tinyurl.com/4bc2469z) for comparison and discussions.\n\n> Inadequate consideration of limitations\n\nThank you for pointing this out. We have included the Limitations section in Appendix D at L714 due to space constraints in the initial submission. We follow your suggestion to discuss the feasibility of obtaining curricular data. Please refer to [global response](https://tinyurl.com/4bc2469z) for details.\n\n> Scale to larger and more complex environments\n\nFirst, we would like to highlight that the environments used in this work, DMLab [5] and RoboMimic [6], are generally considered to be challenging. On DMLab, state-of-the-art algorithms such as IMPALA [7] struggle to make any progress on some difficult tasks. We have shown promising results on difficult tasks like Irreversible Path, which encompasses challenges such as 3D navigation, exploration, long-range planning, etc. Video demos about this task can be found [here](https://tinyurl.com/yypwytt). RoboMimic [6] is a large-scale robotic manipulation benchmark designed to study IL. It consists of demonstrations collected by operators with varying proficiency. Competitive algorithms such as BeT [8] and Diffusion Policy [9] are continuously being developed. Our work falls into the same category to advance the frontier in this challenging robotic benchmark. We leave the experiments on real-world robotics applications to be an exciting future venue.\n\n> CEC vs hierarchical RL?\n\nCEC and hierarchical RL are two orthogonal concepts. Hierarchical RL refers to a group of algorithms that decompose complex tasks into simpler sub-tasks, while CEC is designed to enhance Transformer agents’ learning efficiency and generalization by leveraging cross-episodic experiences. That being said, CEC is complementary to any hierarchical RL approach, and the combination of CEC and hierarchical RL could be an interesting direction for future work.\n\n> Why does CEC perform worse than vanilla RL?\n\nVanilla RL and curriculum RL are directly trained on the test distribution for tens of thousands of episodes until convergence. They are further used as source agents to generate offline training data. Therefore, these two RL agents should be considered as **oracle**. This fact is stated at L178 in our submission. By contrast, our CEC agents are only trained for several hundred episodes generated by them. For our task-difficulty-based curriculum, test tasks are even outside the training distribution (Table 1 on page 5). Even so, our CEC agents perform comparably to and even outperform RL oracles, confirming the superior data efficiency achieved by our methods.\n\n> What factors contribute to the subpar performance of Learning progress?\n\nTo make the discussion more concrete, we visualize different policies on Irreversible Path. Videos can be found [here](https://tinyurl.com/yypwytt). We find that by keeping a task-difficulty-based curriculum in context, the agent is able to learn critical skills such as visual navigation and long-horizon planning from relatively easier tasks, then apply them to the most challenging setting. This is demonstrated by the fact that the agent can navigate to gradually distant goals. Evaluated on the same difficulty, the agent resulting from a learning-progress-based curriculum sometimes makes progress toward the goal. The relatively inferior performance could be due to the fact that source RL agents, which are used to generate training data, also struggle in this task.\n\n> The definition of learning progress\n\nAs mentioned in L77 of our paper, we view all RL tasks in this work as goal-reaching problems. Therefore, learning progress can be defined as the improvement in task success rate or reward over time. We will clarify this in our final version.\n\n> Curriculum pertaining to environmental change\n\nThe field of automatic environment design, albeit interesting, is beyond the scope of this work. Since we focus on improving the learning efficiency of Transformer agents on sub-optimal data, our evaluation has become more comprehensive after we added DT and AT baselines, thanks to your suggestion!\n\nIncorporating automatically generated environments could be an exciting future extension of CEC, and we will discuss PAIRED [10] in our final version.\n\n**References**\n- [1] Mirowski et al., Learning to Navigate in Complex Environments, ICLR 2017.\n- [2] Li et al., Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards, NeurIPS 2019.\n- [3] Liu and Abbeel, Emergent Agentic Transformer from Chain of Hindsight Experience, ICML 2023.\n- [4] Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling, NeurIPS 2021.\n- [5] Beattie et al., DeepMind Lab, 2016.\n- [6] Mandlekar et al., What Matters in Learning from Offline Human Demonstrations for Robot Manipulation, CoRL 2021.\n- [7] Espeholt et al., IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures, ICML 2018.\n- [8] Shafiullah et al., Behavior Transformers: Cloning k modes with one stone, NeurIPS 2022.\n- [9] Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023.\n- [10] Dennis et al., Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design, NeurIPS 2020."}, {"review_id": "seE3GFykpR", "reviewer": "Reviewer_3Crt", "summary": "This work proposes a new method, CEC, to boost the learning efficiency and generalization capability of the agent by structuring multiple episodes for deploying the transformer’s pattern, and sequence recognition capability. The proposed method shows improved performance compared to the baseline and shows generalization capability. ", "questions": "1. If we have to manually set a pair of trajectories and their corresponding curriculum levels, it means we know the quality or progress, or expertise of the obtained trajectories in the online RL process (e.g. Table 1, Eq (1)). I think this is a quite strong assumption or extra burden since the RL framework itself should enable the agent to automatically learn how to solve the problem without external intervention. It is not clear whether given prior curriculum sequence is a fair assumption. \n2. In line 107, it is not clear the meaning of the notation “n” and “t”. if “n” is the number of entire episodes with various types and “t” is the timestep, then does the transformer not refer to the full horizon of the episode when computing the log-likelihood of the policy? \n3. Also, I was wondering how the transformer distinguishes each trajectory’s level or expertise. I think the transformer cannot distinguish it without a learnable token or expertise embedding which works similarly to the position embedding in NLP. If we assume the level is given for embedding, it would be burdensome to collect the accurate level for each trajectory in practice. If the transformer is implemented without these considerations, how can we ensure the transformer leverage the information from various level of trajectories? I think it would be proper to compare with the transformer-based offline RL models such as Decision Transformer [1] or other recently proposed SOTA models to check whether the proposed method just memorizes every sequence pattern in the trajectories or infers something helpful from the curricular data.\n\n[1] Chen, Lili, et al. \"Decision transformer: Reinforcement learning via sequence modeling.\" *Advances in neural information processing systems* 34 (2021): 15084-15097.", "limitations": "The authors do not discuss the potential negative societal impact of their work and do not explicitly discuss the limitations of their method in the manuscript. I do not anticipate ethical issues.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors propose a new approach for the transformer-based agent that leverages the information from the various types of trajectories by deploying the representation capability of the transformer architecture. \n2. The proposed method shows knowledge distillation capability through several evaluation and generalization experiments quantitatively.", "weaknesses": " ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We address your concerns in detail below and will update our paper accordingly.\n\n> [...] It is not clear whether given prior curriculum sequence is a fair assumption.\n\nWe agree with you that accurately formulating a curriculum is challenging. In this work, we introduce and validate three curriculum designs. We discuss the practicality of obtaining each curriculum one by one and will move the discussion to the main text in the final version. Due to the limited reply length, please refer to the [global response](https://tinyurl.com/4bc2469z) for detailed discussions.\n\nFurthermore, to ensure fair comparison, we particularly compare against a new baseline that also assumes prior ordered sequence. We compare against Agentic Transformer (AT [1]), a **concurrent** work with the preprint version released after NeurIPS submission and conference version published at ICML in July. It is closely related to our work by training Transformers on sequences of trajectory **ascending sorted according to their rewards**. Similar to our assumption, this sorting also requires ordered trajectories. In our comparison, we find that our method outperforms AT on difficult tasks, while matching the performance on relatively easy tasks and when probed with new environmental changes. Please refer to the [global response](https://tinyurl.com/4bc2469z) for detailed results and discussions.\n\n> Unclear definition of “n” and “t”\n\nThanks for pointing out the confusion. Here “n” denotes the number of episodes, and “t” is the timestep within the current episode. It means that the policy is conditioned on two parts, the first is the historical observation sequence from previous episodes, and the second is the observation received in the ongoing episode, up to the current time step. We will make notations more clear in the final version.\n\n> How the transformer distinguishes each trajectory’s level or expertise without a learnable token or expertise embedding [...] It would be burdensome to collect the accurate level for each trajectory in practice.\n\nWe agree with you that adding expertise tokens or level embedding requires further assumptions. This is one reason behind our design choice not to include them. Instead, trajectory quality is implicitly included. For example, as shown in the visualization videos posted [here](https://tinyurl.com/yypwytt), in successful trajectories, goals are visible in the RGB observations. Therefore, we hypothesize that our model can still implicitly infer trajectory quality by sensing the change in the observation sequence's distribution.\n\n> I think it would be proper to compare with the transformer-based offline RL models such as Decision Transformer or other recently proposed SOTA models to check whether the proposed method just memorizes every sequence pattern in the trajectories or infers something helpful from the curricular data.\n\nThanks for the suggestion. We extend the main experiments on DMLab with two more baselines, Agentic Transformer (AT [1]) and Decision Transformer (DT [2]). Due to the limit on reply length, please refer to the [global response](https://tinyurl.com/4bc2469z) for comparison and discussions.\n\n> Societal Impact and Limitations\n\nDue to space constraints, we have included the Societal Impact and Limitations sections in the Appendix starting from L714. We will include them in the main text in the final version. Regarding the Limitation section, we will elaborate on the feasibility of obtaining curricular data, as discussed in the [global response](https://tinyurl.com/4bc2469z).\n\n**References**\n- [1] Liu and Abbeel, Emergent Agentic Transformer from Chain of Hindsight Experience, ICML 2023.\n- [2] Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling, NeurIPS 2021."}, {"review_id": "3giIZOMAVd", "reviewer": "Reviewer_btkb", "summary": "This paper presents Cross-Episodic Curriculum (CEC) to boost the learning efficiency and generalization of Transformer agents. Specifically, CEC places the cross-episodic experiences into a Transformer’s context, which forms the basis of a curriculum. The authors also provide three concrete curriculum implementations: Learning-progress-based curriculum, Task-difficulty-based curriculum and Expertise-based curriculum. Experiments on discrete control, such as in DeepMind Lab, and continuous control, as seen in RoboMimic, show the superior performance and strong generalization of CEC.", "questions": "1. Could the authors compare with more recent methods like Algorithm Distillation [1] and TIT [2], or discuss the reason why these methods are not suitable for comparision?  \n\n2. The key insight of CEC is that sequential cross-episodic data manifest useful learning signals that do not easily appear in any separated training episodes. Besides the better performance, could the authors provide more evidence for this insight?  \n\n3. Why Learning-progress-based curriculum is worse thanTask-difficulty-based curriculum in most evaluation environments?  ", "limitations": "It will be better to discuss which curriculum is suitable for which environment setting? ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Simple but effective idea for Cross-Episodic Curriculum.  \n2. Three concrete curriculum implementations for different settings (RL or IL).  \n3. Good paper writting. It is easy to follow the methods and experiments.", "weaknesses": "1.  My major concern is the baselines used in the experiments (and also the related works). Specifically, the authors only compare the Transformer-based BC agent, which is a weak baseline. The reviewer will encourage the authors to compare with more recent methods like Algorithm Distillation [1] and TIT [2], or discuss the reason why these methods are not suitable for comparision. (The reviewer guesses that BCQ and CQL are based on MLP?)   \n\n2. Some important details are missed in the experiments. For example, in the ablation study, the reviewer (and most readers I guess) would like to know how to implement the \"without cross-episodic attention\" and \"Curriculum granularity\", but the authors put these important information in the Appendix. \n\n\n[1] In-context Reinforcement Learning with Algorithm Distillation. ICLR 2023.  \n[2] Transformer in Transformer as Backbone for Deep Reinforcement Learning. 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We address your concerns in detail below and will update our paper accordingly.\n\n> Compare with more recent methods\n\nWe compare against two more relevant baselines. Please refer to [global response](https://tinyurl.com/4bc2469z) for comparison and discussions.\n\n> Differences between AD\n\nFirst, we would like to highlight that we have different focus and scope. AD [1] focuses on learning a meta RL agent that demonstrates in-context improvement during test time without gradient updates. Instead, we focus on how to improve the data efficiency for Transformer-based agents by explicitly formulating data in curricular sequences. To achieve this goal, we propose and validate three solutions in both RL and IL settings, which are underexplored in AD.\n\nSecond, we admit that our learning-progress-based curriculum bears a resemblance to AD at first glance, but they still differ in several angles, as detailed below.\n\n- On data generation. AD requires **N** different single-task source agents to generate data, which amounts to N different copies of model weights. This is evident in the second requirement and line 1 in Algorithm 1 in the AD paper. However, our learning-progress-based curriculum makes minimal assumptions about the diversity of source agents. In fact, we only require a single multi-task agent to generate training data. This makes our method more applicable since it requires less storage and time.\n\n- On evaluation. AD is supposed to only work with a single task instance at test time. This is evident in line 12 in Algorithm 1 in AD paper. In other words, every time the environment is reset, the agent will be spawned at exactly the same location. Goal location and other environmental transitions also keep the same for all evaluation episodes. However, our agent is evaluated with new and different task instances every episode, meaning that the agent's initial location, goal location, and other factors change every test episode. Therefore, our test setting is **strictly harder** than AD’s.\n\nDue to the above two differences, we do not think our work is directly comparable to AD. We will still conceptually compare our learning-progress-based curriculum against AD and clarify in the final version.\n\n> Differences between TIT\n\nAs pointed out in their paper, TIT [2] mainly focuses on designing better deep networks for RL. Albeit important, this is still orthogonal to our focus and not directly comparable. Indeed, TIT is complementary to our CEC method. It will be a promising avenue to combine CEC with TIT in future work. We will cite and discuss TIT, and conceptually compare in the final version.\n\n> Experiment details\n\nFor the variant w/o cross-episodic attention, the attention span is restricted to the ongoing episode. This is achieved by only feeding single-episode data during training time, and clearing up KV caches every environment reset at test time. Regarding curriculum granularity, we approximate it with the number of different difficulty levels and their intervals. For example, a curriculum with fewer difficulty levels is considered to be more coarse. One consisting of levels with larger intervals is considered to be more coarse. We will clarify in the final version.\n\n> More evidence for this insight [behind the proposed method]?\n\nTo qualitatively investigate the policy difference, we visualize the test-time behavior. Videos are provided [here](https://tinyurl.com/yypwytt). We find that by keeping a task-difficulty-based curriculum in context, the agent is able to learn critical skills such as visual navigation and long-horizon planning from relatively easier tasks, then apply them to the most challenging setting. This is demonstrated by the fact that the agent can navigate to gradually distant goals.\n\nFurthermore, a concurrent work [3] provides theoretical support for cross-episodic learning. In multi-arm bandit problems, by conditioning on a distribution over linear bandits (cross-episodic context), the policy can exploit the unknown structure, allowing more informed exploration and decision-making. They show that in-context pretraining over multiple episodes effectively performs posterior sampling — a provably sample-efficient Bayesian RL algorithm that learns faster than source algorithms used to generate the pretraining data.\n\n> Why Learning-progress-based curriculum is worse than task-difficulty-based curriculum?\n\nResulting agents’ performance is largely subjected to source agents’ quality. This is supported by videos provided [here](https://tinyurl.com/yypwytt), where we can see agents from learning-progress-based curriculum sometimes make progress toward the goals. This is due to the fact that source agents, which are used to generate training data, also struggle in this task. Nevertheless, agents from learning-progress-based curriculum still perform better than BC agents trained on expert data, suggesting the useful learning information emergent from cross-episodic experience.\n\n> Which curriculum is suitable for which environment\n\nWe introduced three curricula: Learning-progress-based curriculum (L), Task-difficulty-based curriculum (T), and Expertise-based curriculum (E). In RL settings, we recommend starting with L since it is the most straightforward. However, if the task is exceptionally challenging while easier versions of the task exist, we recommend T. In IL settings, if the data has non-uniform quality (e.g., the demonstrator improves over time), we recommend E.\n \n**References**\n- [1] Laskin et al., In-context Reinforcement Learning with Algorithm Distillation, ICLR 2022.\n- [2] Mao et al., Transformer in Transformer as Backbone for Deep Reinforcement Learning, 2022.\n- [3] Lee et al., Supervised Pretraining Can Learn In-Context Reinforcement Learning, 2023."}]

Author rebuttal:
# Global Response
We sincerely thank all reviewers for their thoughtful and constructive feedback. We really appreciate that all reviewers find our idea novel and important for Transformer-based agents. We attach updated versions of Figs 3 and 4 in the one-page PDF. In our response to each reviewer below, we address their individual questions and comments. We will update the paper and supplementary PDFs with revisions accordingly. We welcome any follow-up discussions!
## More Baseline Comparison
We extend the main experiments on DMLab with two more baselines, Agentic Transformer (AT [1]) and Decision Transformer (DT [2]). AT is a **concurrent** work with the preprint version released after NeurIPS submission and conference version published at ICML in July. It is closely related to our work by training Transformers on sequences of trajectory ascending sorted according to their rewards. DT is a popular Transformer-based offline RL method that conditions on return-to-go and does not use any forms of cross-episodic curriculum. We control data, model capacity, training strategy, etc. to be the same. We denote AT and DT trained on data consisting of a mixture of task difficulties as “AT (Mixed Difficulty)” and “DT (Mixed Difficulty)”. Note that these data are the same used to train “Ours (Task Difficulty)”. Similarly, we denote AT and DT directly trained on test difficulty as “AT (Single Difficulty)” and “DT (Single Difficulty)”. These data are the same used to train “Ours (Learning Progress)”. Figures are attached in the PDF. Averaged numerical results are presented as below.

*Fig 3 averaged results.*
|  | Ours (Task Difficulty), Auto | Ours (Task Difficulty), Fixed | Ours (Learning Progress) | DT (Mixed Difficulty) | DT (Single Difficulty) | AT (Mixed Difficulty) | AT (Single Difficulty) | BC w/ Expert Data | RL (Oracle) | Curriculum RL (Oracle) |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| Average | 51.4 | **54.4** | 32.4 | 35.3 | 11.7 | 42.7 | 33.4 | 14.2 | 40.6 | 50.6 |

*Fig 4 averaged results.*
|  | Ours (Task Difficulty) | Ours (Learning Progress) | DT (Mixed Difficulty) | DT (Single Difficulty) | AT (Mixed Difficulty) | AT (Single Difficulty) | BC w/ Expert Data | RL (Oracle) | Curriculum RL (Oracle) |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| Average | **39.6** | 27.78 | 31.8 | 13.6 | 39.4 | 29.2 | 18.1 | 30.0 | 37.6 |

We can see that our method with task-difficulty-based curriculum performs the best during evaluation (Fig 3, the first table above), confirming the benefit over the concurrent AT approach that leverages chain-of-hindsight experiences. When compared to DT, it outperforms by a significant margin, which suggests that our cross-episodic curriculum helps to squeeze learning signals that are useful for downstream decision-making. When evaluating the generalization ability (Fig 4, the second table above), our method performs better than the concurrent AT baseline and achieves significantly better results than other baselines. This empirically suggests that CEC helps to learn policies that are robust to environmental perturbations and can quickly generalize to new changes.
## Policy Visualization
We visualize agents' behavior on the DMLab task Irreversible Path [here](https://tinyurl.com/yypwytt). The agent trained from task-difficulty-based curriculum can navigate to gradually more distant goals. Keeping the curriculum in context helps to distill useful exploration and long-horizon planning skills, which are critical to the success of the most difficult level. Evaluated on the same difficulty, the agent resulting from a learning-progress-based curriculum sometimes makes progress. The relatively inferior performance could be due to the fact that source RL agents, which are used to generate training data, also struggle in this task. DT and AT demonstrate even worse behavior. They usually choose the wrong direction and get stuck forever.
## Discussion on the Feasibility of Obtaining Curricular Data

- Learning-progress-based curriculum. RL agents generally monotonically improve over time, so they naturally generate increasingly better data. Our curriculum is constructed by a series of checkpoints over the course of training. There is no additional assumption to formulate this curriculum.

- Task-difficulty-based curriculum. a) For environments with parameterized difficulty, it is straightforward to create a schedule based on the difficulty parameter to formulate the curriculum as we did in this work. b) For environments where the difficulty is not parameterized, methods such as those in [3] can be used. An exciting avenue of future research would be applying our method to real-world tasks without explicitly defined difficulty.

- Expertise-based curriculum. One limitation is the need to estimate the demonstrators’ proficiency. Some IL benchmarks, such as RoboMimic [4], already have proficiency labels. Despite this, to apply our method more broadly, one way to approximate proficiency is to rank trajectories by completion time. Moreover, in practice, a demonstrator’s proficiency naturally increases when collecting more data: from not being familiar with the teleoperation systems or the task to collecting data with muscle memory [5]. This progress could provide a rich learning signal using CEC.

We will move this discussion to the main text in the final version.

## References
- [1] Liu and Abbeel, Emergent Agentic Transformer from Chain of Hindsight Experience, ICML 2023.
- [2] Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling, NeurIPS 2021.
- [3] Kanitscheider et al., Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft, 2021.
- [4] Mandlekar et al., What Matters in Learning from Offline Human Demonstrations for Robot Manipulation, CoRL 2021.
- [5] Mandlekar et al., RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation, CoRL 2018.

Meta-review: The paper proposes cross-episode curricular approach to boost learning of transformer-based agents. The work is interesting in its perspective and was made more convincing by adding two extra baselines during the rebuttal, but could be improved by making it clearer under what circumstances CEC is expected to be helpful.
Decision: Accept (poster)