Title: Cross-Episodic Curriculum for Transformer Agents
Venue: NeurIPS 2023
Accepted: yes
Keywords: Transformers

In-context Learning

Reinforcement Learning

Robotics
Abstract: We present a new algorithm, Cross-Episodic Curriculum (CEC), to boost the learning efficiency and generalization of Transformer agents. Central to CEC is the placement of cross-episodic experiences into a Transformer’s context, which forms the basis of a curriculum. By sequentially structuring online learning trials and mixed-quality demonstrations, CEC constructs curricula that encapsulate learning progression and proficiency increase across episodes. Such synergy combined with the potent pattern recognition capabilities of Transformer models delivers a powerful cross-episodic attention mechanism. The effectiveness of CEC is demonstrated under two representative scenarios: one involving multi-task reinforcement learning with discrete control, such as in DeepMind Lab, where the curriculum captures the learning progression in both individual and progressively complex settings; and the other involving imitation learning with mixed-quality data for continuous control, as seen in RoboMimic, where the curriculum captures the improvement in demonstrators' expertise. In all instances, policies resulting from CEC exhibit superior performance and strong generalization. Code is open-sourced on the project website https://cec-agent.github.io/ to facilitate research on Transformer agent learning.

Cross-Episodic Curriculum for Transformer Agents

Abstract

We present a new algorithm, Cross-Episodic Curriculum (CEC), to boost the learning efficiency and generalization of Transformer agents. Central to CEC is the placement of cross-episodic experiences into a Transformer's context, which forms the basis of a curriculum. By sequentially structuring online learning trials and mixed-quality demonstrations, CEC constructs curricula that encapsulate learning progression and proficiency increase across episodes. Such synergy combined with the potent pattern recognition capabilities of Transformer models delivers a powerful cross-episodic attention mechanism. The effectiveness of CEC is demonstrated under two representative scenarios: one involving multi-task reinforcement learning with discrete control, such as in DeepMind Lab, where the curriculum captures the learning progression in both individual and progressively complex settings, and the other involving imitation learning with mixed-quality data for continuous control, as seen in RoboMimic, where the curriculum captures the improvement in demonstrators' expertise. In all instances, policies resulting from CEC exhibit superior performance and strong generalization. Code is opensourced on the project website cec-agent.github.io to facilitate research on Transformer agent learning.

1 Introduction

The paradigm shift driven by foundation models [8] is revolutionizing the communities who study sequential decision-making problems [80], with innovations focusing on control [2, 45, 38, 9], planning [76, 32, 33, 78, 17], pre-trained visual representation [57, 50, 67, 51, 82], among others. Despite the progress, the data-hungry nature makes the application of Transformer [75] agents extremely challenging in data-scarce domains like robotics [52, 53, 19, 38, 9]. This leads us to the question: Can we maximize the utilization of limited data, regardless of their optimality and construction, to foster more efficient learning?

To this end, this paper introduces a novel algorithm named Cross-Episodic Curriculum (CEC), a method that explicitly harnesses the shifting distributions of multiple experiences when organized into a curriculum. The key insight is that sequential cross-episodic data manifest useful learning signals that do not easily appear in any separated training episodes. 1 As illustrated in Figure 1, CEC realizes this through two stages: 1) formulating curricular sequences to capture (a) the policy improvement on single environments, (b) the learning progress on a series of progressively harder environments, or (c) the increase of demonstrators' proficiency; and 2) causally distilling policy improvements into the model weights of Transformer agents through cross-episodic attention . When a policy is trained to predict actions at current time steps, it can trace back beyond ongoing trials and internalize improved behaviors encoded in curricular data, thereby achieving efficient learning

and robust deployment when probed with visual or dynamics perturbations. Contrary to prior works like Algorithm Distillation (AD, Laskin et al. [42]) which, at test time, samples and retains a single task configuration across episodes for in-context refinement, our method, CEC, prioritizes zero-shot generalization across a distribution of test configurations. With CEC, agents are evaluated on a new task configuration in each episode, emphasizing adaptability to diverse tasks.

We investigate the effectiveness of CEC in enhancing sample efficiency and generalization with two representative case studies. They are: 1) Reinforcement Learning (RL) on DeepMind Lab (DMLab) [5], a 3D simulation encompassing visually diverse worlds, complicated environment dynamics, ego-centric pixel inputs, and joystick control; and 2) Imitation Learning (IL) from mixed-quality human demonstrations on RoboMimic [53], a framework designed to study robotic manipulation with proprioceptive and external camera observations and continuous control. Despite RL episodes being characterized by state-action-reward tuples and IL trajectories by state-action pairs, our method exclusively employs state-action pairs in its approach.

In challenging embodied navigation tasks, despite significant generalization gaps (Table 1), our method surpasses concurrent and competitive method Agentic Transformer (AT, Liu and Abbeel [47]). It also significantly outperforms popular offline RL methods such as Decision Transformer (DT, Chen et al. [13]) and baselines trained on expert data, with the same amount of parameters, architecture, and data size. It even exceeds RL oracles directly trained on test task distributions by 50% in a zero-shot manner. CEC also yields robust embodied policies that are up to 1 . 6 × better than RL oracles when zero-shot probed with unseen environment dynamics. When learning continuous robotic control, CEC successfully solves two simulated manipulation tasks, matching and outperforming previous well-established baselines [53, 25, 41]. Further ablation reveals that CEC with cross-episodic attention is a generally effective recipe for learning Transformer agents, especially in applications where sequential data exhibit moderate and smooth progression.

2 Cross-Episodic Curriculum: Formalism and Implementations

In this section, we establish the foundation for our cross-episodic curriculum method by first reviewing the preliminaries underlying our case studies, which encompass two representative scenarios in sequential decision-making. Subsequently, we formally introduce the assembly of curricular data and the specifics of model optimization utilizing cross-episodic attention. Lastly, we delve into the practical implementation of CEC in the context of these two scenarios.

2.1 Preliminaries

Reinforcement learning. We consider the setting where source agents learn through trial and error in partially observable environments. Denoting states s ∈ S and actions a ∈ A , an agent interacts in a Partially Observable Markov Decision Process (POMDP) with the transition function p ( s t +1 | s t , a t ) : S × A → S . It observes o ∈ O emitted from observation function Ω( o t | s t , a t -1 ) : S × A → O and receives scalar reward r from R ( s, a ) : S × A → R . Under the episodic task setting, RL seeks to learn a parameterized policy π θ ( ·| s ) that maximizes the return over a fixed length T of interaction steps: π θ = arg max θ ∈ Θ ∑ T -1 t =0 γ t r t , where γ ∈ [0 , 1) is a discount factor. Here we follow the canonical definition of an episode τ as a series of environment-agent interactions with length T , τ := ( s 0 , a 0 , r 0 , . . . , s T -1 , a T -1 , r T -1 , s T ) , where initial states s 0 are sampled from initial state distribution s 0 ∼ ρ 0 ( s ) and terminal states s T are reached once the elapsed timestep exceeds T . Additionally, we view all RL tasks considered in this work as goal-reaching problems [39, 26] and constrain all episodes to terminate upon task completion. It is worth noting that similar to previous work [42], training data are collected by source RL agents during their online learning. Nevertheless, once the dataset is obtained, our method is trained offline in a purely supervised manner.

Imitation learning. We consider IL settings with existing trajectories composed only of state-action pairs. Furthermore, we relax the assumption on demonstration optimality and allow them to be crowdsourced [10, 12, 11]. Data collected by operators with varying expertise are therefore unavoidable. Formally, we assume the access to a dataset D N := { τ 1 , . . . , τ N } consisting of N demonstrations, with each demonstrated trajectory τ i := ( s 0 , a 0 , . . . , s T -1 , a T -1 ) naturally identified as an episode. The goal of IL, specifically of behavior cloning (BC), is to learn a policy π θ that accurately models the distribution of behaviors. When viewed as goal-reaching problems, BC policies can be evaluated by measuring the success ratio in completing tasks [26].

2.2 Curricular Data Assembly and Model Optimization

Meaningful learning signals emerge when multiple trajectories are organized and examined crossepisodically along a curriculum axis. This valuable information, which is not easily discernible in individual training episodes, may encompass aspects such as the improvement of an RL agent's navigation policy or the generally effective manipulation skills exhibited by operators with diverse proficiency levels. With a powerful model architecture such as Transformer [75, 16], such emergent and valuable learning signals can be baked into policy weights, thereby boosting performance in embodied tasks.

̸

For a given embodied task M , we define its curriculum C M as a collection of trajectories τ consisting of state-action pairs. A series of ordered levels [ L 1 , . . . , L L ] partitions this collection such that ⋃ l ∈{ 1 ,...,L } L l = C M and ⋂ ∀ i,j ∈{ 1 ,...,L } ,i = j L { i,j } = ∅ . More importantly, these ordered levels characterize a curriculum by encoding, for example, learning progress in single environments, learning progress in a series of progressively harder environments, or the increase of the demonstrator's expertise.

With a curriculum C M := { τ i } N i =1 and its characteristics [ L 1 , . . . , L L ] , we construct a curricular sequence T that spans multiple episodes and captures the essence of gradual improvement in the following way:

The symbol ⊕ denotes the concatenation operation. U ( J K K ) denotes a uniform distribution over the discrete set { k ∈ N , k ≤ K } . In practice, we use values smaller than |L l | considering the memory consumption.

100B

119

150

100100

100123

(e) Can

We subsequently learn a causal policy that only depends on cross-episodic historical observations π θ ( ·| o ( ≤ n ) ≤ t ) . Note that this modeling strategy differs from previous work that views sequential decision-making as a big sequence-modeling problem [13, 37, 42, 38]. It instead resembles the causal policy in Baker et al. [4]. Nevertheless, we still follow the best practice [36, 60, 22] to provide previous action as an extra modality of observations in POMDP RL tasks.

We leverage the powerful attention mechanism of Transformer [75] to enable cross-episodic attention. Given observation series O ( n ) t := { o (1) 0 , . . . , o ( ≤ n ) ≤ t } (shorthanded as O hereafter for brevity), Transformer projects it into query Q = f Q ( O ) , key K = f K ( O ) , and value V = f V ( O ) matrices, with each row being a D -dim vector. Attention operation is performed to aggregate information:

Depending on whether the input arguments for f Q and f { K,V } are the same, attention operation can be further divided into self-attention and cross-attention. Since tasks considered in this work do not require additional conditioning for task specification, we follow previous work [4, 83] to utilize self-attention to process observation series. Nevertheless, ours can be naturally extended to handle, for example, natural language or multi-modal task prompts, following the cross-attention introduced in Jiang et al. [38].

Finally, this Transformer policy is trained by simply minimizing the negative log-likelihood objective J NLL of labeled actions, conditioned on cross-episodic context:

Regarding the specific memory architecture, we follow Baker et al. [4], Adaptive Agent Team et al. [1] to use Transformer-XL [16] as our model backbone. Thus, during deployment, we keep its hidden states propagating across test episodes to mimic the training settings.

2.3 Practical Implementations

We now discuss concrete instantiations of CEC for 1) RL with DMLab and 2) IL with RoboMimic. Detailed introductions to the benchmark and task selection are deferred to Sec. 3. We investigate the following three curricula, where the initial two pertain to RL, while the final one applies to IL:

Learning-progress-based curriculum. In the first instantiation, inspired by the literature on learning progress [54, 27, 65, 40], we view the progression of learning agents as a curriculum. Concretely, we train multi-task PPO agents [70, 63] on tasks drawn from test distributions. We record their online interactions during training, which faithfully reflect the learning progress. Finally, we form the learning-progress-based curriculum by sequentially concatenating episodes collected at different learning stages. Note that this procedure is different from Laskin et al. [42], where for each environment, the learning dynamics of multiple single-task RL agents has to be logged. In contrast, we only track a single multi-task agent per environment.

Task-difficulty-based curriculum. In the second instantiation, instead of taking snapshots of RL agents directly trained on test configurations, we collect learning progress on a series of easier

but progressively harder tasks. For instance, in an embodied navigation task, the test configuration includes 20 rooms. Rather than logging source agents' learning progression in the 20-room maze, we record in a series of mazes with 5, 10, and 15 rooms. We then structure stored episodes first following learning progress and then the increase of layout complexity. This practice naturally creates a task-difficulty-based curriculum , which resembles curriculum RL that is based on task difficulty [54, 58]. We find it especially helpful for hard-exploration problems where the source RL agent does not make meaningful progress.

Expertise-based curriculum. For the setting of IL from mixed-quality demonstrations, we instantiate a curriculum based on demonstrators' expertise. This design choice is motivated by literature on learning from heterogeneous demonstrators [6, 81], with the intuition that there is little to learn from novices but a lot from experts. To realize this idea, we leverage the Multi-Human dataset from RoboMimic [53]. Since it contains demonstrations collected by human demonstrators with varying proficiency, we organize offline demonstration trajectories following the increase of expertise to construct the expertise-based curriculum .

3 Experimental Setup

In this section, we elaborate on the experimental setup of our case studies. Our investigation spans two representative and distinct settings: 1) online reinforcement learning with 3D maze environments of DMLab [5], and 2) imitation learning from mixed-quality human demonstrations of RoboMimic [53]. For each of them, we discuss task selection, baselines, and training and evaluation protocols. Teasers of these tasks are shown in Figure 2.

3.1 Task Settings and Environments

DeepMind Lab [5] is a 3D learning environment with diverse tasks. Agents spawn in visually complex worlds, receive ego-centric (thus partially observable) RGB pixel inputs, and execute joystick actions. We consider three levels from this benchmark: Goal Maze, Watermaze [56], and Sky Maze with Irreversible Path. They challenge agents to explore, memorize, and plan over a long horizon. Their goals are similar - to navigate in complicated mazes and find a randomly spawned goal, upon which sparse rewards will be released. Episodes start with randomly spawned agents and goals and terminate once goals are reached or elapsed steps have exceeded pre-defined horizons.

RoboMimic [53] is a framework designed for studying robot manipulation and learning from demonstrations. Agents control robot arms with fixed bases, receive proprioceptive measurements and image observations from mounted cameras, and operate with continuous control. We evaluate two simulated tasks: 'Lift' and 'Can'. In the 'Lift' task, robots are tasked with picking up a small cube. In the 'Can' task, robots are required to pick up a soda can from a large bin and place it into a smaller target bin. Episodes start with randomly initialized object configuration and terminate upon successfully completing the task or exceeding pre-defined horizons.

3.2 Baselines

The primary goal of these case studies is to assess the effectiveness of our proposed cross-episodic curriculum in increasing the sample efficiency and boosting the generalization capability of Transformer agents. Therefore, in online RL settings, we compare against source RL agents which generate training data for our method and refer to them as oracles . These include a) PPO agents directly

trained on test task distributions, denoted as ' RL (Oracle) ' hereafter, and b) curriculum PPO agents that are gradually adapted from easier tasks to the test difficulty, which is referred to as ' Curriculum RL (Oracle) '. Furthermore, we compare against one concurrent and competitive method Agentic Transformer [47], denoted as ' AT '. It is closely related to our method, training Transformers on sequences of trajectory ascending sorted according to their rewards. We also compare against popular offline RL method Decision Transformer [13], denoted as ' DT '. Additionally, we include another behavior cloning agent that has the same model architecture as ours but is trained on optimal data without cross-episodic attention. This baseline is denoted as ' BC w/ Expert Data '. For the case study on IL from mixed-quality demonstrations, we adopt the most competing approach, BC-RNN , from Mandlekar et al. [53] as the main baseline. We also include comparisons against other offline RL methods [44] such as Batch-Constrained Q-learning ( BCQ ) [25] and Conservative Q-Learning ( CQL ) [41].

3.3 Training and Evaluation

We follow the best practice to train Transformer agents, including adopting AdamW optimizer [49], learning rate warm-up and cosine annealing [48], etc. Training is performed on NVIDIA V100 GPUs. During evaluation, for agents resulting from our method, each run involves several test rollouts to fill the context. We keep hidden states of Transformer-XL [16] propagating across episodes. We run other baselines and oracles for 100 episodes to estimate their performances. For our methods on RL settings, we compute the maximum success rate averaged across a sliding window over all test episodes to account for in-context improvement. The size of the sliding window equals one-quarter of the total test episodes. These values are averaged over 20 runs to constitute the final reporting metric. For our methods on the IL setting, since all training data are successful trajectories, we follow Mandlekar et al. [53] to report the maximum success rate achieved over the course of training, directly averaged over test episodes.

4 Experiments

We aim to answer the following four research questions through comprehensive experiments.

To what extent can our cross-episodic curriculum increase the sample efficiency of Transformer agents and boost their generalization capability?

Is CEC consistently effective and generally applicable across distinct learning settings?

What are the major components that contribute to the effectiveness of our method?

4.1 Main Evaluations

We answer the first two questions above by comparing learned agents from our method against 1) Reinforcement Learning (RL) oracles in online RL settings and 2) well-established baselines on learning from mixed-quality demonstrations in the Imitation Learning (IL) setting.

We first examine agents learned from learning-progress-based and task-difficulty-based curricula in challenging 3D maze environments. The first type of agent is denoted as ' Ours (Learning Progress) '. For the second type, to ensure that the evaluation also contains a series of tasks with increasing difficulty, we adopt two mechanisms that control the task sequencing [58]: 1) fixed sequencing where agents try each level of difficulty for a fixed amount of times regardless of their performance and 2) dynamic sequencing where agents are automatically promoted to the next difficulty level if they consecutively succeed in the previous level for three times. We denote these two variants as ' Ours (Task Difficulty), Fixed ' and ' Ours (Task Difficulty), Auto ', respectively. Note that because the task-difficulty-based curriculum does not contain any training data on test configurations, these two settings are zero-shot evaluated on test task distributions. We summarize these differences in Table 1. We denote AT and DT trained on data consisting of a mixture of task difficulties as ' AT (Mixed Difficulty) ' and ' DT (Mixed Difficulty) '. Note that these data are the same used to train 'Ours (Task Difficulty)'. Similarly, we denote AT and DT directly trained on test difficulty as 'AT (Single Difficulty)' and 'DT (Single Difficulty)'. These data are the same used to train 'Ours (Learning Progress)'.

Cross-episodic curriculum results in sample-efficient agents. As shown in Figure 3, on two out of three examined DMLab levels, CEC agents perform comparable to RL oracles and outperform the BC baselines trained on expert data by at most 2 . 8 × . On the hardest level Irreversible Path where agents have to plan the route ahead and cannot backtrack, both the BC baseline and RL oracle fail. However, our agents succeed in proposing correct paths that lead to goals and significantly outperform the curriculum RL oracle by 50% even in a zero-shot manner. Because CEC only requires environment interactions generated during the course of training of online source agents (the task-difficulty-based curriculum even contains fewer samples compared to the curriculum RL, as illustrated in Table 1), the comparable and even better performance demonstrates that our method yields highly sample-efficient embodied policies. On average, our method with task-difficulty-based curriculum performs the best during evaluation (Table A.5), confirming the benefit over the

concurrent AT approach that leverages chain-of-hindsight experiences. When compared to DT, it outperforms by a significant margin, which suggests that our cross-episodic curriculum helps to squeeze learning signals that are useful for downstream decision-making.

Cross-episodic curriculum boosts the generalization capability. To further investigate whether CEC can improve generalization at test time, we construct settings with unseen maze mechanisms (randomly open/closed doors), out-of-distribution difficulty, and different environment dynamics. See the Appendix, Sec. C.2 for the exact setups. As demonstrated in Figure 4, CEC generally improves Transformer agents in learning robust policies that can generalize to perturbations across various axes. On three settings where the BC w/ Expert Data baseline still manages to make progress, CEC agents are up to 2 × better. Compared to oracle curriculum RL agents, our policies significantly outperform them under three out of five examined scenarios. It is notable that on Irreversible Path with out-of-distribution difficulty, CEC agent is 1 . 6 × better than the curriculum RL oracle trained on the same data. These results highlight the benefit of learning with curricular contexts. On average, our method surpasses the concurrent AT baseline and achieves significantly better performance than other baselines (Table A.6). This empirically suggests that CEC helps to learn policies that are robust to environmental perturbations and can quickly generalize to new changes.

Cross-episodic curriculum is effective across a wide variety of learning scenarios. We now move beyond RL settings and study the effectiveness of the expertise-based curriculum in the IL setting with mixed-quality demonstrations. This is a common scenario, especially in robotics, where demonstrations are collected by human operators with varying proficiency [52]. As presented in Table 2, visuomotor policies trained with the expertise-based curriculum are able to match and outperform the well-established baseline [53] on two simulated robotic manipulation tasks and achieve significantly better performance than agents learned from prevalent offline RL algorithms [25, 41]. These results suggest that our cross-episodic curriculum is effective and broadly applicable across various problem settings. More importantly, it provides a promising approach to utilizing limited but sub-optimal data in data-scarce regimes such as robot learning.

4.2 Ablation Studies

In this section, we seek to answer the third research question to identify the components critical to the effectiveness of our approach. We focus on three parts: the importance of cross-episodic attention, the influence of curriculum granularity, and the effect of varying context length. Finally, we delve into the fourth question, identifying scenarios where CEC is expected to be helpful.

Importance of cross-episodic attention. The underlying hypothesis behind our method is that cross-episodic attention enables Transformer agents to distill policy improvement when mixed-optimality trajectories are viewed collectively. To test this, on DMLab levels and RoboMimic tasks, we train the same Transformer agents with the same curricular data and training epochs but

without cross-episodic attention. We denote such agents as ' Ours w/o Cross-Episodic Attention ' in Table 3. Results demonstrate that the ablated variants experience dramatic performance degradation on four out of five examined tasks, which suggests that naively behaviorally cloning sub-optimal data can be problematic and detrimental. Cross-episodic attention views curricular data collectively, facilitating the extraction of knowledge and patterns crucial for refining decision-making, thereby optimizing the use of sub-optimal data.

Curriculum granularity. We perform this ablation with the task-difficulty-based curriculum on DMLab levels, due to the ease of adjusting granularity. We treat the curricula listed in the column 'Ours (Task Difficulty)' in Table 1 as 'Fine', and gradually make them coarser to study the impact. Note that we ensure the same amount of training data. See the Appendix, Sec. C.4 for how we define granularity levels 'Medium' and 'Coarse'. We visualize the performance relative to the most fine-grained in Figure 5. The monotonic degradation of policy performance with respect to curriculum coarseness suggests that fine-grained curricula are critical for Transformer agents to mostly benefit from cross-episodic training.

Varying context length. Lastly, we study the effect of varying context length on DMLab and visualize it in Figure 6. We normalize all performance values relative

Fine

Medium

Coarse

Curriculum Granularity

80

85

90

95

100

Relative Performance (%)

Watermaze

Irreversible Path

to those of 'Ours (Task Difficulty), Auto' reported in Figure 3. It turns out that both too short and unnecessarily long context windows are harmful. On two out of three levels, using a shorter context decreases the performance even more. This finding coincides with Laskin et al. [42] that a sufficiently long Transformer context is necessary to retain cross-episodic information. Furthermore, we also discover that an unnecessarily long context is also harmful. We hypothesize that this is due to the consequent training and optimization instability.

Curriculum selection based on task complexities and data sources. For RL tasks, we recommend starting with the learning-progress-based curriculum. However, if the task itself is too challenging, such that source algorithms barely make progress, we recommend the task-difficultybased curriculum. In IL settings, we further investigate the performance of the learning-progress-based curriculum on RoboMimic tasks considered in this work. Detailed setup and results are included in Appendix, Sec C.5. To summarize, if human demonstrations are available, even if they are generated to be heterogeneous in quality, we recommend using the expertise-based curriculum. However, in the absence of human demonstrations and only with access to machine-generated data (e.g., generated by RL agents), our learning-progress-based curriculum is recommended because it achieves non-trivial performance and significantly outperforms offline RL methods such as CQL [41].

5 Related Work

0

20

40

60

80

100

Relative Performance (%)

300

400

500

800

1600

Sequential decision-making with Transformer agents. There are many ongoing efforts to replicate the strong emergent properties demonstrated by Transformer models for sequential decisionmaking problems [80]. Decision Transformer [13] and Trajectory Transformer [37] pioneered this thread by casting offline RL [44] as sequence modeling problems. Gato [68] learns a massively multitask agent that can be prompted to complete embodied tasks. MineDojo [22] and VPT [4] utilize numerous YouTube videos for large-scale pre-training in the video game Minecraft . VIMA [38] and RT-1 [9] build Transformer agents trained at scale for robotic manipulation tasks. BeT [71] and C-BeT [14] design novel techniques to learn from demonstrations with multiple modes with Trans-

formers. Our causal policy most resembles to VPT [4]. But we focus on designing learning techniques that are generally effective across a wide spectrum of learning scenarios and application domains.

Cross-episodic learning. Cross-episodic learning is a less-explored terrain despite that it has been discussed together with meta-RL [77] for a long time. RL 2 [18] uses recurrent neural networks for online meta-RL by optimizing multi-episodic value functions. Meta-Q-learning [21] instead learns multi-episodic value functions in an offline manner. Algorithm Distillation (AD) [42] and Adaptive Agent (AdA) [1] are two recent, inspiring methods in cross-episodic learning. Though at first glance our learning-progress-based curriculum appears similar to AD, significant differences emerge. Unlike AD, which focuses on in-context improvements at test time and requires numerous single-task source agents for data generation, our approach improves data efficiency for Transformer agents by structuring data in curricula, requiring only a single multi-task agent and allowing for diverse task instances during evaluations. Meanwhile, AdA, although using cross-episodic attention with a Transformer backbone, is rooted in online RL within a proprietary environment. In contrast, we focus on offline behavior cloning in accessible, open-source environments, also extending to IL scenarios unexplored by other meta-learning techniques. Complementary to this, another recent study [43] provides theoretical insight into cross-episodic learning.

Curriculum learning. Curriculum learning represents training strategies that organize learning samples in meaningful orders to facilitate learning [7]. It has been proven effective in numerous works that adaptively select simpler task [58, 74, 69, 62, 15, 55, 59, 46] or auxiliary rewards[35, 72]. Tasks are also parameterized to form curricula by manipulating goals [24, 30, 66], environment layouts[79, 3, 64], and reward functions [28, 34]. Inspired by this paradigm, our work harnesses the improving nature of sequential experiences to boost learning efficiency and generalization for embodied tasks.

6 Conclusion

In this work, we introduce a new learning algorithm named Cross-Episodic Curriculum to enhance the sample efficiency of policy learning and generalization capability of Transformer agents. It leverages the shifting distributions of past learning experiences or human demonstrations when they are viewed as curricula. Combined with cross-episodic attention, CEC yields embodied policies that attain high performance and robust generalization across distinct and representative RL and IL settings. CEC represents a solid step toward sample-efficient policy learning and is promising for data-scarce problems and real-world domains.

Limitations and future work. The CEC algorithm relies on the accurate formulation of curricular sequences that capture the improving nature of multiple experiences. However, defining these sequences accurately can be challenging, especially when dealing with complex environments or tasks. Incorrect or suboptimal formulations of these sequences could negatively impact the algorithm's effectiveness and the overall learning efficiency of the agents. A thorough exploration regarding the attainability of curricular data is elaborated upon in Appendix, Sec D.

In subsequent research, the applicability of CEC to real-world tasks, especially where task difficulty remains ambiguous, merits investigation. A deeper assessment of a demonstrator's proficiency trajectory - from initial unfamiliarity to the establishment of muscle memory - could offer a valuable learning signal. Moreover, integrating real-time human feedback to dynamically adjust the curriculum poses an intriguing challenge, potentially enabling CEC to efficiently operate in extended contexts, multi-agent environments, and tangible real-world tasks.

Acknowledgments and Disclosure of Funding

We thank Guanzhi Wang and Annie Xie for helpful discussions. We are grateful to Yifeng Zhu, Zhenyu Jiang, Soroush Nasiriany, Huihan Liu, and Rutav Shah for constructive feedback on an early draft of this paper. We also thank the anonymous reviewers for offering us insightful suggestions and kind encouragement during the review period. This work was partially supported by research funds from Salesforce and JP Morgan.

References

Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria LoksThompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rockt¨ aschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei Zhang. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv: Arxiv-2301.07608 , 2023.

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv: Arxiv-2204.01691 , 2022.

Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations , 2020.

Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv: Arxiv-2206.11795 , 2022.

Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich K¨ uttler, Andrew Lefrancq, Simon Green, V´ ıctor Vald´ es, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. arXiv preprint arXiv: Arxiv-1612.03801 , 2016.

M. Beliaev, Andy Shih, S. Ermon, Dorsa Sadigh, and Ramtin Pedarsani. Imitation learning by estimating expertise of demonstrators. International Conference On Machine Learning , 2022.

Yoshua Bengio, J´ erˆ ome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning , 2009.

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher R´ e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tram` er, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. arXiv preprint arXiv: Arxiv-2108.07258 , 2021.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv: Arxiv-2212.06817 , 2022.

Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. arXiv preprint arXiv: Arxiv-1904.06387 , 2019.

Zhangjie Cao and Dorsa Sadigh. Learning from imperfect demonstrations from agents with varying dynamics. arXiv preprint arXiv: Arxiv-2103.05910 , 2021.

Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. arXiv preprint arXiv: Arxiv-2010.11723 , 2020.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 15084-15097, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 7f489f642a0ddb10272b5c31057f0663-Abstract.html .

Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv: Arxiv-2210.10047 , 2022.

Wojciech Czarnecki, Siddhant M. Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Nicolas Manfred Otto Heess, Simon Osindero, and Razvan Pascanu. Mix&match - agent curricula for reinforcement learning. In International Conference on Machine Learning , 2018.

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv: Arxiv-1901.02860 , 2019.

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. arXiv preprint arXiv: Arxiv-2303.03378 , 2023.

Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl 2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: Arxiv1611.02779 , 2016.

Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv: Arxiv-2109.13396 , 2021.

Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv: Arxiv-1802.01561 , 2018.

Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J. Smola. Meta-q-learning. arXiv preprint arXiv: Arxiv-1910.00125 , 2019.

Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv: Arxiv-2206.08853 , 2022.

Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. arXiv preprint arXiv: Arxiv-1509.06113 , 2015.

S´ ebastien Forestier, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. ArXiv , abs/1708.02190, 2017.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv: Arxiv-1812.02900 , 2018.

Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv: Arxiv-1912.06088 , 2019.

Alex Graves, Marc G. Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. arXiv preprint arXiv: Arxiv-1704.03003 , 2017.

Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-learning for reinforcement learning. ArXiv , abs/1806.04640, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015. URL http://arxiv.org/abs/1512.03385 . arXiv:1512.03385 [cs].

David Held, Xinyang Geng, Carlos Florensa, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning , 2018.

Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalnikov, Piotr Sta´ nczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, L´ eonard Hussenot, Robert Dadashi, Gabriel Dulac-Arnold, Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed Kamyar Seyed Ghasemipour, Sertan Girgin, Olivier Pietquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio G´ omez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979 , 2020. URL https://arxiv.org/abs/2006.00979 .

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zeroshot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesv´ ari, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 9118-9147. PMLR, 2022. URL https://proceedings.mlr.press/v162/huang22a.html .

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv: Arxiv2207.05608 , 2022.

Allan Jabri, Kyle Hsu, Ben Eysenbach, Abhishek Gupta, Alexei A. Efros, Sergey Levine, and Chelsea Finn. Unsupervised curricula for visual meta-reinforcement learning. In Advances in Neural Information Processing Systems , 2019.

Max Jaderberg, Volodymyr Mnih, Wojciech Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations , 2017.

Max Jaderberg, Wojciech M. Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Casta˜ neda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science , 364(6443):859-865, 2019. doi: 10.1126/science.aau6249. URL https://www.science.org/doi/abs/10.1126/science. aau6249 .

Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 1273-1286, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 099fe6b0b444c23836c4a5d07346082b-Abstract.html .

Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv: Arxiv-2210.03094 , 2022.

Leslie Pack Kaelbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence , 1993.

Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton, Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, Oleg Klimov, and Jeff Clune. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint arXiv: Arxiv-2106.14876 , 2021.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv: Arxiv-2006.04779 , 2020.

Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, maxime gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=hy0a5MMPUv .

Jonathan N. Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. arXiv preprint arXiv: Arxiv-2306.14892 , 2023.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv: Arxiv-2005.01643 , 2020.

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv: Arxiv-2209.07753 , 2022.

Xingyu Lin, Harjatin Singh Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. In Advances in Neural Information Processing Systems , 2019.

Hao Liu and Pieter Abbeel. Emergent agentic transformer from chain of hindsight experience. arXiv preprint arXiv: Arxiv-2305.16554 , 2023.

Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview. net/forum?id=Skq89Scxx .

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7 .

Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv: Arxiv-2210.00030 , 2022.

Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv: Arxiv-2303.18240 , 2023.

Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, and Li Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. arXiv preprint arXiv: Arxiv-1811.02790 , 2018.

Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Mart´ ın-Mart´ ın. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv: Arxiv2108.03298 , 2021.

Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. arXiv preprint arXiv: Arxiv-1707.00183 , 2017.

Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. In IEEE transactions on neural networks and learning systems , 2019.

Richard G.M. Morris. Spatial localization does not require the presence of local cues. Learning and Motivation , 12(2):239-260, 1981. doi: 10.1016/0023-9690(81)90020-5. URL https: //app.dimensions.ai/details/publication/pub.1028012961 .

Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv: Arxiv-2203.12601 , 2022.

S. Narvekar, J. Sinapov, and P. Stone. Autonomous task sequencing for customized curriculum design in reinforcement learning. In International Joint Conference on Artificial Intelligence , 2017.

Sanmit Narvekar and Peter Stone. Learning curriculum policies for reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems . International Foundation for Autonomous Agents and Multiagent Systems, 2019.

OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal J´ ozefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: Arxiv-1912.06680 , 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´ eBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

B. Peng, J. MacGlashan, R. Loftin, M. Littman, D. Roberts, and Matthew E. Taylor. Curriculum design for machine learners in sequential decision tasks. IEEE Transactions on Emerging Topics in Computational Intelligence , 2:268-277, 2018.

Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, and Vladlen Koltun. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. arXiv preprint arXiv: Arxiv-2006.11751 , 2020.

R´ emy Portelas, C´ edric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning , 2019.

R´ emy Portelas, C´ edric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. arXiv preprint arXiv: Arxiv-1910.07224 , 2019.

S´ ebastien Racani` ere, Andrew Kyle Lampinen, Adam Santoro, David P. Reichert, Vlad Firoiu, and Timothy P. Lillicrap. Automated curriculum generation through setter-solver interactions. In International Conference on Learning Representations , 2020.

Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. arXiv preprint arXiv: Arxiv2210.03109 , 2022.

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. arXiv preprint arXiv: Arxiv-2205.06175 , 2022.

Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing-solving sparse reward tasks from scratch. In International Conference on Machine Learning , 2018.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: Arxiv-1707.06347 , 2017.

Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone. arXiv preprint arXiv: Arxiv-2206.11251 , 2022.

William B Shen, Danfei Xu, Yuke Zhu, Leonidas J Guibas, Li Fei-Fei, and Silvio Savarese. Situational fusion of visual representation for visual navigation. In Proceedings of the IEEE International Conference on Computer Vision , pages 2881-2890, 2019.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

M. Svetlik, Matteo Leonetti, J. Sinapov, Rishi Shah, Nick Walker, and P. Stone. Automatic curriculum graph generation for reinforcement learning agents. In AAAI Conference on Artificial Intelligence , 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: Arxiv1706.03762 , 2017.

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305.16291 , 2023.

Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: Arxiv-1611.05763 , 2016.

Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv: Arxiv-2302.01560 , 2023.

Jan W¨ ohlke, Felix Schmitt, and Herke van Hoof. A performance-based start state curriculum framework for reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems , pages 1503-1511, 2020.

Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv: Arxiv-2303.04129 , 2023.

Songyuan Zhang, ZHANGJIE CAO, Dorsa Sadigh, and Yanan Sui. Confidence-aware imitation learning from demonstrations with varying optimality. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 12340-12350. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ 670e8a43b246801ca1eaca97b3e19189-Paper.pdf .

Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta, Dinesh Jayaraman, Yecheng Jason Ma, and Luca Weihs. Universal visual decomposer: Long-horizon manipulation made easy, 2023.

Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for visionbased manipulation with object proposal priors. arXiv preprint arXiv: Arxiv-2210.11339 , 2022.

A Model Architecture

In this section, we provide comprehensive details about the Transformer model architectures considered in this work. We implement all models in PyTorch [61] and adapt the implementation of Transformer-XL from VPT [4].

A.1 Observation Encoding

Experiments conducted on both DMLab and RoboMimic include RGB image observations. For models trained on DMLab, we use a ConvNet [29] similar to the one used in Espeholt et al. [20]. For models trained on RoboMimic, we follow Mandlekar et al. [53] to use a ResNet-18 network [29] followed by a spatial-softmax layer [23]. We use independent and separate encoders for images taken from the wrist camera and frontal camera. Detailed model parameters are listed in Table A.1.

Since DMLab is highly partially observable, we follow previous work [20, 22, 4] to supply the model with previous action input. We learn 16-dim embedding vectors for all discrete actions.

To encode proprioceptive measurement in RoboMimic, we follow Mandlekar et al. [53] to not apply any learned encoding. Instead, these types of observation are concatenated with image features and passed altogether to the following layers. Note that we do not provide previous action inputs in RoboMimic, since we find doing so would incur significant overfitting.

A.2 Transformer Backbone

We use Transformer-XL [16] as our model backbone, adapted from Baker et al. [4]. Transformer-XL splits long sequences into shorter sub-sequences that reduce the computational cost of attention while allowing the hidden states to be carried across the entire input by attending to previous keys and values. This feature is critical for the long sequence inputs necessary for cross-episodic attention. Detailed model parameters are listed in Table A.2.

A.3 Action Decoding

To decode joystick actions in DMLab tasks, we learn a 3-layer MLP whose output directly parameterizes a categorical distribution. This action head has a hidden dimension of 128 with ReLU activations. The 'Goal Maze' and 'Irreversible Path' tasks have an action dimension of 7, while 'Watermaze' has 15 actions. To decode continuous actions in RoboMimic, we learn a 2-layer MLP that parameterizes a Gaussian Mixture Model (GMM) with 5 modes that generates a 7-dimensional action. This network

has a hidden dimension of 400 with ReLU activations. During deployment, we employ the 'low-noise evaluation' trick [31].

B Training Details and Hyperparameters

All experiments are conducted on cluster nodes with NVIDIA V100 GPUs. We utilize DDP (distributed data parallel) to accelerate the training if necessary. Training hyperparameters are listed in Table A.3.

C Experiment Details

C.1 DMLab Main Experiment

Our DMLab main experiment is conducted on three levels with task IDs

explore goal locations large ,

rooms watermaze ,

and skymaze irreversible path hard .

We use no action repeats during training and evaluation. For experiments with varying task difficulty, we select difficulty parameters 'room numbers', 'spawn radius', and 'built-in difficulty' for these three levels, respectively. We adopt environment wrappers and helper functions from Petrenko et al. [63] to flexibly and precisely maneuver task difficulties.

Due to different task horizons, we tune the context length of Transformer-XL models and vary curricular trajectories accordingly. These differences are summarized in Table A.4.

RL oracles serve as source agents used to generate training data for our methods and the 'BC w/ Expert Data' baseline. They are trained with the PPO [70] implementation from Petrenko et al. [63]. The 'BC w/ Expert Data' baselines have the same model architecture, training hyperparameters, and amount of training data as our method, but are trained solely on trajectories generated by the best performing RL oracles without cross-episodic attention.

C.2 DMLab Generalization

This series of experiments probe the zero-shot generalization capabilities of embodied agents in unseen maze configurations, out-of-distribution difficulty levels, and varying environment dynamics. For the task 'Goal Maze w/ Unseen Mechanism', we use the level with task ID

explore obstructed goals large , which adds randomly opened and closed doors into the maze while ensuring a valid path to the goal always exists. An example of an agent's ego-centric observation is visualized in Figure A.1.

The task 'Irreversible Path (OOD. Difficulty)' corresponds to configurations with the built-in difficulty of 1 (agents are only trained on difficulty up to 0.9, as noted in Table 1). For tasks with varying environment dynamics, we directly test agents with an action repeat of 2. This is different from the training setting with no action repeat.

C.3 RoboMimic Main Experiment

We leverage the Multi-Human (MH) dataset from Mandlekar et al. [53]. It consists of demonstrations collected by operators with varying proficiency. We construct the expertise-based curriculum by following the order of 'worse operators, okay operators, then better operators'. We use a context length of 200 for both tasks. There are 90 trajectories per expertise level. To determine the number of trajectories per expertise level when constructing curricular data, we uniformly sample an integer from [1 , 5] (inclusively). The 'Lift' and 'Can' tasks are solved after training for 33 epochs and 179 epochs, respectively. We control for the same number of training epochs in subsequent ablation studies.

C.4 Ablation Study on Curriculum Granularity

We perform this ablation with the task-difficulty-based curriculum on DMLab levels due to the ease of adjusting granularity. The definition of varying levels of curriculum coarseness is listed in Table A.7.

C.5 Comparison of Curricula in RoboMimic

In IL settings, we further explored the efficacy of various curricula. For the RoboMimic tasks examined, we employed a learning-progress-based curriculum, ensuring the total training trajectories matched those of the expertise-based curriculum (i.e., 270 trajectories per task). All other parameters remained consistent, with the training data derived from RoboMimic's machine-generated dataset.

Table A.8 indicates that when heterogeneous-quality human demonstrations are accessible, the expertise-based curriculum is preferable due to its superior performance over the learning-progressbased approach. Conversely, without expert demonstrations and relying solely on machine-generated data, the learning-progress-based curriculum is still commendable. It offers noteworthy results and surpasses offline RL methods like CQL [41], even though CQL is trained on the full RoboMimic dataset, encompassing 1500 trajectories for the Lift task and 3900 for the Can task.

D Feasibility of Obtaining Curricular Data

The challenge of accurately orchestrating a curriculum is non-trivial and hinges on various factors. In the present work, three curriculum designs are introduced and validated, each with its practical considerations and underlying assumptions, discussed herein.

Learning-Progress-Based Curriculum. RL agents typically exhibit monotonic improvement over training epochs, thereby naturally producing incrementally better data. The curriculum here is devised through a series of checkpoints throughout the training duration, necessitating no supplementary assumptions for its formulation.

Task-Difficulty-Based Curriculum. In contexts where environmental difficulty is parameterizable, curricula can be structured through a schedule, determined by the relevant difficulty parameter, as demonstrated within this work. In scenarios lacking parameterized difficulty, alternatives such as methods proposed by Kanitscheider et al. [40] may be employed. The application of our method to tasks where difficulty is not explicitly characterized presents an intriguing avenue for future research.

Expertise-Based Curriculum. A notable limitation resides in the requisite to estimate demonstrators' proficiency. While some IL benchmarks, e.g., RoboMimic [53], come pre-equipped with proficiency labels, a broader application of our method necessitates an approximation of proficiency. One plausible approach entails ranking trajectories via completion time. Furthermore,

a demonstrator's proficiency is likely to organically improve-from initial unfamiliarity with teleoperation systems or tasks, to a stage of executing data collection with muscle memory [52]. This progression potentially provides a rich learning signal conducive for CEC application.

E Broader Impact

Our Cross-Episodic Curriculum can significantly enhance Transformer agent learning but carries potential societal impacts. The efficiency of our method depends on the curriculum's design. If the curriculum unintentionally reflects biases, it could lead to the amplification of these biases in learned policies, potentially perpetuating unfair or discriminatory outcomes in AI-driven decisions. Furthermore, the computational intensity of our approach at evaluation could contribute to increased energy usage, which has implications for the environmental footprint of AI applications.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Cross-Episodic Curriculum (CEC) to boost the learning efficiency and generalization of Transformer agents. Specifically, CEC places the cross-episodic experiences into a Transformer’s context, which forms the basis of a curriculum. The authors also provide three concrete curriculum implementations: Learning-progress-based curriculum, Task-difficulty-based curriculum and Expertise-based curriculum. Experiments on discrete control, such as in DeepMind Lab, and continuous control, as seen in RoboMimic, show the superior performance and strong generalization of CEC.\n\nSTRENGTHS:\n1. Simple but effective idea for Cross-Episodic Curriculum.  \n2. Three concrete curriculum implementations for different settings (RL or IL).  \n3. Good paper writting. It is easy to follow the methods and experiments.\n\nWEAKNESSES:\n1.  My major concern is the baselines used in the experiments (and also the related works). Specifically, the authors only compare the Transformer-based BC agent, which is a weak baseline. The reviewer will encourage the authors to compare with more recent methods like Algorithm Distillation [1] and TIT [2], or discuss the reason why these methods are not suitable for comparision. (The reviewer guesses that BCQ and CQL are based on MLP?)   \n\n2. Some important details are missed in the experiments. For example, in the ablation study, the reviewer (and most readers I guess) would like to know how to implement the \"without cross-episodic attention\" and \"Curriculum granularity\", but the authors put these important information in the Appendix. \n\n\n[1] In-context Reinforcement Learning with Algorithm Distillation. ICLR 2023.  \n[2] Transformer in Transformer as Backbone for Deep Reinforcement Learning. 2022.\n\nQUESTIONS:\n1. Could the authors compare with more recent methods like Algorithm Distillation [1] and TIT [2], or discuss the reason why these methods are not suitable for comparision?  \n\n2. The key insight of CEC is that sequential cross-episodic data manifest useful learning signals that do not easily appear in any separated training episodes. Besides the better performance, could the authors provide more evidence for this insight?  \n\n3. Why Learning-progress-based curriculum is worse thanTask-difficulty-based curriculum in most evaluation environments?\n\nLIMITATIONS:\nIt will be better to discuss which curriculum is suitable for which environment setting?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new method, CEC, to boost the learning efficiency and generalization capability of the agent by structuring multiple episodes for deploying the transformer’s pattern, and sequence recognition capability. The proposed method shows improved performance compared to the baseline and shows generalization capability.\n\nSTRENGTHS:\n1. The authors propose a new approach for the transformer-based agent that leverages the information from the various types of trajectories by deploying the representation capability of the transformer architecture. \n2. The proposed method shows knowledge distillation capability through several evaluation and generalization experiments quantitatively.\n\nQUESTIONS:\n1. If we have to manually set a pair of trajectories and their corresponding curriculum levels, it means we know the quality or progress, or expertise of the obtained trajectories in the online RL process (e.g. Table 1, Eq (1)). I think this is a quite strong assumption or extra burden since the RL framework itself should enable the agent to automatically learn how to solve the problem without external intervention. It is not clear whether given prior curriculum sequence is a fair assumption. \n2. In line 107, it is not clear the meaning of the notation “n” and “t”. if “n” is the number of entire episodes with various types and “t” is the timestep, then does the transformer not refer to the full horizon of the episode when computing the log-likelihood of the policy? \n3. Also, I was wondering how the transformer distinguishes each trajectory’s level or expertise. I think the transformer cannot distinguish it without a learnable token or expertise embedding which works similarly to the position embedding in NLP. If we assume the level is given for embedding, it would be burdensome to collect the accurate level for each trajectory in practice. If the transformer is implemented without these considerations, how can we ensure the transformer leverage the information from various level of trajectories? I think it would be proper to compare with the transformer-based offline RL models such as Decision Transformer [1] or other recently proposed SOTA models to check whether the proposed method just memorizes every sequence pattern in the trajectories or infers something helpful from the curricular data.\n\n[1] Chen, Lili, et al. \"Decision transformer: Reinforcement learning via sequence modeling.\" *Advances in neural information processing systems* 34 (2021): 15084-15097.\n\nLIMITATIONS:\nThe authors do not discuss the potential negative societal impact of their work and do not explicitly discuss the limitations of their method in the manuscript. I do not anticipate ethical issues.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel algorithm, referred to as Cross-Episodic Curriculum (CEC), which aims to improve the learning efficiency and generalization capabilities of Transformer agents in multi-task RL settings. The algorithm has been specifically developed to exploit the limited availability of sub-optimal data in environments with a scarcity of data, such as robot learning. The researchers examine the efficacy of Cooperative Evolutionary Computation (CEC) in two distinct and representative scenarios: online RL using 3D maze environments in DeepMind Lab, and imitation learning from human demonstrations of varying quality in RoboMimic. The findings indicate that visuomotor policies, which were trained using the expertise-based curriculum, exhibit the capability to surpass established baselines and achieve superior performance on simulated robotic manipulation tasks. Furthermore, these policies demonstrate significantly better performance compared to agents trained using offline RL algorithms. The researchers reach the conclusion that Cross-Entropy Method (CEC) offers a potentially effective strategy for leveraging restricted yet sub-optimal data in contexts characterized by a scarcity of data, such as robot learning.\n\nSTRENGTHS:\n1. The authors conduct an empirical assessment to examine the efficacy of CEC in two distinct and representative scenarios: online reinforcement learning using 3D maze environments from DeepMind Lab, and imitation learning from human demonstrations of varying quality in RoboMimic.\n\n2. Comparative analysis: The findings indicate that visuomotor policies, which were trained using the expertise-based curriculum, exhibit the capability to equal or surpass established baselines in simulated robotic manipulation tasks. Furthermore, these policies demonstrate significantly superior performance compared to agents trained using offline reinforcement learning algorithms.\n\n3. Extensive applicability: The researchers reach the conclusion that CEC (Contextual Embedding Clustering) presents a promising strategy for leveraging limited yet sub-optimal data in scenarios characterized by a scarcity of data, such as robot learning. Moreover, they assert that CEC is both effective and widely applicable in diverse problem contexts.\n\n4. The open-source code pertaining to the algorithm is provided as supplementary materials and will be publicly accessible to facilitate further research on the learning of Transformer agents.\n\nWEAKNESSES:\n1. Restricted comparison: The study presents a comparison between the proposed algorithm and a small number of established baselines. However, it does not encompass a comprehensive range of contemporary algorithms, thereby constraining the breadth of the comparison(e.g.[1], [2]). \n\n2. Inadequate consideration of limitations: The manuscript fails to adequately address the limitations associated with the proposed algorithm or the conducted experiments. This omission may impede the comprehension of the potential drawbacks and challenges that may arise when employing the algorithm in real-world scenarios.\n\n3. How does the proposed algorithm scale to larger and more complex environments? The paper only considers relatively small and simple environments, and it is unclear how well the algorithm would perform in larger and more complex environments, such as those encountered in real-world robotics applications.\n\n[1] Piotr Mirowsk, Razvan Pascanu, Fabio Viola, Learning to Navigate in Complex Environments.\n\n[2] Siyuan Li, Rui Wang, Minxue Tang, Chongjie Zhang, Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards.\n\nQUESTIONS:\n1. What distinguishes CEC from hierarchical RL?\n2. The experimental results presented in section 4 raise two inquiries: (1) What are the reasons behind the comparatively inferior performance of CEC in comparison to vanilla RL when applied to relatively uncomplicated tasks? (2) What factors contribute to the subpar performance of Learning progress?\n3. The fundamental concept of CEC, as I comprehend it, is to facilitate the exchange of experiences derived from diverse environments, tasks, and episodes in order to enhance the performance of RL. The aforementioned viewpoint aligns with the fundamental principles of various curriculum RL approaches. However, extensive research has demonstrated that measuring learning progress is a highly efficacious metric. In contrast, the findings presented in this article deviate from this prevailing notion. The authors do not expound upon the specific definition of learning progress, which may be a crucial factor and potentially the primary underlying cause of Q.3.\n4. The experimental control group setting described earlier in this paper is deemed insufficient. In addition, it is requested that the authors incorporate a comparative analysis of the curriculum pertaining to environmental change, e.g. PAIRED (Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design).\n\nLIMITATIONS:\nThe authors do not discuss the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to study mechanisms of cross-episode attention to effectively learn to improve polices by training on contexts containing gradually improving trajectories.\n\nSTRENGTHS:\nOn the whole, this paper is well written.  The topic of transformers in-context learning as an approach to planing and RL is of increasing importance. This work provides another important datapoint in that space for a technique which has scientific value to try.\n\nWEAKNESSES:\nThe main weakness with this work is its lack of comparison to existing approaches in the space.  The empirical results are only compared against approaches that are not intending to take advantage of in-context learning.  This looses most of the scientific value of the work when several other prominent works have used the in-context learning capabilities of transformers in various ways to great success. It is known that in-context learning can help with these sorts of tasks, the question which this paper is in the position to answer, is if this way of applying in-context learning works better or worse than the others which have been tried. Even if this approach is importantly distinct from those, it is important to compare against the other natural approaches that have already been tried, such as Algorithmic Distillation or AdA, to understand if this approach to applying in-context learning works better or worse than other approaches.\n\nOn a similar note, the paper is often vague about how their approach is different from other approaches in the literature. Specifically the distinction between \"test time meta-learning\" and \"generalization across varying test configurations in each episode\" is difficult to understand.\n\nQUESTIONS:\nHow does your RL approach in the DM labs differ from the general approach from Adaptive Agents Team et al?\n\nLIMITATIONS:\nThe paper could include more discussion of the feasibility of getting data with the requisite ordering of poor data first, better data next, best data last.  It could also be useful for the paper to include some mention of the limitations of the imitation paradigm.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}