SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We first examine agents learned from learning-progress-based and task-difficulty-based curricula in challenging 3D maze environments. The first type of agent is denoted as "Ours (Learning Progress)". For the second type, to ensure that the evaluation also contains a series of tasks with increasing difficulty, we adopt two mechanisms that control the task sequencing [58]: 1) fixed sequencing where agents try each level of difficulty for a fixed amount of times regardless of their performance and 2) dynamic sequencing where agents are automatically promoted to the next difficulty level if they consecutively succeed in the previous level for three times. We denote these two variants as "Ours (Task Difficulty), Fixed" and "Ours (Task Difficulty), Auto", respectively. Note that because the task-difficulty-based curriculum does not contain any training data on test configurations, these two settings are zero-shot evaluated on test task distributions. We summarize these differences in Table 1. We denote AT and DT trained on data consisting of a mixture of task difficulties as "AT (Mixed Difficulty)" and "DT (Mixed Difficulty)". Note that these data are the same used to train "Ours (Task Difficulty)". Similarly, we denote AT and DT directly trained on test difficulty as "AT (Single Difficulty)" and "DT (Single Difficulty)". These data are the same used to train "Ours (Learning Progress)".
Cross-episodic curriculum results in sample-efficient agents. As shown in Figure 3, on two out of three examined DMLab levels, CEC agents perform comparable to RL oracles and outperform the BC baselines trained on expert data by at most  $2.8 \times$ . On the hardest level Irreversible Path where agents have to plan the route ahead and cannot backtrack, both the BC baseline and RL oracle fail. However, our agents succeed in proposing correct paths that lead to goals and significantly outperform the curriculum RL oracle by  $50\%$  even in a zero-shot manner. Because CEC only requires environment interactions generated during the course of training of online source agents (the task-difficulty-based curriculum even contains fewer samples compared to the curriculum RL, as illustrated in Table 1), the comparable and even better performance demonstrates that our method yields highly sample-efficient embodied policies. On average, our method with task-difficulty-based curriculum performs the best during evaluation (Table A.5), confirming the benefit over the
Table 2: Evaluation results on RoboMimic. Visuomotor policies trained with our expertise-based curriculum outperform the most competing history-dependent behavior cloning baseline, as well as other offline RL algorithms. For our method on the Lift task, we conduct 5 independent runs each with 10 rollout episodes. On the Can task, we conduct 10 independent runs each with 5 rollout episodes due to the longer horizon required to complete the task. Standard deviations are included.
Task	Ours	BC-RNN [53]	BCQ [25]	CQL [41]
Lift	100.0 ± 0.0	100.0 ± 0.0	93.3 ± 0.9	11.3 ± 9.3
Can	100.0 ± 0.0	96.0 ± 1.6	77.3 ± 6.8	0.0 ± 0.0
Table 3: Ablation on the importance of cross-episodic attention. Transformer agents trained with the same curricular data but without cross-episodic attention degrade significantly during evaluation, suggesting its indispensable role in learning highly performant policies.
DMLab	RoboMimic
Goal Maze	Watermaze	Irreversible Path	Lift	Can
Ours	65.2 ± 6.7	50.9 ± 6.6	38.2 ± 7.0	100.0 ± 0.0	100.0 ± 0.0
Ours w/o Cross-Episodic Attention	35.0 ± 7.1	20.0 ± 2.5	3.8 ± 4.9	75.9 ± 12.3	99.3 ± 0.9
concurrent AT approach that leverages chain-of-hindsight experiences. When compared to DT, it outperforms by a significant margin, which suggests that our cross-episodic curriculum helps to squeeze learning signals that are useful for downstream decision-making.
Cross-episodic curriculum boosts the generalization capability. To further investigate whether CEC can improve generalization at test time, we construct settings with unseen maze mechanisms (randomly open/closed doors), out-of-distribution difficulty, and different environment dynamics. See the Appendix, Sec. C.2 for the exact setups. As demonstrated in Figure 4, CEC generally improves Transformer agents in learning robust policies that can generalize to perturbations across various axes. On three settings where the BC w/ Expert Data baseline still manages to make progress, CEC agents are up to  $2 \times$  better. Compared to oracle curriculum RL agents, our policies significantly outperform them under three out of five examined scenarios. It is notable that on Irreversible Path with out-of-distribution difficulty, CEC agent is  $1.6 \times$  better than the curriculum RL oracle trained on the same data. These results highlight the benefit of learning with curricular contexts. On average, our method surpasses the concurrent AT baseline and achieves significantly better performance than other baselines (Table A.6). This empirically suggests that CEC helps to learn policies that are robust to environmental perturbations and can quickly generalize to new changes.
Cross-episodic curriculum is effective across a wide variety of learning scenarios. We now move beyond RL settings and study the effectiveness of the expertise-based curriculum in the IL setting with mixed-quality demonstrations. This is a common scenario, especially in robotics, where demonstrations are collected by human operators with varying proficiency [52]. As presented in Table 2, visuomotor policies trained with the expertise-based curriculum are able to match and outperform the well-established baseline [53] on two simulated robotic manipulation tasks and achieve significantly better performance than agents learned from prevalent offline RL algorithms [25, 41]. These results suggest that our cross-episodic curriculum is effective and broadly applicable across various problem settings. More importantly, it provides a promising approach to utilizing limited but sub-optimal data in data-scarce regimes such as robot learning.
In this section, we seek to answer the third research question to identify the components critical to the effectiveness of our approach. We focus on three parts: the importance of cross-episodic attention, the influence of curriculum granularity, and the effect of varying context length. Finally, we delve into the fourth question, identifying scenarios where CEC is expected to be helpful.
Importance of cross-episodic attention. The underlying hypothesis behind our method is that cross-episodic attention enables Transformer agents to distill policy improvement when mixed-optimality trajectories are viewed collectively. To test this, on DMLab levels and RoboMimic tasks, we train the same Transformer agents with the same curricular data and training epochs but
without cross-episodic attention. We denote such agents as "Ours w/o Cross-Episodic Attention" in Table 3. Results demonstrate that the ablated variants experience dramatic performance degradation on four out of five examined tasks, which suggests that naively behaviorally cloning sub-optimal data can be problematic and detrimental. Cross-episodic attention views curricular data collectively, facilitating the extraction of knowledge and patterns crucial for refining decision-making, thereby optimizing the use of sub-optimal data.
Curriculum granularity. We perform this ablation with the task-difficulty-based curriculum on DMLab levels, due to the ease of adjusting granularity. We treat the curricula listed in the column "Ours (Task Difficulty)" in Table 1 as "Fine", and gradually make them coarser to study the impact. Note that we ensure the same amount of training data. See the Appendix, Sec. C.4 for how we define granularity levels "Medium" and "Coarse". We visualize the performance relative to the most fine-grained in Figure 5. The monotonic degradation of policy performance with respect to curriculum coarseness suggests that fine-grained curricula are critical for Transformer agents to mostly benefit from cross-episodic training.
Varying context length. Lastly, we study the effect of varying context length on DMLab and visualize it in Figure 6. We normalize all performance values relative
to those of "Ours (Task Difficulty), Auto" reported in Figure 3. It turns out that both too short and unnecessarily long context windows are harmful. On two out of three levels, using a shorter context decreases the performance even more. This finding coincides with Laskin et al. [42] that a sufficiently long Transformer context is necessary to retain cross-episodic information. Furthermore, we also discover that an unnecessarily long context is also harmful. We hypothesize that this is due to the consequent training and optimization instability.
Figure 5: We compare the performance relative to agents trained with the fine-grained curricula. Performance monotonically degrades as task-difficulty-based curricula become coarser.
Curriculum selection based on task complexities and data sources. For RL tasks, we recommend starting with the learning-progress-based curriculum. However, if the task itself is too challenging, such that source algorithms barely make progress, we recommend the task-difficulty-based curriculum. In IL settings, we further investigate the performance of the learning-progress-based curriculum on RoboMimic tasks considered in this work. Detailed setup and results are included in Appendix, Sec C.5. To summarize, if human demonstrations are available, even if they are generated to be heterogeneous in quality, we recommend using the expertise-based curriculum. However, in the absence of human demonstrations and only with access to machine-generated data (e.g., generated by RL agents), our learning-progress-based curriculum is recommended because it achieves non-trivial performance and significantly outperforms offline RL methods such as CQL [41].
Figure 6: Both short and unnecessarily long context windows decrease the performance. Numbers in the legend denote context lengths. Performance values are relative to those of "Ours (Task Difficulty), Auto" reported in Figure 3. "Irrevers. Path" stands for the task "Irreversible Path".
Sequential decision-making with Transformer agents. There are many ongoing efforts to replicate the strong emergent properties demonstrated by Transformer models for sequential decision-making problems [80]. Decision Transformer [13] and Trajectory Transformer [37] pioneered this thread by casting offline RL [44] as sequence modeling problems. Gato [68] learns a massively multi-task agent that can be prompted to complete embodied tasks. MineDojo [22] and VPT [4] utilize numerous YouTube videos for large-scale pre-training in the video game Minecraft. VIMA [38] and RT-1 [9] build Transformer agents trained at scale for robotic manipulation tasks. BeT [71] and C-BeT [14] design novel techniques to learn from demonstrations with multiple modes with Trans
formers. Our causal policy most resembles to VPT [4]. But we focus on designing learning techniques that are generally effective across a wide spectrum of learning scenarios and application domains.
Cross-episodic learning. Cross-episodic learning is a less-explored terrain despite that it has been discussed together with meta-RL [77] for a long time.  $\mathsf{RL}^2$  [18] uses recurrent neural networks for online meta-RL by optimizing multi-episodic value functions. Meta-Q-learning [21] instead learns multi-episodic value functions in an offline manner. Algorithm Distillation (AD) [42] and Adaptive Agent (AdA) [1] are two recent, inspiring methods in cross-episodic learning. Though at first glance our learning-progress-based curriculum appears similar to AD, significant differences emerge. Unlike AD, which focuses on in-context improvements at test time and requires numerous single-task source agents for data generation, our approach improves data efficiency for Transformer agents by structuring data in curricula, requiring only a single multi-task agent and allowing for diverse task instances during evaluations. Meanwhile, AdA, although using cross-episodic attention with a Transformer backbone, is rooted in online RL within a proprietary environment. In contrast, we focus on offline behavior cloning in accessible, open-source environments, also extending to IL scenarios unexplored by other meta-learning techniques. Complementary to this, another recent study [43] provides theoretical insight into cross-episodic learning.
Curriculum learning. Curriculum learning represents training strategies that organize learning samples in meaningful orders to facilitate learning [7]. It has been proven effective in numerous works that adaptively select simpler task [58, 74, 69, 62, 15, 55, 59, 46] or auxiliary rewards[35, 72]. Tasks are also parameterized to form curricula by manipulating goals [24, 30, 66], environment layouts[79, 3, 64], and reward functions [28, 34]. Inspired by this paradigm, our work harnesses the improving nature of sequential experiences to boost learning efficiency and generalization for embodied tasks.
In this work, we introduce a new learning algorithm named Cross-Episodic Curriculum to enhance the sample efficiency of policy learning and generalization capability of Transformer agents. It leverages the shifting distributions of past learning experiences or human demonstrations when they are viewed as curricula. Combined with cross-episodic attention, CEC yields embodied policies that attain high performance and robust generalization across distinct and representative RL and IL settings. CEC represents a solid step toward sample-efficient policy learning and is promising for data-scarce problems and real-world domains.
Limitations and future work. The CEC algorithm relies on the accurate formulation of curricular sequences that capture the improving nature of multiple experiences. However, defining these sequences accurately can be challenging, especially when dealing with complex environments or tasks. Incorrect or suboptimal formulations of these sequences could negatively impact the algorithm's effectiveness and the overall learning efficiency of the agents. A thorough exploration regarding the attainability of curricular data is elaborated upon in Appendix, Sec D.
In subsequent research, the applicability of CEC to real-world tasks, especially where task difficulty remains ambiguous, merits investigation. A deeper assessment of a demonstrator's proficiency trajectory — from initial unfamiliarity to the establishment of muscle memory — could offer a valuable learning signal. Moreover, integrating real-time human feedback to dynamically adjust the curriculum poses an intriguing challenge, potentially enabling CEC to efficiently operate in extended contexts, multi-agent environments, and tangible real-world tasks.
We thank Guanzhi Wang and Annie Xie for helpful discussions. We are grateful to Yifeng Zhu, Zhenyu Jiang, Soroush Nasiriany, Huihan Liu, and Rutav Shah for constructive feedback on an early draft of this paper. We also thank the anonymous reviewers for offering us insightful suggestions and kind encouragement during the review period. This work was partially supported by research funds from Salesforce and JP Morgan.
[80] Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv: Arxiv-2303.04129, 2023.
[81] Songyuan Zhang, ZHANGJIE CAO, Dorsa Sadigh, and Yanan Sui. Confidence-aware imitation learning from demonstrations with varying optimality. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 12340-12350. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/670e8a43b246801ca1eaca97b3e19189-Paper.pdf.
[82] Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta, Dinesh Jayaraman, Yecheng Jason Ma, and Luca Weihs. Universal visual decomposer: Long-horizon manipulation made easy, 2023.
[83] Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. arXiv preprint arXiv: Arxiv-2210.11339, 2022.
A Model Architecture
In this section, we provide comprehensive details about the Transformer model architectures considered in this work. We implement all models in PyTorch [61] and adapt the implementation of Transformer-XL from VPT [4].
A.1 Observation Encoding
Experiments conducted on both DMLab and RoboMimic include RGB image observations. For models trained on DMLab, we use a ConvNet [29] similar to the one used in Espeholt et al. [20]. For models trained on RoboMimic, we follow Mandlekar et al. [53] to use a ResNet-18 network [29] followed by a spatial-softmax layer [23]. We use independent and separate encoders for images taken from the wrist camera and frontal camera. Detailed model parameters are listed in Table A.1.
Table A.1: Model hyperparameters for vision encoders.
Hyperparameter	Value
DMLab
Image Size	72 × 96
Number of ConvNet Blocks	1
Channels per Block	[16, 32, 32]
Output Size	256
RoboMimic
Image Size	84 × 84
Random Crop Height	76
Random Crop Width	76
Number of Randomly Cropped Patches	1
ConvNet Backbone	ResNet-18 [29]
Output Size	64
Spatial-Softmax Number of Keypoints	32
Spatial-Softmax Temperature	1.0
Output Size	64
Since DMLab is highly partially observable, we follow previous work [20, 22, 4] to supply the model with previous action input. We learn 16-dim embedding vectors for all discrete actions.
To encode proprioceptive measurement in RoboMimic, we follow Mandlekar et al. [53] to not apply any learned encoding. Instead, these types of observation are concatenated with image features and passed altogether to the following layers. Note that we do not provide previous action inputs in RoboMimic, since we find doing so would incur significant overfitting.
A.2 Transformer Backbone
We use Transformer-XL [16] as our model backbone, adapted from Baker et al. [4]. Transformer-XL splits long sequences into shorter sub-sequences that reduce the computational cost of attention while allowing the hidden states to be carried across the entire input by attending to previous keys and values. This feature is critical for the long sequence inputs necessary for cross-episodic attention. Detailed model parameters are listed in Table A.2.
A.3 Action Decoding
To decode joystick actions in DMLab tasks, we learn a 3-layer MLP whose output directly parameterizes a categorical distribution. This action head has a hidden dimension of 128 with ReLU activations. The "Goal Maze" and "Irreversible Path" tasks have an action dimension of 7, while "Watermaze" has 15 actions. To decode continuous actions in RoboMimic, we learn a 2-layer MLP that parameterizes a Gaussian Mixture Model (GMM) with 5 modes that generates a 7-dimensional action. This network
Table A.2: Model hyperparameters for Transformer-XL.
Hyperparameter	Value (DMLab)	Value (RoboMimic)
Hidden Size	256	400
Number of Layers	4	2
Number of Heads	8	8
Pointwise Ratio	4	4
has a hidden dimension of 400 with ReLU activations. During deployment, we employ the "low-noise evaluation" trick [31].
B Training Details and Hyperparameters
All experiments are conducted on cluster nodes with NVIDIA V100 GPUs. We utilize DDP (distributed data parallel) to accelerate the training if necessary. Training hyperparameters are listed in Table A.3.
Table A.3: Hyperparameters used during training.
Hyperparameter	Value (DMLab)	Value (RoboMimic)
Learning Rate	0.0005	0.0001
Warmup Steps	1000	0
LR Cosine Annealing Steps	100000	N/A
Weight Decay	0.0	0.0
C.1 DMLab Main Experiment
Our DMLab main experiment is conducted on three levels with task IDs
- explore_goal Locations_large,
- rooms_watermaze,
- and skymaze_irreversible_path-hard.
We use no action repeats during training and evaluation. For experiments with varying task difficulty, we select difficulty parameters "room numbers", "spawn radius", and "built-in difficulty" for these three levels, respectively. We adopt environment wrappers and helper functions from Petrenko et al. [63] to flexibly and precisely maneuver task difficulties.
Due to different task horizons, we tune the context length of Transformer-XL models and vary curricular trajectories accordingly. These differences are summarized in Table A.4.
RL oracles serve as source agents used to generate training data for our methods and the "BC w/ Expert Data" baseline. They are trained with the PPO [70] implementation from Petrenko et al. [63]. The "BC w/ Expert Data" baselines have the same model architecture, training hyperparameters, and amount of training data as our method, but are trained solely on trajectories generated by the best performing RL oracles without cross-episodic attention.
C.2 DMLab Generalization
This series of experiments probe the zero-shot generalization capabilities of embodied agents in unseen maze configurations, out-of-distribution difficulty levels, and varying environment dynamics. For the task "Goal Maze w/ Unseen Mechanism", we use the level with task ID
Table A.4: Experiment details on DMLab tasks. Columns "Epoch" denote the exact training epochs with best validation performance. We select these checkpoints for evaluation. For task-difficulty-based curriculum, the column "Training Trajectories" with  $n \times m$  entries means  $n$  trajectories per difficulty level ( $m$  levels in total). The column "Sampled Episodes" with  $[i,j]$  entries means we first determine the number of episodes per difficulty level by uniformly sampling an integer from  $[i,j]$  (inclusively).
Level Name	Context Length	Task-Difficulty-Based Curriculum	Learning-Progress-Based Curriculum
Epoch	Training Trajectories	Sampled Episodes	Epoch	Training Trajectories	Sampled Episodes
Goal Maze	500	84	100 x 3	[1, 5]	88	300	9
Watermaze	400	89	100 x 3	[1, 5]	80	300	9
Irreversible Path	1600	90	100 x 4	[1, 3]	97	400	8
Table A.5: Evaluation results on DMLab, averaged over three tasks (Figure 3).
Ours (Task Difficulty), Auto	Ours (Task Difficulty), Fixed	Ours (Learning Progress)	DT (Mixed Difficulty)	DT (Single Difficulty)	AT (Mixed Difficulty)	AT (Single Difficulty)	BC w/ Expert Data	RL (Oracle)	Curriculum RL (Oracle)
51.4	54.4	32.4	35.3	11.7	42.7	33.4	14.2	40.6	50.6
Table A.6: Generalization results on DMLab, averaged over five settings (Figure 4).
Ours (Task Difficulty)	Ours (Learning Progress)	DT (Mixed Difficulty)	DT (Single Difficulty)	AT (Mixed Difficulty)	AT (Single Difficulty)	BC w/ Expert Data	RL (Oracle)	Curriculum RL (Oracle)
39.6	27.8	31.8	13.6	39.4	29.2	18.1	30.0	37.6
explore_abstructed_goals_large, which adds randomly opened and closed doors into the maze while ensuring a valid path to the goal always exists. An example of an agent's ego-centric observation is visualized in Figure A.1.
The task "Irreversible Path (OOD. Difficulty)" corresponds to configurations with the built-in difficulty of 1 (agents are only trained on difficulty up to 0.9, as noted in Table 1). For tasks with varying environment dynamics, we directly test agents with an action repeat of 2. This is different from the training setting with no action repeat.
C.3 RoboMimic Main Experiment
We leverage the Multi-Human (MH) dataset from Mandlekar et al. [53]. It consists of demonstrations collected by operators with varying proficiency. We construct the expertise-based curriculum by following the order of "worse operators, okay operators, then better operators". We use a context length of 200 for both tasks. There are 90 trajectories per expertise level. To determine the number of trajectories per expertise level when constructing curricular data, we uniformly sample an integer from [1, 5] (inclusively). The "Lift" and "Can" tasks are solved after training for 33 epochs and 179 epochs, respectively. We control for the same number of training epochs in subsequent ablation studies.
C.4 Ablation Study on Curriculum Granularity
We perform this ablation with the task-difficulty-based curriculum on DMLab levels due to the ease of adjusting granularity. The definition of varying levels of curriculum coarseness is listed in Table A.7.