SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table A.7: Definitions of varying levels of curriculum coarseness.
Level Name	Difficulty Parameter	Test Difficulty	Fine	Medium	Coarse
Goal Maze	Room Numbers	20	5→10→15	5→10	5→15
Watermaze	Spawn Radius	580	150→300→450	150→300	150→450
Irreversible Path	Built-In Difficulty	0.9	.1→.3→.5→.7	.1→.5→.7	.1→.3→.5
Figure A.1: A visualization of the task "Goal Maze (Unseen Mechanism)". It includes doors that are randomly opened or closed.
Table A.8: Results show the performance of different curricula on two robotic manipulation tasks: Lift and Can. Standard deviations are included.
Task	Expertise-Based Curriculum	Learning-Progress-Based Curriculum	CQL [41]
Lift	100.0 ± 0.0	32.0 ± 17.0	2.7 ± 0.9
Can	100.0 ± 0.0	30.0 ± 2.8	0.0 ± 0.0
Average	100.0	31.0	1.4
C.5 Comparison of Curricula in RoboMimic
In IL settings, we further explored the efficacy of various curricula. For the RoboMimic tasks examined, we employed a learning-progress-based curriculum, ensuring the total training trajectories matched those of the expertise-based curriculum (i.e., 270 trajectories per task). All other parameters remained consistent, with the training data derived from RoboMimic's machine-generated dataset.
Table A.8 indicates that when heterogeneous-quality human demonstrations are accessible, the expertise-based curriculum is preferable due to its superior performance over the learning-progress-based approach. Conversely, without expert demonstrations and relying solely on machine-generated data, the learning-progress-based curriculum is still commendable. It offers noteworthy results and surpasses offline RL methods like CQL [41], even though CQL is trained on the full RoboMimic dataset, encompassing 1500 trajectories for the Lift task and 3900 for the Can task.
D Feasibility of Obtaining Curricular Data
The challenge of accurately orchestrating a curriculum is non-trivial and hinges on various factors. In the present work, three curriculum designs are introduced and validated, each with its practical considerations and underlying assumptions, discussed herein.
Learning-Progress-Based Curriculum. RL agents typically exhibit monotonic improvement over training epochs, thereby naturally producing incrementally better data. The curriculum here is devised through a series of checkpoints throughout the training duration, necessitating no supplementary assumptions for its formulation.
Task-Difficulty-Based Curriculum. In contexts where environmental difficulty is parameterizable, curricula can be structured through a schedule, determined by the relevant difficulty parameter, as demonstrated within this work. In scenarios lacking parameterized difficulty, alternatives such as methods proposed by Kanitscheider et al. [40] may be employed. The application of our method to tasks where difficulty is not explicitly characterized presents an intriguing avenue for future research.
Expertise-Based Curriculum. A notable limitation resides in the requisite to estimate demonstrators' proficiency. While some IL benchmarks, e.g., RoboMimic [53], come pre-equipped with proficiency labels, a broader application of our method necessitates an approximation of proficiency. One plausible approach entails ranking trajectories via completion time. Furthermore,
a demonstrator's proficiency is likely to organically improve—from initial unfamiliarity with teleoperation systems or tasks, to a stage of executing data collection with muscle memory [52]. This progression potentially provides a rich learning signal conducive for CEC application.
Our Cross-Episodic Curriculum can significantly enhance Transformer agent learning but carries potential societal impacts. The efficiency of our method depends on the curriculum's design. If the curriculum unintentionally reflects biases, it could lead to the amplification of these biases in learned policies, potentially perpetuating unfair or discriminatory outcomes in AI-driven decisions. Furthermore, the computational intensity of our approach at evaluation could contribute to increased energy usage, which has implications for the environmental footprint of AI applications.