Heterogeneous Adversarial Play in Interactive Environments
Authors: Manjie Xu
Xinyi Yang
Jiayu Zhan
Wei Liang
Chi Zhang
Yixin Zhu
Venue: NeurIPS 2025 poster ()
Keywords: self-play
curriculum learning
interative reasoning
Primary area: general_machine_learning

Abstract: Self-play constitutes a fundamental paradigm for autonomous skill acquisition, whereby agents iteratively enhance their capabilities through self-directed environmental exploration. Conventional self-play frameworks exploit agent symmetry within zero-sum competitive settings, yet this approach proves inadequate for open-ended learning scenarios characterized by inherent asymmetry. Human pedagogical systems exemplify asymmetric instructional frameworks wherein educators systematically construct challenges calibrated to individual learners' developmental trajectories. The principal challenge resides in operationalizing these asymmetric, adaptive pedagogical mechanisms within artificial systems capable of autonomously synthesizing appropriate curricula without predetermined task hierarchies. Here we present Heterogeneous Adversarial Play (HAP), an adversarial Automatic Curriculum Learning framework that formalizes teacher-student interactions as a minimax optimization wherein task-generating instructor and problem-solving learner co-evolve through adversarial dynamics. In contrast to prevailing automatic curriculum learning methodologies that employ static curricula or unidirectional task selection mechanisms, HAP establishes a bidirectional feedback system wherein instructors continuously recalibrate task complexity in response to real-time learner performance metrics. Experimental validation across multi-task learning domains demonstrates that our framework achieves performance parity with SOTA baselines while generating curricula that enhance learning efficacy in both artificial agents and human subjects.

Heterogeneous Adversarial Play in Interactive Environments
Manjie Xu 1.2,4,5
Xinyi Yang 1.2,4,5
Jiayu Zhan 2.1,4.5
mjzu250stu, pku. edu
xinyiyang25@stu, pku. edu. jiayu. zhan@pku edu
Wei Liang 340
Chi Zhang 1,04
Yixin Zhu 2,1,4,,,7,04
liangwei@hit.edu.co wellyzhangc@gmail.ccm
zwa.in..uupkuueeeo.m edu. en
Institute for Artificial Intelligence, Peking University
School Psychological and Cognitive Sciences, Peking University
3 School of Computer Science & Technology, Beijing Institute of Technology
State Key Lab of General AI, Peking University
Beijjin Key Laboratory of Behavior and Mental Health, Peking University
Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing
7 Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence
sttp::///iteesggoggee. www/vviwwhapp-eerraeeeee
Abstract
Self-play constitutes a fundamental paradigm for autonomous skill acquisition, whereby agents iteratively enhance their capabilities through self-directed envi- ronmental exploration (Silver et al., 2018). Conventional self-play frameworks exploit agent symmetry within zero-sum competitive settings (Balduzzi al., 2019), yet this approach proves inadequate for open-ended learning scenarios characterized by inherent asymmetry. Human pedagogical systems exemplify asymmetric instructional frameworks wherein educators systematically construct challenges calibrated individual learners' developmental trajectories (Bobbitt, 1918; Bengio et al., 2009). The principal challenge resides in operationalizing these asymmetric, adaptive pedagogical mechanisms within artificial systems ca- pable of autonomously synthesizing appropriate curricula without predetermined task hierarchies. Here we present Heterogeneous Adversarial Play (HAP), an adversarial Automatic Curriculum Learning (ACL) framework that formalizes teacher-student interactions as minimax optimization wherein task-generating instructor and problem-solving learner co-evolve through adversarial dynamics. contrast to prevailing ACL methodologies that employ static curricula or uni- directional task selection mechanisms, HAP establishes bidirectional feedback system wherein instructors continuously recalibrate task complexity in response to real-time learner performance metrics. Experimental validation across multi-task learning domains demonstrates that our framework achieves performance parity with state-of-the of-te-of-the-art -thh--art (SOTA) baselines while generating curricula that enhance learning efficacy both artificial agents and human subjects.
1 Introduction
The ability to incrementally acquire and consolidate knowledge via environmental interactions progressing from foundational concepts to sophisticated sxpertise-conssitusse defining character- istic of human intelligence (Elman, 1993; Rohde and Plaut, 1999; Bengio et al., 2009). Curriculum Learning (CL), as a structured pedagogical paradigm, enables humans decompose complex tasks into manageable milestones, fostering robust comprehension and skill mastery (Bengio et al., 2009; Prideaux et al., 2003; Zhang al., 2024c,d). Inspired by this biological precedent, machine learning researchers have endeavored to emulate progressive learning strategies for artificial agents, partic-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Automatic Curriculum Learning TTeacher-Studentt
(b) Heterogeenes Adversarial Play (Ours)
Learning Feedbock
Heuristic
OTT
HAP Teacher
Teachhr
Learning Feedback
Tearher Boward
Snsdent Jeward
teacher
Switching Polkay
Student
(oracle expert
ENV
A
Student
solvinng rate
Studknt Lasin Record 1l Aap .R, [يد
Obs
ENV Touchhr
Taok
TT
student
Action
TT
Action
Studinnt Stadknt Boaard
Seacter Suever
stucinnt
Training
salving
Reward
Selected Task
Student
rate
Designed Qurriculum
ENV
Reward
ENV
Figure 1: Comparison of different learning frameworks. (a) Automatic Curriculum Learning (ACL): heuristic teacher selects tasks from a predefined curriculum pool and provides feedback to guide student learning through environments of increasing complexity. The teacher relies on domain expertise and rulc- based policies to sequence tasks appropriately. (b) Heterogencous Adversarial Play (HAP);: Our framework extends ACL through adversarial co-evolution. The teacher learns generate challenging but solvable tasks that maximize student learning, while the student adapts solve the teacher's 's evolving problem proposals.
ularly in domains requiring long-horizon reasoning or multi- proficiency (Pentina et al., 2015; Lowe et al., 2017; Narvekar and Stone, 2019; Yarats et al., 2022; Xu et al., 2023a). However, translating human pedagogical principles to Artificial Intelligence (AI) systems remains challenging due to fundamental differences in how humans and machines internalize and generalize knowledge (Khan al., 2011; Jiang et al., 2015; Vinyals et al., 2019; Jiang et al., 2023).
Traditional CL frameworks employ static, human- designd curricula predicated on predefined task difficulty hierarchies (Graves al., 2017; Narvekar et al., 2017; Chen et al., 2021), assuming that sequences ordered by increasing complexity universally optimize learning. This paradigm suffers from two critical limitations: (i) learners' internal states remain largely unobservable, and (ii) cur- ricula cannot adapt dynamically to evolving capabilities. Consequently, agents may stagnate when encountering tasks that are either insufficiently challenging or prohibitively difficult, impeding ex- ploration and convergence (Wang et al., 2021; Xu et al., 2023b). Current Automatic Curriculum Learning (ACL) methods partially ameliorate these issues by dynamically selecting tasks based on heuristic metrics (e.g., success rates or loss functions) (Zhang et al., 2020; Kim et al., 2021; Portelas al., 2021; Seo et al., 2022; Yang et al., 2024; Wu et al., 2024). However, these approaches typ- ically operate unidirectionally, emphasizing either task generation or difficulty assessment without establishing cohesive feedback mechanisms between these components.
Cognitive science research reveals fundamental principles underlying effective curriculum design that current automated methods largely disregard. Optimal curricula necessitate individualized and adaptive task selection, embodying "hypothesis space navigation" -the systematic exploration of knowledge structures guided by learners' evolving comprehension (Tenenbaum al., 2011). Effec- tive instruction requires dynamic updating based on conceptual understanding models, analogous to how human educators continuously refine their mental representations of students' knowledge states (Baker et al., 2017; Chu and Schulz, 2019). Moreover, successful learning depends on bidirectional feedback loops wherein task generation and performance evaluation reciprocally inform each other (Zhu et al., 2020, 2023). These principles collectively suggest establishing productive tension be- tween task generators that systematically challenge learners and students that continuously adapt, creating balanced adversarial dynamics characteristic of effective human developmental learning (Lake ct al., 2017; Tenenbaum, 2020).
Unlike traditional self-play, which requires perfect agent symmetry (Sukhbaatar et al., 2018; Racaniere al., 2020; Dennis al., 2020), the teacher-student interaction naturally accommodates heterogeneous roles within an adversarial framework. We formalize this relationship as a zero-sum game wherein task generators receive rewards when problem solvers fail, while problem solvers are rewarded for successfully addressing proposed challenges. Building upon this insight, we introduce Heterogeneous Adversarial Play (HAP), an adversarial learning framework that operationalizes the challenge-response dynamics fundamental to human learning. HAP employs dual-network ar- chitecture wherein the teacher network generates tasks calibrated to challenge student capabilities, while the student network strives to master these evolving challenges. This adversarial equilib-

---

rium produces curricula that dynamically balance task complexity against learners' developing pro- ficiency, and facilitates robust knowledge consolidation as well as effective exploration.
Experimental validation across multi-task learning environments of increasing complexity demon- strates HAP's efficacy. In grid navigation domains, the framework exhibits autonomous adaptive behavior: teachers escalate task difficulty as students improve while reverting to foundational chal- lenges when progress stagnates. In complex Minecraft-inspired environments featuring hierarchical task dependencies (Johnson et al., 2016; Hafner, 2022; Fan et al., 2022), HAP surpasses SOTA base- lines in both completion rates and learning efficiency. Human studies confirm that -e--geeeee curricula mirror effective pedagogical strategies, including strategic skill reinforcement and adap- tive difficulty scaling. These findings indicate that adversarial co-adaptation enables robust learning in Al systems without requiring handerafted curricula, suggesting that adversarial optimization dis- covers fundamental instructional principles shared across artificial and human learning systems.
Our contributions are threefold: (i) theoretical framework grounding HAP in adversarial optimiza- tion that formalizes pedagogical interactions as minimax games, (ii) empirical validation demon- strating superior performance and efficiency compared to existing baselines across complex multi- task environments, and (iii) insights revealing alignment between human pedagogical principles, establishing adversarial co-adaptation as principled bridge between symmetric self-play and asymmetric curriculum learning.
curricula and
2 Related Work
Self-Play and Adversarial Training Self-play has emerged as a transformative paradigm for training agents in competitive environments, whereby agents iteratively improve through interac- tions with evolving variants of themselves (Silver al., 2016; Vinyals al., 2019; Baker et al., 2019). This methodology naturally engenders curriculum formation, as progressively sophisticated agents generate increasingly challenging opponents. However, traditional self-play presupposes symmetric roles and objectives, constraining its applicability to domains requiring fundamentally heterogeneous agent capabilities (Eccles et al., 2019; Christianos et al., 2020). Recent investigations have explored asymmetric self-play configurations wherein agents assume distinct yet structurally analogous roles (Baker et al., 2019; Eccles et al., 2019; Zhang et al., 2024b). Adversarial train- ing extends these principles by explicitly formulating interactions zero-sum games, thereby en- hancing robustness and generalization capabilities (Goodfellow et al., 2014; Ho and Ermon, 2016). Complementary approaches, including domain randomization and adversarial domain adaptation, employ adversarial objectives to facilitate transfer learning (Tzeng et al., 2017; Ganin et al., 2016; Pinto al., 2017). While inspired by these adversarial dynamics, HAP framework specifically addresses the inherent asymmetry hetween pedagogical and learning functions, transcending the symmetric constraints of conventional self-play methodologies.
Multi- Learning and Transfer Multi-task learning seeks to enhance generalization through simultaneous acquisition of multiple related tasks, exploiting shared representations and inter-task knowledge transfer (Caruana, 1997; Ruder, 2017; Crawshaw, 2020). Traditional methodologies as- sume statie task distributions and emphasize architectural or regularization strategies to promote knowledge sharing (Misra et al., 2016; Standley et al., 2020). These approaches frequently en- counter difficulties when task complexities vary substantially or when particular tasks dominate training regimes, precipitating negative transfer phenomena (Wang et al., 2019b; Fifty et al., 2021). Contemporary advances address these limitations through ACL techniques, encompassing adaptive task weighting (Chen et al., 2018; Liu et al., 2019; Yang et al., 2023), leward-hased transitions (Narvekar and Stone, 2019; Chen et al., 2021: Polder-Holder et al., 2022: Li al., 2024), and curriculum strategies governing task exposure sequences (Dennis et al., 2020; Kong et al., 2021; Soviany et al., 2022; Yang et al., 2023; Diaz et al., 2024). These ACL methodologies collectively aim to enhance sample efficiency and ultimate performance through active training guidance while enabling mastery of complex multi-goal tasks via systematic progression from elementary to ad- vanced subtasks (Narvekar et al., 2020; Hekimoglu et al., 2023; Zhang et al., 2020; Forestier et al., 2022). Our approach builds upon this foundation (Florensa et al., 2018; Matiisen et al., 2019; Kong et al., 2021; Jiang et al., 2021) by introducing a principled mechanism for autonomous task gen- eration and sequencing based on learner progression, simultaneously addressing task selection and difficulty calibration challenges.
3

---

Meta-Learning and Learning to Learn Meta-learning, or "learning to learn," endeavors to de- velop algorithms capable of rapid adaptation to novel tasks through exploitation of prior experi- ence (Thrun and Pratt, 1998; Finn et al., 2017; Zhang et al., 2019). Model-agnostic meta-learning (MAML) and related variants accomplish this objective by learning initialization parameters that facilitate swift adaptation via gradient descent (Finn et al., 2017; Nichol et al., 2018; Antoniou et al., 2019). Alternative memory-based approaches utilize external memory systems or recurrent architec- tures to store and retrieve pertinent experiences (Santoro et al., 2016; Zhao et al., 2021; Genewein et al., 2023; Xu ct al., 2025). Recent research has investigated automated curriculum generation within meta-learning contexts, wherein meta-learners identify optimal task sequences for efficient few-shot learning (Khodadadeh et al., 2019; Wang et al., 2020; Zhang et al., 2024a). Teacher-student distillation provides an additional relevant framework wherein teacher networks guide student learn- ing through soft targets or intermediate representations (Hinton et al., 2015; Zagoruyko and Ko- modakis, 2017; Sengupta al., 2023). While such methodologies focus knowledge transfer from pre-trained instructors, HAP facilitates dynamic co-evolution wherein teachers continuously adapt generate appropriate challenges while students concurrently develop solution capabilities, yielding more flexible and responsive learning process.
3 The Heterogeneous Adversarial Play (HAP)
We formulate HAP as an adversarial optimization framework wherein a teacher agent autonomously generates challenging tasks to accelerate student learning through strategic curriculum adapta- tion. This asymmetric adversarial paradigm extends traditional self-play methodologies (Sukhbaatar et al., 2018) by accommodating heterogeneous agent roles with distinct capabilities and opposing objectives, thereby transcending the symmetric constraints inherent in conventional approaches.
HAP addresses fundamental limitations of manually designed curricula through continuous adaptive mechanisms. Rather than employing static task sequences that may inadequately align with student developmental trajectories, HAP implements a dynamic feedback system that modulates task dif- ficulty in response to evolving student capabilities. Following the generative adversarial paradigm (Goodfellow et al., 2014), we model this interaction as a minimax game wherein the teacher gen- crates progressively demanding challenges while the student systematically masters each proposed task, establishing self-regulating curriculum that scales organically with learning progress.
3.1 Framework Description
We initiate our exposition with a discrete task formulation for conceptual clarity, noting that contin- uous extensions follow naturally. Consider a structured task space T {2i, T2, ..., Iw wherein each task Т; exhibits varying complexity levels and potential interdependencies encoded within a directed acyclic graph G (T, E). Within this environment, teacher and student agents engage in adversarial co-evolution through complementary yet opposing roles.
Student Agent The student agent seeks maximize expected performance across tasks sampled from the teacher's adaptive distribution. Formally, the student optimizes policy parameters ac- cording to:
max Jatudem (0)
(0) = ET (T) ( [[(:
(1)
where Pa (T) represents the teacher's task selection distribution parameterized by Φ, 7 denotes trajec- tories generated by the student policy TT-12';0) 0), and R(T:T)- = >n1 n'T, constitutes the discounted cumulative reward for task T.
Teacher Agent The teacher agent evaluates student progress and strategically modulates task se- lection to maintain optimal challenge levels. We implement the teacher as a neural network that processes the student's behavioral history he {Ti, T2, ... Tel and outputs task selection proba- (T.AA))
bilities through softmax normalization: (T;T)( ΣN exp (T)) where Lø represents the
teacher's scoring function. The teacher's objective directly opposes the student's performance, es- tablishing a zero-sum adversarial dynamic:
max Jreacher (6) = Er [E -- [-I(T;
(2)

---

Algorithm 1: Training loop of the Heterogencous Adversarial Play (HAP)
Data: Initial 0, d; learning rates a, /
while not converged do
/* Step 1. Teacher's Adversarial Task Selection:
Generate task distribution: Pa exp(@))))())))
:/* Minimization strategy: Sample task T Po(T) to challenge current т */
: :/* Step 2. Student's Policy Maximization:
Execute Tals,, ('; 0). collect trajectory T:
Compute reward signal: [[;; T) -
Update 8 to maximize returns:;
0 - 0 + VVE,, [R(T;
:/* Step 3. Teacher's s Adversarial Update:
Update á to minimize student success:;
Φ VV T)]; Er (VologPo(T) R(T;
where VoJiecchrr
I1 end
This adversarial formulation ensures that the teacher continuously recalibrates task difficulty in response to student capability evolution, thereby maintaining appropriate pedagogical challenge throughout the learning process.
3.2 Adversarial Formulation
The teacher-student interaction constitutes a minimax optimization problem that enables tractable implementation through alternating gradient- gradient-based updates:
min max J(0,,) Φ).
(3)
wherein the student agent maximizes expected task performance through policy optimization while the teacher agent minimizes student success by strategically selecting challenging tasks that neces- sitate continued adaptation.
While the student may employ any differentiable policy optimization method (e.g., PPO. SAC), the teacher must adapt its parameters ò to systematically diminish student expected returns. Applying the policy gradient theorem to the teacher's 's task selection policy ps (h))), we derive the gradient with respect to @ as:
(4) This gradient formulation enables the teacher to increase the probability of selecting tasks where the student performs poorly while decreasing selection probability for tasks where the student ex- cels. The training procedure alternates between teacher task generation, student policy execution on selected tasks, and adversarial parameter updates for both agents, detailed in Algorithm
V Juacher)))) = -p V logpo(T) Ey T:0) [R(;;T]]]-
Crucially, this adversarial framework accommodates inherently asymmetric agent teachereachers and students possess fundamentally different capabilities, objectives, and network architectures- distinguishing HAP from traditional self-play methodologies that require agent symmetry. This asymmetric adversarial paradigm enables principled curriculum adaptation without predetermined task hierarchies, as the teacher continuously discovers optimal challenge sequences through ad- versarial optimization against the evolving student. Extended implementation details, including stabilization techniques and convergence analysis, are provided in Section B.
3.3
Implementation Details
Cold Start Problem Adversarial training encounters a fundamental bootstrapping challenge dur- ing initialization. The teacher possesses no prior knowledge regarding task difficulty or student ca- pabilities, while the student begins with randomly initialized parameters yielding poor performance across all tasks. This information deficit creates an unreliable feedback mechanism that significantly impedes initial optimization progress.
We resolve this issue through a structured warm-up protocol. The student independently explores each task for predetermined duration without teacher involvement, enabling both agents to acquire

---

Training Curres
146 (oum)
DONiopp, www.d.0 DONiAqptreerd.c
DON ceq, Tep
SAC
ADC
Bvocoss Rats
0.5
Selection Probabiiity
Maln
Probakdby
0.2-
0.0
04
2.5
20cc
7:2c
1.02% 122kk
1003c
1750k
200%
Training Steps
Stepe
Figure 2: Adversarial curriculum dynamics in navigation benchmark. (a) Training reward curves demon- strate that HAP achieves faster convergence and higher overall performance compared baseline approaches, reaching optimal performance around 35k steps. (b) Task-specific success rates reveal that HAP learns all dif- ficulty levels uniformly, while baselines exhibit pronounced performance gaps between casier and harder tasks throughout training. (c) Underlying adversarial mechanism: solid lines show task success rates during eval- uation, while dashed lines indicate corresponding task sampling prohabilities from the teacher's perspective, illustrating the positive feedback loop (increased sampling for failed tasks) and negative feedback mechanism (reduced sampling for mastered tasks) that drive HAP's effectiveness.
essential baseline information. During this phase, the student develops rudimentary competencies while the teacher observes relative task difficulties and establishes initial performance benchmarks.
Underfitting from Task Overload When students attempt simultaneous learning across excessive numbers of tasks, task underfitting emerges due to insufficient attention allocation to individual challenges. This phenomenon occurs under two primary conditions: during early training when teachers maintain uniform task distributions lacking prior knowledge, and when curricula encompass numerous challenging tasks that exceed student learning capacity.
We mitigate task overload through entropy regularization of the teacher's objective:
Jrescher ()) (T [E ( (T0) [- R(;; T)] + A HTp)(T)),
(5)
where (T)) denotes task distribution entropy and X modulates regularization intensity. This modification encourages teachers to focus student attention on manageable task subsets while main- taining appropriate exploration diversity.
Catastrophic Forgetting Adversarial optimization naturally diminishes selection probability for tasks exhibiting high student success rates, redirecting focus toward more challenging objectives. While this adaptive mechanism promotes continuous learning, it may precipitate catastrophic for- getting (Kirkpatrick et al., 2017; Hadsell al., 2020) wherein students lose proficiency on previously mastered tasks due to insufficient practice.
Although teachers could theoretically modulate selection policies within the adversarial framework to preserve task diversity, such adjustments frequently introduce training oscillations and conver- gence inefficiencies. We implement more practical solution by enforeing probabilistic lower bounds on task selection, ensuring minimum exposure that prevents any task probability from ap- proaching zero while maintaining overall curriculum balance.
3.4 Preliminary Experiments
We demonstrate HAP's fundamental advantages through controlled navigation experiment de- signed to elucidate core adversarial dynamics. This environment requires agents to navigate from designated start positions to specified destinations using symbolic map inputs. The experimental framework encompasses four hierarchically ordered tasks locks-simple, mid, hard, and extremely hard- differentiated by navigation sequence length while maintaining task independence. Our teacher implementation employs learnable logits for each task, which are transformed into categorical dis- tributions for task sampling. We evaluate HAP against established baselines using identical network architectures, conducting all experiments on single NVIDIA A100 GPU. Detailed implementation specifications for both baselines and HAP are provided in Section C.

---

The reward trajectories in Figure 2(a) demonstrate that HAP achieves superior convergence effi- ciency, reaching optimal performance within approximately 35k training steps while sustaining the highest cumulative rewards throughout the learning process. While the environment's ' relative sim- plicity eventually enables most agents to master all tasks, TD3's convergence failure illustrates that even controlled experimental conditions pose substantial challenges for conventional Reinforcement Learning (RL) methodologies.
Task-specific success rates across training epochs, presented in Figure 2(b), reveal pronounced per- formance disparities between simpler and more challenging tasks under baseline approaches. This differential stems from extensive training on easier tasks leading to overfitting phenomena that sub- sequently impede progress on harder challenges, while task transitions precipitate catastrophic for- getting of previously acquired competencies. Conversely, HAP rapidly achieves proficiency across all tasks without exhibiting these detrimental learning pathologies.
The underlying adversarial mechanism becomes evident through Figure 2(c)'s detailed analysis of teacher-student dynamics. Solid lines denote task success rates during evaluation phases, while dashed lines indicate corresponding sampling probabilities from the teacher's selection policy. As formalized in Section 3.1, HAP operates through two synergistic feedback mechanisms: a posi- tive reinforcement loop wherein teachers increase sampling probability for frequently failed tasks, thereby accelerating targeted skill acquisition, and a negative regulation mechanism that reduces sampling probability for mastered tasks, preventing redundant practice sessions.
These results underscore the fundamental significance of adversarial curriculum design in multi-task learning scenarios. Even within environments sufficiently tractable for eventual agent success with- out sophisticated intervention, dynamic task assignment regulation produces substantially smoother learning trajectories while mitigating overfitting and catastrophic forgetting. Through continuous sampling probability adjustment based on student progression, the teacher policy efficiently acceler- ates learning on challenging tasks while circumventing excessive repetition of mastered competen- cies, demonstrating that adversarial curriculum adaptation promotes both accelerated convergence and enhanced stability across diverse task distributions.
4 Experiments
We evaluate HAP's scalability across increasingly complex task distributions, encompassing open- world scenarios and environments featuring intricate task dependencies and interconnections.
4.1 Experimental Settings
Playground Our evaluation framework encompasses three distinct environments that span varying complexity levels and task structural characteristics. Minigrid (Chevalier-Boiss et al., 2019) pro- vides a highly configurable grid-world platform well-suited for examining fundamental task struc- tures and validating core adversarial mechanisms. CRAFT (Andreas et al., 2017) represents a clas- sic multi-task RL environment inspired by Minecraft, incorporating hierarchical crafting dependen- cies that necessitate systematic skill progression across interdependent subtasks. Crafter (Hafner, 2022) introduces ppen-world elements and stochastic dynamics, presenting substantial challenges that closely approximate real-world learning scenarios. Typical environment layouts and represen- tative tasks are illustrated in Figure A.. Our task selection methodology prioritizes progressive difficulty scaling while ensuring meaningful dependencies between constituent subtasks. Detailed environment specifications and comprchensive task descriptions are provided in Section A.
Baselines We benchmark HAP against extensive baseline suite spanning three methodological categories. Standard RL algorithms- Seerithms-inncuudnn DQN (Mnih et al., 2013), A2C (Mnih et al., 2016), PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018), and TD3 (Fujimoto et al., 2018) provide insights into traditional approaches' multi-task handling capabilities. DreamerV3 (Hafner et al., 2023) serves as SOTA world model baseline, enabling assessment against contemporary model-based methodologies.
Curriculum learning approaches include (tacher-Student Curriculum Learning (TSCL) (Matiisen et al., 2019) and EXP3 auto-curriculum (Gajane et al., 2015), alongside a manually designed easy- to-hard curriculum baseline for comparative analysis. Additionally, we establish expert human
7

---

performance benchmarks through 18 trained participants possessing minimum bachelor's 's degree qualifications. Human performance provides an empirical upper bound reference, with participants achieving verified task mastery through rigorous pre-test qualification procedures. Complete base- line implementations and HAP specifications are detailed in Section C.
Table Performance evaluation across multi-task environ- ments with increasing complexity. Task success rates for al- gorithmic approaches compared to human experts (gray column) across Easy (basic skills). Middle (intermediate composition), and Hard (complex reasoning) difficulty levels. General scores are weighted averages across difficulties. HAP achieves superior per- formance on Middle and Hard tasks in Minigrid and CRAFT, with competitive results in Crafter.
4.2 Quantitative Results
Table I establishes HAP's superior performance relative to existing al- gorithmic approaches across the ma- jority of evaluated tasks and en- vironments. In Minigrid environ- ments, HAP achieves a general score of 0.527, surpassing all RL base- lines while attaining 71% of human performance on challenging tasks. CRAFT experiments reveal particu- larly pronounced advantages on com- plex tasks, with HAP scoring 0.31 compared to DreamerV3's 0.27 on hard tasks, yielding a general score of 0.562 that narrows the human- algorithm performance gap by 30% relative to previous SOTA method- ologies.
Algocithms
Eav
Ordered DQN
A2C
PPO SAC TD3
DreamerV3 TSCL
EXP3 HAP Human
Minigrid
Easy
0.95
Middle
0.12
0.21
0.25
0.22
0.27
0.26
0.34
0.21
0.24
0.46
0.78
Hard
0.20
0.00
0.00
0.00
0.13
0.08
0.18
0.16
0.18
0.20
0.46
General
0.33
0.407
0.397
0.367
0.457
0.43
0.493
0.443
0.463
0.527
0.747
CRAFT
Easy
0.84
0.86
0.34
Middle
0.21
0.26
0.45
0.48
0.42
0.42
0.55
0.24
0.56
0.43
0.86
Hard
0.25
0.02
0.14
0.12
0.15
0.14
0.27
0.03
0.24
0.31
0.66
General
0.26
0.278
0.415
0.126
0:413
0.407
0.516
0.307
0.513
0.562
0.802
Easy
0.27
0.61
0.79
0.94
0.91
0.84
0.91
0.82
0.91
0.99
Middle
0.16
0.28
0.37
0.67
0.47
0.39
0.66
0.45
0.58
0.68
0.82
Hard
0:14
0.00
0.30
0.47
0.22
0.29
0.52
0.00
0.00
0.58
0.74
General
0.297
0.387
0.533
0.507
0.697
0.723
Crafter environments pose substantial challenges for all algorithmic approaches, wherein even HAP exhibits performance limitations despite consistently outperforming baselines on middle-difiiculty tasks. These results illuminate two fundamental patterns across experimental domains. First, curriculum- based methodologies (HAP, EXP3) systematically surpass standard RL approaches on complex task configurations, demonstrating the critical importance of structured learning progres- sion. Second, all algorithms exhibit pronounced performance degradation with increasing task com- plexity, contrasting sharply with human participants who maintain relatively stable performance across difficulty gradients.
4.3 Qualitative Analysis
Detailed examination of HAP's performance dynamics in Crafter environments (Figure 3) reveals distinctive patterns across task complexity gradients. On elementary tasks, most methodologies achieve comparable performance with consistently high success rates, indicating that conventional learning approaches possess sufficient exploration capabilities for fundamental environment inter- actions. However, pronounced performance disparities emerge as task complexity escalates.
Models
DreamerV2
DreamerV3
(ours)
Human Experts
Score (log scale)
100
0.1
Collect Drink
0.01
Collect Sapling
Collect Wood
Make Word Rolava
Plant
Make Wood Sword
Collect Coal
collect stone
Make stone Pickake
Ear Cow
Make Stone Sword
Place Piant
Place Stone
Place Tabil
Collect Diamond
Collect fron
Defeat Skeleton
Dereat Zombio
Make fron Richaee
Make Iran Sword
place Furrace
Final Score
Wake
Figure 3: Task-specific performance breakdown in Crafter. Average scores across individual achievements calculated from task success rates. HAP performs comparably to DreamerV3 on most tasks while demonstrat- ing superior performance on complex, multi-step challenges requiring hicrarchical skill composition.
8

---

HAP demonstrates substantially elevated success rates on challenging tasks relative to baseline ap- proaches, with advantages becoming particularly evident on complex objectives such as "Defeat Skeleton" and "Make Iron Pickaxe." These tasks necessitate sophisticated skill composition and ex- tended planning horizons, representing quintessential challenges in hierarchical learning domains. The observed performance improvements derive from HAP's adversarial architecture, which auto- matically identifies and proposes prerequisite skills prior to attempting complex task execution.
This systematic prerequisite identification effectively addresses the exploration bottlenecks that fun- damentally constrain standard RL methodologies in hierarchical domains. While conventional ap- proaches struggle with the exponential search spaces inherent in complex skill composition, HAP's stucher-student dynamics naturally decomposes challenging objectives into manageable learning sequences. The adversarial framework thus enables more efficient navigation of task dependency structures, facilitating robust skill acquisition across sophisticated behavioral repertoires.
4.4 Discussion
Our findings illuminate critical insights regarding the current capabilities and limitations of auto- mated curriculum learning methodologies. Despite substantial algorithmic advances, human supe- riority persists on extremely challenging tasks requiring sophisticated planning and compositional reasoning. Our optimal model achieves merely 65% of human performance on demanding Minigrid tasks and 47% on CRAFT environments, underscoring fundamental disparities in how humans and contemporary AI systems approach abstraction and adaptive problem-son solving particularly within compositional reasoning scenarios exemplified by CRAFT's multi-step crafting dependencies.
The effcetivvness of adversarial curriculum design exhibits pronounced anvironment-dependee variation. HAP yields substantial improvements for challenging tasks within Minigrid and CRAFT environments, where task structures demonstrate clear hierarchical organization and well-defined dependency relationships. Conversely, performance gains prove more modest within Crafter's 's open- world configuration, suggesting that while adversarial curricula excel in structured task hierarchies, they may require supplementary mechanisms to support environments demanding autonomous ex- ploration and self-directed learning strategies.
Environmental complexity introduces disproportionate performance degradation for algorithmic approaches relative to human learners. Performance declines precipitously from deterministic Minigrid environments to stochastic, partially observable Crafter scenarios, with certain basclines (TD3/PPO) achieving zero success Crafter's challenging tasks while human participants main- tain consistent overall performance levels. This algorithmic brittleness highlights fundamental lim- itations in handling real-world characteristies including sparse reward and delayed feedback.
Setting: (1FA(CFA
On CIFAR-110-Stndant
On CIFAR 100- 200-immbalanoed--5 50
Accuncy
Chul
Clei
B
B
Dhick
Standard/waay)
Triinnng Epoche
Traiting
Setting: On NLP Task (RTE)
On RTE-Standard
On No-RE-Nooladd..44
Ausunaey
2.6
Standardrossy)
No seahhadd)
Training Epoctr
Toering Epoctes
Figure 4: Supervised curriculum learning performance on CIFAR-100 and RTE benchmarks. HAP con- sistently outperforms established curriculum learning baselines across both computer vision and NLP tasks, demonstrating particular robustness in challenging scenarios with class imbalance (CIFAR-100) and label noise (RTE). Results show superior convergence speed and final performance compared to data selection and loss reweighting approaches. Min-max ranges shown for HAP for visual clarity.
9

---

5 Extended Evaluations
Supervised Curriculum Learning Although HAP was designed for RL, we evaluate its applica- bility supervised learning, where it functions as data-level curriculum algorithm that dynam- ically selects training samples for performance. We assess HAP using the Curbench benchmark (Zhou et al., 2024) across computer vision (CIFAR-100 (Krizhevsky, 2009)) and natural language processing (Recognizing Textual Entailment (RTE) from GLUE (Wang al., 2019a)). We com- pare against established curriculum methods in two categories: data selection approaches (TTCL (Weinshall al., 2018), SPL (Kumar al., 2010)) that prioritize samples by difficulty, and loss reweighting methods (LRE (Ren et al., 2018), ScreenerNet (Kim and Choi, 2018), MW-Net (Shu al., 2019)) that adjust sample importance during training. To stress-test curriculum efficacy, we use challenging conditions: imbalanced class distribution for CIFAR-100 and noisy labels for RTE.
Figure 4 shows HAP achieves competitive performance with SOTA curriculum algorithms across both domains. Under standard conditions, HAP and most baselines approach performance up- per bounds, confirming curriculum benefits are most pronounced in challenging scenarios. In de- manding settings with noise or imbalance, HAP consistently outperforms most baselines, match- ing ScreenerNet faster convergence. These results indicate that adversarial curriculum principles underlying HAP generalize effectively beyond RL to supervised learning domains, supporting broader applicability of teacher-student adversarial frameworks across diverse machine learning paradigms.
and TTCL on RTE--Niised-044 while demonstrating
Human Study To validate HAP-generated cur- ricula, we conducted a human study investigat- ing whether adversarially optimized curricula share qualities of effective human instruction. We re- cruited 30 participants via Prolific and measured learning in Minigrid across three conditions: (i) no tutorial control group establishing baseline perfor- mance, (ii) expert tutorial group receiving step-by- step instruction sequences crafted by Minigrid ex- perts, and (iii -en-geeeeaatd tutorial group expe- riencing dynamically adapted curricula where the framework continuously adjusted tasks based on real-time performance. This design enables direct comparison between adversarial optimization and established pedagogical strategies.
Without lutorial Tutorial
With Auto Tuterial
Standard Error (SE)
Score
3
Range
0
1
6
7
8
Step
Figure Human learning performance across curriculum conditions in Minigrid. Compari- son of human subjects" learning trajectories under three conditions: tutorial, expert step-by-step tutorial, and HAP-generated tutorial, demonstrat- ing the effectiveness of adversarial curriculum sign for human leamerr.
Figure 5 demonstrates that structured curricula sig- nificantly accelerate early-stage skill acquisition. Both expert-designed and HAP-generated tutorials produced similar learning rates and final performance levels, indicating that adversarial optimization successfully discovers effective pedagogical principles. While experts provided superior within-step improvement, HAP offered more individualized curricula responding to perticipant-speciin learning patterns, resulting in faster overall performance gains across curriculum progression.
These findings suggest HAP's adversarial dynamics naturally converge toward instructional strate- gies consistent with effective human teaching practices, including strategic scaffolding and adap- tive difficulty adjustment. The framework's ability autonomously discover optimal instructional sequences without explicit pedagogical programming demonstrates the fundamental connection be- tween adversarial optimization and effective curriculum design.
Conclusion
We propose Heterogeneous Adversarial Play (HAP), an adversarial learning framework where teacher and student models achieve superior task success rates and enhanced responsiveness dur- ing learning plateaus. Human studies demonstrate that HAP's curricula enhance both artificial and human learning without requiring handcrafted instruction sequences. Our work adapts dynamically to learner capabilities while maintaining pedagogical effectiveness comparable to expert instruction.
10

---

Acknowledgments and Disclosure of Funding
This work is supported in part by the National Science and Technology Innovation 2030 Major Pro- gram (20252D02194022) the National Natural Science Foundation of China (32471098), the PKU- BingJi Joint Laboratory for Artificial Intelligence, and the National Comprehensive Experimental Base for Governance of Intelligent Society, Wuhan East Lake High-Tech Development Zone.
References
Andreas, J., Klein, D., and Levine, S. (2017). Modular multitask reinforcement learning with policy sketches. In Proceedings of International Conference on Machine Learning (ICML). 7, A2
Antoniou, A., Edwards, H., and Storkey, A. (2019). How to train your maml. Proceedings of International Conference Learning Representations (ICLR). 4
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., and Mordatch, 1. (2019)- Emergent tool use from multi-agent autocurricula. In Proceedings of International Conference Learning Representations (ICLR).
Baker, C. L., Jara- Eitine,,, J., Saxe, R., and Tenenbaum. J. B. (2017). Rational quantitative attribu- tion of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4):1-10.
Balduzzi, D., Garnelo, M., Bachrach, Y., Czarnecki, W., Perolat, J., Jaderberg, M., and Graepel,
T. (2019). Open-ended learning in symmetric zero-sum games. In International Conference Machine Learning.
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009). Curriculum learning. In Proceedings of International Conference on Machine Learning (ICML).
Bobbitt, F. (1918). The curriculum. Houghton Mifflin.
Caruana, R. (1997). Multitask learning. Machine Learning, 28(1):41-75. 3
Chen, J., Zhang, Y. Xu, Y., Ma, H., Yang, H., Song, J., Wang, Y., and Wu, Y. (2021). Variational au- tomatic curriculum learning for sparse-reward cooperative multi-agent problems. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). 2,3
Chen, Z., Badrinarayanan, V., Lee, C.-Y., and Rabinovich, A. (2018). Gradnorm:
Gradient normal- ization for adaptive loss balancing in deep multitask networks. Proceedings of International Conference on Machine Learning (ICML).
Choveeier-Boisvertt, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. (2019). Babyai: A platform to study the sample efficiency of grounded language learning. In Proceedings International Conference Learning Representations (ICLR). 7, A8
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2023). Palm: Scaling language modeling with pathways. Journal of Machine Learning Research (JMLR), 24(240):1-113, A10
Christiano, F., Leike, J., Brown, T. B., Martic, M., Legg. S., and Amodei, (2017). Deep rein- forcement learning from human preferences. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). A10
Christianos, F., Schäfer, L., and Albrecht, S. (2020). Shared experience actor-critic for multi-agent reinforcement learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS).
Chu, J. and Schulz, L. (2019), Learning who knows what: Children adjust their inquiry to gather information from others. Cognitive Science, 43(2):e12112
Crawshaw, M. (2020). Multi-task learning with deep neural networks: survey. arXiv preprint arXiv:2009.09996.3

---

Dennis, M., Jaques, N., Vinitsky, E.. Bayen, A., Russell, S., Critch, A., and Levine, S. (2020). Emergent complexity and zero-shot transfer via unsupervised environment design. Proceedings of the 34th International Conference on Neural Information Processing Systems. 3
Diaz, M., Paull, L.,, and Tacchetti, A. (2024). Rethinking teacher-student curriculum learning through the cooperative mechanics of experience. Transactions on Machine Learning Research.
Eccles, T., Bachrach, Y., Lever, G., Lazaridou, A., and Graepel, T. (2019). Biases for emergent communication in multi-agent reinforcement learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). 3
Elman, J. L. (1993). Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71-99.
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H.,, Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. (2022). Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). 3
Fifty, C., Amid, E., Zhao, Z. Yu, T., Anil, R., and Finn, C. (2021). Efficiently identifying task groupings for multi-task learning. Proceedings of Advances in Neural Information Processing Systems (NeurIPS). 3
Finn, C., Abbeel, P., and Levine, (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of International Conference on Machine Learning (ICML). 4
Florensa, C., Held, D., Geng, X., and Abbeel, P. (2018). Automatic goal generation for reinforce- ment learning agents. In Proceedings of International Conference on Machine Learning (ICML).
Forestier, S., Portelas, R., Mollard, Y., and Oudeyer, P.-Y. (2022). Intrinsically motivated goal exploration processes with automatic curriculum learning. Journal of Machine Learning Research (JMLR), 23(152):1-41.3
Fujimoto. S., Hoof, H., and Meger, D. (2018). Addressing function approximation eтror in actor- critic methods. In Proceedings of International Conference on Machine Learning (ICML). 7
Gajane, P., Urvoy, T., and Clérot, F. (2015). A relative exponential weighing algorithm for ad- versarial based dueling bandits. In Proceedings of International Conference Machine Learning (ICML). 7
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., March, M., and Lempitsky, V. (2016). Domain-advend adversarial training of neural networks. Journal of Machine Learning Research (JMLR), 17(59):1-35.5
Genewein, T., Delétang, G., Ruoss, A., Wenliang, I. K., Catt, E., Dutordoir, V., Grau-Moya, J., Orseau, L.,, Hutter, M., and Veness, J. (2023). Memory- meta-learning on non-stationary distributions. In Proceedings of International Conference on Machine Learning (ICML). 4
Goodfellow, I., Pouget-Abadic, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio. Y. (2014). Generative adversarial nets. In Proceedings Advances in Neural Information Processing Systems (NeurIPS). 3,
Graves, A., Bellemare, M. G., Menick, J.. Munos, R., and Kavukcuoglu, K. (2017). Automated curriculum learning for neural networks. In Proceedings of International Conference on Machine Learning (ICML).2
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of International Conference Machine Learning (ICML).
Hadsell, R., Rao, D., Rusu, A. A., and Pascanu, R. (2020). Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 24(12):1028-1040. 6
12

---

Hafner, D. (2022). Benchmarking the spectrum of agent capabilities, In Proceedings of International Conference on Learning Representations (ICLR). 3, 7
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap. T. (2023). Mastering diverse domains through world models. arXiv preprini arXiv:2301.04//0
Hekimoglu, A., Friedrich, P., Zimmer, W., Schmidt, M.. Marcos-Ramiro, A., and Knoll, A. (2023).
Multi-task consistency for active learning. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR). 3
Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in neural network. arXiv preprint 1.XXvv5503.02555
Ho, J. and Ermon, S. (2016). Generative adversarial imitation learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS).
Jiang, G., Xu, M., Xin, S., Liang, W., Peng, Y., Zhang, C., and Zhu, Y. (2023). Mewl: Few-shot mul- timodal word learning with referential uncertainty. In Proceedings of International Conference on Machine Learning (ICML). 2
Jiang, L., Meng, D., Zhao, Q.. Shan, S., and Hauptmann. A. (2015). Self-paced curriculum learning. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI). 2
Jiang, M., Grefenstette, E.,, and Rocktäschel, T. (2021). Prioritized level replay. In Proceedings of International Conference on Machine Learning (ICML). 3
Johnson, M., Hofmann, K., Hutton, T., and Bignell, D. (2016). The malmo platform for artificial intelligence experimentation. In International Joint Conference on Artificial Intelligence (IJCAI).
Khan, F., Mutlu, B., and Zhu, (2011). How do humans teach: On curriculum learning and teaching dimension. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). 2
Khodadadeh, S., Boloni, L.. and Shah, M. (2019). Unsupervised meta-leaming for few-shot image classification. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS).
Kim, K., Park, D., Kim, K. I., and Chun, S. Y. (2021). Task-aware variational adversarial active learning, Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR). 2
Kim, T.-H. and Choi, J. (2018). Screenernet: Learning self-paced curriculum for deep neural net- works. arXiv preprint arXiv: 1801.080044 10
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barne Barwinska, A., et al. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 114(13):3521- 3526. 6
Kong, Y., Liu, L., Wang, J., and Tao, D. (2021). Adaptive curriculum learning. In Proceedings of International Conference Computer Vision 3
Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Master's thesis, University of Toronto. 10
Kumar, M., Packer, B., and Koller, D. (2010). Self-paced learning for latent variable models. In Proceedings of Advances in Newral Information Processing Systems (NeurIPS), volume 23. 10
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, J. (2017). Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253.
Li, H., Yang, X., Wang, Z., Zhu, X., Zhou, J., Qiao, Y., Wang. X., Li, H., Lu, L., and Dai, J. (2024). Auto me-reward: Automated dense reward design with large language models for minecraft. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR).
13

---

Liu, S., Johns, E., and Davison, A. J. (2019). End-to-end multi-task learning with attention. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR).
Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. (2017). Multi-agent actor-critic for mixed pepperative-oompeeee ral Information Processing Systems (NeurlPS).
environments. In Proceedings of Advances in Neu-
Matüseen, T., Oliver, A., Cohen, T., and Schulman, J. (2019). Teacher-student curriculum learning. IEEE Transactions on Neural Networks and Learning Systems, 31(9):732-3340. 3,7
Misra, I., Shrivastava, A., Gupta, A., and Hebert, M. (2016). Cross-stitch networks for multi-task learning. In Proceedings of Conference Computer Vision and Pattern Recognition (CVPR). 3
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In Proceedings of Interna- fional Conference on Machine Learning (ICML).
Mnih, V. Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv: 132.5223
Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E.,, and Stone, P. (2020). Curricu- lum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learming Research (JMLR),
Narvekar, S., Sinapov, J., and Stone, (2017). Autonomous task sequencing for customized curricu- lum design in reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI). 2
Narvekar, and Stone, (2019). Learning curriculum policies for reinforcement learning. In International Conference Aulonomous Agents and Multiagent Systems (AAMAS). 2,
Nichol, A., Achiam, J., and Schulman, J. (2018). On first-order meta-learning algorithms. arXiv preprint
4
OpenAl (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774, A10
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). A10
Parker-Holder, Jiang, M., Dennis, M., Samvelyan, M., Foerster, J., Grefenstette, E., and Rocktäschel, T. (2022). Evolving curricula with regret-based environment design. In Proceedings of International Conference on Machine Learning (ICML).
Pentina, A., Sharmanska, V., and Lampert, C. H. (2015). Curriculum learning multiple tasks. In Proceedings of Conference Computer Vision and Putlern Recognition (CVPR). 2
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. (2022). Red teaming language models with language models. In Annual Conference on Empirical Methods Natural Language Processing (EMNLP). A10
Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. (2017). Robust adversarial reinforcement learning. In Proceedings of International Conference Machine Learning (ICML). 3
Portelas, R., Colas, C., Weng. L., Hofmann, K., and Oudeyer, P.-Y. (2021). Automatic curriculum learning for deep rl: A short survey. In International Joint Conference on Artificial Intelligence (IJCAI). 2
Prideaux, D., Cantillon, P., and Hutchinson, (2003). Curriculum design. British Medical Journal, 326(7383):268-277)
Racaniere, S., Lampinen, A., Santoro, A., Reichert, D., Firoiu, V., and Lillicrap, T. (2020). Auto- mated curriculum generation through setter-solver interactions. International conference on learning representations. 2
14

---

Ren, M., Zeng, W., Yang, B., and Urtasun, R. (2018). Learning to reweight examples for robust deep learning. In Proceedings of International Conference Machine Learning (ICML). 10
Rohde, D. L. and Plaut, D. C. (1999). Language acquisition in the absence explicit negative evidence: How important is starting small? Cognition, 72(1):67-109.
Romac, C., Portelas, R., Hofmann, K., and Oudeyer, P.-Y. (2023). Teachmyagent: A benchmark for automatic curriculum learning in deep rl. Journal of Machine Learning Research (JMLR), 24(132):1-49, A10
Ruder, S. (2017). An overview of multi-task learning in deep neural networks. arXiv preprint 10X1v:1706.05098. 3
Ryu, K., Liao, Q., Li, 7., Delgosha, P., Sreenath, K., and Mehr, N. (2024). Curricullm: Automatic task curricula design for learning complex robot skills using large language models. arXiv preprint
11X.:22409.18382. A10
Santoro, A., Bartunov, S., Botvinick, M.. Wierstra, D., and Lillicrap. T. (2016). Meta-learning with memory-angmented neural networks. In Proceedings of International Conference on Machine Learning (ICMI.). 4
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy opti- mization algorithms. arXiv preprint arXiv:1707.06347 7
Sengupta, A., Dixit, S., Akhtar, M. S., and Chakraborty, T. (2023). A good learner can teach better: Teacher-studer student collaborative knowledge distillation. In Proceedings of International Conference on Learning Representutions (ICLR).
Sco, S., Kim, D., Ahn, Y., and Lee, K.-H. (2022). Active learning on pre-trained language model with task iaseinndeeendent triplet loss. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI).
Shu, J., Xie, Q. Zhao, Q., Zhou, S., Xu, Z.,, and Meng, D. (2019). Meta-weight-net: Learning an explicit mapping for sample weighting, In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). 10
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, I.,, Van Den Driessche, G., Schritwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489. A10
Silver, D., Hubert, T., Schrittwicser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel. T., et al. (2018). A general reinforeement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419)
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L.,, Lai, M., Bolton, A., et al. (2017). Mastering the game of go without human knowledge. Nature, 550(7676):354-59,, A10
Soviany, P., Ionescu, R. T., Rota, P., and Sebe, N. (2022). Curriculum learning: A survey. Interna- tional Journal of Computer Vision (IJCV). 13066):1526-1555 1565.
Standley, T., Zamir, A., Chen, D., Guibas, L., Malik, J., and Savarese, S. (2020). Which tasks should be learned together in multi-task learning? In Proceedings of International Conference Machine Learning (ICML). 3
Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A., and Fergus, R. (2018). Intrinsic motivation and automatic curricula via asymmetric self-play. In Proceedings of International Conference on Learning Representations (ICLR). 2,
Tenenbaum, J. B. (2020). Computational cognitive science: Building bridge to physics, narrowing the gap with biology. Current Opinion Behavioral Sciences, 38:105-11
Tenenbaum, J. B., Kemp, C., Griffiths, I.., and Goodman, N. (2011). How to grow a mind: Statistics, structure, and abstraction. Science, 131(6622) 2
15

---

Thrun, S. and Pratt, L. (1998). Learning to learn: Introduction and overview. In Learning to learn, pages 3-17. Springer.
Tzeng, E., Hoffman, J., Saenko, K., and Darrell, (2017). Adversarial discriminative domain adaptation. In Proceedings of Conference on Computer Vision and Puttern Recognition (CVPR).
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019). Grandmaster level in starcraft ii using multi- agent reinforcement learning, Nature, 575(7782)3350-344 2,
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2019a). Glue: A multi- task benchmark and analysis platform for natural language understanding. In Proceedings of International Conference on Learning Representations (ICLR). 10
Wang, G., Li. Y., Smith, L., McCann, B., Levine, S., Zhou, L., and Fang, Y. (2023). Voyager: An open-ended embodied agent with large language models. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). A10
Wang, X., Chen, Y., and Zhu, W. (2021). A survey on curriculum learning. Transactions Pattern Analysis and Machine Intelligence (TPAMI), 44(9):4555-44 4576. 2
Wang, Y., Yao, Q., Kwok, J. T., and Ni, L. M. (2020). Generalizing from few examples: A survey on sho- learning, ACM Computing Surveys, 53(3):1-34.
Wang, Z., Dai, Z., Póczos, B., and Carbonell, J. (2019b). Characterizing and avoiding negative transfer. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR). 3
Weinshall, D., Cohen, G., and Amir, D. (2018). Curriculum learning by transfer learning: Theory and experiments with deep networks. In Proceedings of International Conference on Machine Learning (ICML). 10
Wu, J., Hao, Yang, T.,, Hao, X., Zheng, Y., Wang, W., and Taylor, M. E. (2024). Portal: Automatic curricula generation for multiagent reinforcement learning. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI). 2, A9
Xu, M., Jiang, G., Liang, W., Zhang, C., and Zhu, (2023a). Active reasoning in an open-world environment. Advances in Neural Information Processing Systems, 36:11716-11736, 2
Xu, M., Jiang, G., Liang, W., Zhang, C., and Zhu, Y. (2023b). Interactive visual reasoning under uncertainty. Advances in Neural Information Processing Systems, 96:42409-42433
Xu, M., Yang, X., Liang, W., Zhang, C., and Zhu, Y. (2025). Learning to plan with personalized preferences. arXiv preprint arXiv:2502.00883 4
Yang, E., Pan, J., Wang, X., Yu, H., Shen, L., Chen, X., Xiao, L., Jiang, J., and Guo, G. (2023). Adatask: A task-aware adaptive learning rate approach to multi-task learning. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI), pages 10745-10753.
Yang, Y., Zhou, T., Han, L.,, Fang, M., and Pechenizkiy, M. (2024). Automatic curriculum for unsupervised reinforcement learning. In International Conference Autonomous Agents and Multiagent Systems (AAMAS).
Yarats, D., Brandfonbrener, D., Liu, H., Laskin, M., Abbeel, P., Lazaric, A., and Pinto, L. (2022). Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint 2rXiv:200..13422
Zagoruyko, S. and Komodakis, N. (2017). Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. In Proceedings of International Conference on Learning Representations (ICLR). 4
Zhang, B., Luo, C., Yu, D., Li, X., Lin, H., Ye, Y., and Zhang, B. (2024a). Metadiff: Meta-learning with conditional diffusion for few-shot learning. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI). 4
16

---

Zhang, C., Biswas, S., Wong, K., Fallah, K., Zhang, L., Chen, D., Casas, S., and Urtasun, R. (2024b). Learning to drive via asymmetric self-play. In Proceedings of European Conference on Computer Vision (ECCV). 3
Zhang, C., Jia, B., Zhu, Y., and Zhu, S.-C. (2024c). Human-level few-shot concept induction through minimax entropy learning. Science Advances, 10(16):a
Zhang, C., Song, J., Li, S., Liang, Y., Ma, Y., Wang, W., Zhu, Y., and Zhu, S.-C. (2024d). Proposing and solving olympiad geometry with guided tree search. arXiv preprint 2rX1v:2412.10673.
Zhang, C., Zhu, Y., and Zhu, S.-C. (2019). Metastyle: Three-way trade-off among speed, flexi- bility, and quality in neural style transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1254-1261.
Zhang, Y. Abbeel, P., and Pinto, (2020). Automatic curriculum learning through value disagree- ment. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). 2,
Zhao, Y., Zhong, Z., Yang. F., Luo, Z., Lin, Y., Li, S., and Sebe, N. (2021). Learning to generalize unseen domains via memory- based multi-source meta-learning for person re-identification. In Computer Vision and Pattern Recognition (CVPR). 4
Proceedings of Conference
Zhou, Y., Pan, Z., Wang, X., Chen, H., Li, H., Huang, Y., Xiong, Z., Xiong, F., Xu, P., Zhu, W., al. (2024). Curbench: curriculum learning benchmark. In Proceedings of International Conference on Machine Learning (ICML). 10, A10
Zhu, Y., Edmonds, M., Qi, S., Wu, Y. N., and Tenenbaum, B. (2023). Understanding physical and conceptual primitives for reasoning, In Annual Meeting of the Cognitive Science Society (CogSci).
Zhu, Y., Gao, T., Fan, L., Huang, S., Edmonds, M., Liu, H., Gao, F., Zhang, C., Qi. S., WU, Y. N., et al. (2020). Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense. Engineering, 6(3):310-345,
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's s contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Section F for the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specificattin, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [es]]
18

---

Justification:
We have provided the detailed assumptions and proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have included necessary details the paper and the supplementary mate- rial. The code will be released in our project website.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the implementation of the used benchmarks and algorithms in the supplemental material. The code will be released in our project website.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niin cc/
public/guided guuide//Coeeuuuissseeesseeeeee/e www.iiisssssoooooooooooooooooooo wwwsssssoooliooooooooooioooiiiii ooiccy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /niiss.. cc/ hte//ublic/// wwwpc//gg//eeeeeeeceeeeeeseceees /CoeeS// ssi aesionPoll onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes. We have included these details in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: Yes. See the curves the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory. time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes. We have included these details Section 3.4 and Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: h/tp:://n.e...eeeeeeee httbblii//hhhcccccieeeeeees
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have included the discussion in Section F.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g,, deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We include the required licenses in the supplemental material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
If assets are released,
22

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmoell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Ye]]
Justification:
We have included the human experiment pipeline in Section D.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We have obtained IRB approvals from our institution (School of Psychologi- cal and Cognitive Sciences, Peking University).
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23

---

A Environments
Minigrid
Simple
Hard
Empty
Crossing
Fetch
LockedRoom
GoToDoor
Playground
Craft/Crafter
Simple
Took: get wood
Tosk: get dimond
maake
make
Figure Al: Experimental environments spanning discrete navigation open-world scenarios. Top: Mini- grid environment with tasks arranged increasing difficulty (left to right), from simple navigation (Empty, Crossing) to complex reasoning (Playground), enabling systematic curriculum evaluation on well-defined hier- archies. Bottom left: CRAFT and Crafter environments provide Minecraft- innsrrff--nnspred multi-task scenarios with crafting mechanics and procedural generation, testing curriculum adapiation complex domains. Bottom right: Task dependency graph showing hierarchical skill structure, where nodes represent individual skills and edges indicate prerequisites. Tasks with longer dependency chains present increased complexity, evaluating HAP's ability to navigate multi-step step planning challenges.
We mainly include Minigrid, CRAFT, and Crafter as our testbed. We divide tasks in each benchmark into three levels: easy, middle, and hard, to test the learning progress separately. Considering factors like resource availability, crafting complexity, and or level of skill or progression required. To ensure fair comparison across methods, we customize the task structures within each environment to create unified benchmarks. Each environment contains tasks of varying difficulty levels with complex interdependencies designed to test different aspects curriculum learning.
A.1 Minigrid
Minigrid is collection of 2D grid-world environments with goal-oriented tasks. Our implementa- tion is based on the open-sourced code!. We include six tasks from the Minigrid task pool:
Empty The agent must reach green goal square in empty room, with sparse reward and step penalty.
The agent must navigate to the goal while avoiding deadly lava rivers, which have single safe crossing points.
Crossing
DoorKey This environment has a key that the agent must pick up in order to unlock a door and then get to the green goal square.
FourRooms In this classic four-room RL environment, the agent must navigate in a maze com- posed of four rooms interconnected by gaps in the walls. To obtain a reward, the agent must reach the green goal square. Both the agent and the goal square are randomly placed in any of the four rooms.
MultiRoom This environment has a series of connected rooms with doors that must be opened in order get the next room. The final room has the green goal square the agent must get to.
A1

---

Playground An environment with multiple rooms and random objects. This enviromment origi- nally has no specific goals or rewards. The agents are tasked to collect all objects in our research.
has been shown that hard tasks in Minigrid can be solved in a curriculum way, iee. the MultiRoom environment can be solved by gradually increasing the number of rooms with human-defined curriculum. However, such curricula rely on human expertise and lack generalization. We explore automated curriculum policies across diverse tasks in this work.
Table A1: Tasks in Minigrid.
Level
Tasks
Easy
Empty, Crossing (Simple navigation)
Middle
DoorKey, FourRooms (Tool using/ Multi-room)
Hard
MultiRoom, Playground (Larger maps with challenging tasks)
A.2
Craft
Table A2: Tasks in CRAFT.
Level
Tasks
Easy
get[grass], gel[wood], make[stick], make[plank], get[rock]
get(iron), get(gold], make[axe], make|bench), rope] make/ arrow J. make/knife). makeesshears] make(slingshot), make[cloth]
Middle
get[gem], make|bed], make/bow) make[brrige)), make[bundle]
Hard
make| [flg] make (oldagoold make[hammert make|ladder!
CRAFT CRAFT (CraftEnv) is 2D crafting simulation adapted from Andreas et al. (2017), de- signed to support flexible, hierarchical tasks with sparse rewards in a fully procedural world. Agents must navigate, collect items, manage inventory, and transform materials at workshops to accom- plish range of tasks. Many tasks are multi-step and require combining resources and actions in sequence- such as building a bridge access gold-which which be challenging for agents using random exploration. The environment supports different tasks varying in complexity, from simple collection tasks to intricate multi-step crafting objectives. Our implementation is based open source code?.
We split the tasks in CRAFT into different levels. Easy tasks in CRAFT are straightforward, require minimal resources, and can be done early in the game with basic tools or no tools at all. Middle tasks require more resources, better tools, or intermediate crafting steps. They are achievable after some progression in the game. Tasks labeled hard are more advanced, require rare resources, or involve complex crafting chains. They are typically done later in the game. Table A2 shows all tasks in CRAFT.
Table A3: Tasks in Crafter.
Level
Tasks
collect[wood], collect(sapling] catlplant), htcpe//ood__picceoeee
Easy
make/wood_swordl place/plant), wake_up
collect[stone), collect(iron], collect[coal], faakee/stne..iccaanee
Middle
make[stonc swordd) placefstone). J. placoftable], cat[cow]
collect(diamond), defeat(skeleton), collect[drink], makefiron pickaxe] makefiron.swool flanacf defeat[zombie]
Hard
Crafter The tasks in Crafter can be divided into three categories similar to those in CRAFT, as shown in Table A3. Since the original Crafter environment is too challenging for current agents, particularly in survival tasks, removed the survival requirements better evaluate performance on more complex tasks. This modification led to generally improved performance compared to the
A2

---

baselines reported in the original Crafter paper. For clearer comparison, the results of the baselines in Crafter have been normalized to the range [o, 1]. We include a fair comparison with the officially reported algorithms, plus the human-expert results reported in Accheveement-Dissillaaionn as shown in Table A4. Our implementation is based on open source code".
Table A4: Performance comparison of different algorithms in Crafter.
Algorithm
Score (%)
Reward
Open Source
Curious Replay
19.4
AttomonoussgcatLLaoorrrtsss
PPO (ResNet)
1.6
10.3
snu-mllab/ Aatanlaai-Acheeeeme----iiillaal
DreamerV3
1..5
11.7#1.9
danijar/dreamerv3
LSTM-SPCNN
12.1
astanic/crafter-ood
EDE
11.71100
yidingjaang/ede
OC-SA
11.1+0.7
astanic/crafter-ood
DreamerV2
1..0
9.0
danijar/dreamerv2
PPO
4.6+0.3
4.2+1.2
DLR- R-/sRRMMe--baselllllaa
Rainbow
4.310.2
6.01
Laixhin/Rainbow
HAP (Ours)
14.311.2
9.240.9
HAP (Ours, in casier mode)
23.1+3.2
15.2+2.1
Humans (Achievement Distillation)
50.516.8
14.3+2.3
Humans (Ours, in easier mode)
60.5+9.2
17.8+2.5
B The Heterogeneous Adversarial Play (HAP)
Unlike traditional cooperative curriculum learning- where the teacher selects tasks in an optimal "Goldilocks zone" difficulty to facilitate learning- we intentionally frame the teacher-student in- teraction in HAP zero-sum, adversarial process. This design centers a dynamic equilibrium: as the student masters current tasks, the teacher autonomously generates more challenging problems, continually raising the bar and expanding the space of solvable tasks. The teacher's objective is not merely to assist learning, but to produce tasks that are maximally challenging and valuable, thereby driving the student acquire advanced capabilities.
Formally, let Fam 1 n Di= ri denote the student's average performance. The teacher's reward is then defined as
0,
if stu or Fsu & -
Tieacher
(A1)
Fau otherwise,
The student's reward can be structured in analogous manner, reinforcing the zero-sum setup:
O0
if = 0 or Fsu & 1 -
T Tstu
(A2)
Fstu.
otherwise
The core motivation for this adversarial framing is that, if the student consistently solves tasks, there are no further meaningful learning opportunities-tte system ceases to be educationally useful. Instead, our framework ensures that teacher and student are continually co-adapting: the teacher constructs tasks just beyond the student's 's current ability, and the student strives keep up. This dynamic bootstrapping not only accelerates learning progress but also encourages the emergence of both a problem proposer and a solver capable meeting highly challenging, diverse tasks. con- trast to static, handcrafted curricula, adversarial optimization discovers and instantiates fundamental pedagogical principles underlying effective instruction in both artificial and natural systems.
This pure adversarial setup may introduce training difficulties. Ideally, the zero-sum formulation creates a dynamic equilibrium where the teacher finds tasks that maximally challenge the student's current capabilities. But for teachers like simple probability teacher, there is indeed no such guarantee. We do encounter cases when training collapses due to pathological teacher task selection, so we further introduce entropy regularization and cold-start policies to help avoid these cases.
We provide two versions of HAP for reference:
A3

---

Algorithm 2: Detailed adversarial training loop of the Heterogeneous Adversarial Play (HAP).
Data:
Initial student policy parameters dg. teacher parameters do: learning rates a. 9: task set rollout batch size N. trajectory length H
for iteration k = 1, K do
/+ Observe Student History
Retrieve or update student behavior window ha (e.g., recent returns, trajectories, success rates);
/+ Teacher Task Distribution Computation
Compute teacher logits f = f0.-1 (ha); (ha);
Compute task probabilities P (by voftmax((4,))=1]
/* Task Sampling and Environment Setup
Sample mini-batch of N tasks (2<()) Poc_i (T)b);;
/* Student Policy Rollouts for each task ( in the batch do Initialize environment starting state 80 ~ E(T(()) Roll out student policy nT, 1 [a|s, (()) for steps;
Record trajectory (() {(s,, az r1)3
Compute total (discounted) task return: R(r(0; (()) Σ Y'G
10
end
/* Student Policy Update (Maximization)
Estimate or compute advantage A for each trajectory, egg., with baseline or critie;
12
13
>i=1 V log mo, (0.111 Update: 02 0 gu:
(i) T(1) A00;
14
/t Teacher Adversarial Update (Minimization)
DN V log ( (Te) (()()),(0))
15
9o
16
Update: Φ6 - 0h-1 3 Do:
/+ (Optional) Logging and Evaluation Log statistics: average returns, task distribution, teacher entropy, etc.: convergence or early stopping criteria met then break;
17
18
19
29
end
21
end
Or, with simple probability teacher:
A4

---

Algorithm 3: Simple probability teacher.
Require: Initial student policy parameters 0 teacher parameters
Require: Learning rates (student), 5 (teacher)
while not converged
/* Teacher Task Selection:
Compute task probabilities using a softmax over teacher parameters:;
Po(T:) Σ exp(t,))
exp(o.)
Sample task T from the distribution p+(T):
T ~'~ (T); /+ Student Policy Execution:
Student interacts with the environment on task using policy a(a 8, T:0);
Collect trajectory 7 {80, do. To...., 88) and compute cumulative reward:;
T) -i 7'ri;
/+ Student Update:
Update student policy parameters 0 to maximize expected reward::
10
0 + (0nsem)))))
11
where;
12
Jandem (0) B.--(t [R]::T)]:
/+ Teacher Update: Compute the gradient of the teacher's objective:;
14
V Vashacchr()) -Er-ps (T)) [V (o) Er-x(/rr)) [R(r;T)]]
15
Update teacher parameters o to minimize the student's 's expected reward:;
á ,V v
17 end
C Experiment Details
Table AS: Model Parameters
Nav Task.
Component
Parameter
Value Description
Framework
A2C
Actor/Critic Hidden layers
2
Actor/Critic Hidden units/layer
256, 128
Activation
ReLU
Student Policy
Optimizer
Adam
Learning rate
lc-4
Discount (7)
0.99
Task embedding dim
512
Network type
MLP
Input (history vec)
Last 100 student returns
Hidden layers
2
Teacher Policy
Hidden units/layer
256, 128
Update Freq
1000 steps
Activation
ReLU
Optimizer
Adam
Learning rate
lc-4
Task Window
Batch size (trajectories/update)
32
Max steps (per episode)
200
Nav See Table A5.
Minigrid See Table A6.
CRAFT See Table A7.
Crafter See Table A8.
A5

---

Table A6: Model Parameters Minigrid Task.
Component
Parameter
Value / Description
Framework
PPO
Actor/Critic Hidden layers
Actor/Critic Hidden units/layer
256, 128
Activation
ReLU
Optimizer
Adam
Learning rate
Policy: 3e-4; Value: 1e-3
Student Policy
0.99
Task embedding dim
512
0.1
GAE
0.95
ent.coef
0.01
vf.coef
0.5
Network type
MLP
Input (history vec)
Last 100 task indices
Hidden layers
Teacher Policy
Hidden units/layer
256, 128
Update Freq
100 episodes
Activation
ReLU
Optimizer
Adam
Learning rate
10-3
Task Window
Batch (trajectoriesuupdate)
Max steps (per episode)
200
Table A7: Model Parameters CRAFT Task.
Component
Parameter
Value / Description
Framework
PPO
Actor/Critic Hidden layers
Actor/Critic Hidden units/layer
512, 256, 256, 128 ReLU
Activation
Optimizer
Adam
Learning rate
Policy: le-4; Value: le-4
Student Policy
0,99
Task embedding dim
512
0.1
GAE A
0.95
coef
0.01
vf coef
Network type
MLP Last 100 task indices
Input (history voc)
Hidden layers
Teacher Policy
Hidden units/layer
512, 256, 128, 128 50 episodes
Update Freq
Activation
ReLU
Optimizer
Adam
Learning rate
le-4
Task Window
12
Batch size (trajectories/updatt)
128
Max steps (per episode)
1000
For the remaining baselines reported in the main draft, most are based on open-source implementa- tions from BableeBasselines3, the and the official CRAFT and Crafter Repo. The EXP3 baseline re-implemented following the official blog.
A6

---

Table A8: Model Parameters Crafter Task.
Component
Parameter
Value Description
Framework
PPO
Actor/Critic Hidden layers
Actor/Critic Hidden units/layer
512, 256, 256, 128
Activation
ReLU
Optimizer
Adam
Learning rate
Policy: lc-4; Value: lc-4
Student Policy
0.99
Task embedding dim
512
0.1
GAE
0.95
ent.coef
0.01
vf.coef
0.5
Network type
MLP
Input (history vec)
Last 100 task indices
Update Freq
50 episodes
Teacher Policy
Hidden layers
Hidden units/layer
256, 128, 128 ReLU
Activation
Optimizer
Adam
Learning rate
le4
Task Window
Batch size (trajectories/updatt)
128
Max steps (per episode)
1000
C.1 Ablation Study: Effect of Student History on Teacher Performance
In all experiments, our teacher leverages the student's recent reward history, which we find impor- tant for assessing both current state and longer-term learning trajectories. We performed ablation in the Minigrid environment to measure the impact of history length on overall curriculum effective- ness. Table Table A9 summarizes the results.
Table A9: Performance in Minigrid under different history lengths teacher. General denotes average
across all difficulty levels.
Method
Easy
Middle
Hard
General
Last 1k history
0.92
0.44
0.18
0.510
Last 100 history
0.92
0.46
0.20
0.527
Without history
0.92
0.43
0.11
0.487
Including student history leads to substantial improvements, especially in more difficult tasks and overall generalization. However, incorporating too much history (e.g., last 1,000 steps), may dilute sensitivity to the student's current ability and reduce performance. Using an appropriately sized history window better reflects the learner's status and maximizes adaptive curriculum benefits.
C.2 Teacher Update Frequency and Asynchronous Scheduling
As detailed in the tables above, the teacher's update frequency is task-dependent and tuned for cach experimental setting. Typically, the teacher is updated after fixed number of student steps, with the interval chosen to balance adaptation speed and computational efficiency. In more complex environments, updating the teacher requires a full evaluation of the student, which can introduce time bottlenecks.
We also experimented with asynchronous teacher updates, but observed no significant improvement student performance compared to synchronous updates. Consequently, we adopt synchronous updates with carefully selected intervals to ensure efficient and effective curriculum adaptation. Further implementation details and related empirical analyses are provided in the supplementary material.
A7

---

C.3 Sensitivity Analysis and Ablation of Hyperparameters
We conducted extensive ablation studies to evaluate the impact of key hyperparameters, including entropy regularization and warm-up (cold start) duration, by running each configuration 10 times with different random seeds the Minigrid environment. Results are reported in Table A10.
Table A10: Performance in Minigrid under various ablation settings. General denotes the average across
all difficulty levels.
Configuration
Easy
Middle
Hard
General
Original model
0.92
0.46
0.20
0.527
w/o entropy regularization
0.91
0.38
0.11
0.467
w/o cold start w/o lower bounds
0.92
0.45
0.20
0,523
3/10
1/10
0.21
Easy tasks are reliably learnable in all cases. Entropy regularization crucial for performance on harder tasks; its removal leads to marked degradation in the 'Hard' setting. Cold start primarily impacts training efficiency, affecting convergence rate rather than final results. Lower bounds are essential for stability as models without them often fail to converge for middle and easy tasks and exhibit catastrophic forgetting on simpler tasks. This sensitivity analysis substantiates the critical role of these empirically set components in robust training across difficulty levels.
D Human Study
Our human study aims to verify the importance of curriculum in human learning and to evaluate the effectiveness our algorithm in generating suitable curricula based on the human learning curve. We recruited 30 participants via the Prolific platform to ensure diverse and controlled sampling. Inclusion criteria were: age between 18 and 40, fluent English proficiency, normal or corrected- to-normal vision, and at least bachelor's degree. Participants provided informed consent before proceeding and were compensated in accordance with institutional and Prolific guidelines. The study protocol was reviewed and monitored by formally constituted ethics committee at our institute, in compliance with biomedical research regulations involving human subjects.
To minimize confounding factors and better align the human experimental setting with that of AI models, we modified the standard Minigrid Betevalier-Boiveee et al., 2019) environment as follows:
Visual Redesign The environment's color palette and the icons for agents, goals, and objects were replaced with high-contrast, universally interpretable symbols, but without explicit semantic meaning. This ensured that participants could not leverage any real-world prior knowledge or bias related to these elements.
Implicit Buttons To intuitively guide action selection and reduce interface learning curves, all actionable elements (e.g., use, toggle door, pick up) were renamed to generic labels such as "Button 1." "Button 2." etc. If tutorial was provided, participants had to discover the function of each button through trial and error. However, movement buttons were made explicit, matching the clarity of available actions the Al agents.
Reward Design We adjusted the reward structure to encourage participants to maximize their score by exploring, collecting keys, opening related doors, and avoiding harmful elements. The underlying reward mechanism was not explicitly described to participants; they were only instructed to maximize their score.
Tutorial Conditions
We constructed three tutorial conditions for the experiment:
No Tutorial (Control Group): Participants began directly in the test environment, receiving only the instruction to maximize reward.
Expert Step-by-Step Tutorial: A team of Minigrid-expprreeeed researchers manually designed an optimal skill progression canonical curriculum. Each mini-tutorial covered one incremental skill (e.g., "navigate to goal," "unlock door," "collect target object"), and each step was presented visually and explained textually.
A8

---

Stage Empty Environment
Stage Crossing
Stage Door Key
Stage Four Rooms
Stage Locked Room
Stage Playground
Reagee an Stgt Tiral Rewart
(Gument Plaas/ 100 (Teal Shepe)
L.
Don
Hight
Centieser
Figure A2: Human study platform demonstration. After completing their assigned curriculum, participants are tested the modified Playground scenario. The left panel displays the available subtasks and illustrates sequence of mini-tutorials provided in the Expert Step-by-Step condition.
Al-Generated Automatic Tutorial: Leveraging the HAP curriculum-leaming framework, we au- tomatically generated adaptive lesson sequences. At the end of each round, the AI evaluated each participant's performance and dynamically selected the next lesson to address observed weak- nesses. Note that, because the human study involved far fewer training epochs than typical AI settings, we customized HAP's hyperparameters, particularly the feedback parameters, to ensure timely and effective online adaptation of the curriculum for human learners.
Figure A2 presents a demonstration of our human study platform. After completing their assigned curriculum, all participants ultimately tested in the modified Playground setting. The left panel shows the available subtasks and the sequence of Expert-designed Step-hy-Step Tutorials.
We also set a post-experiment for the subjects, asking them about the functionality of the am- biguous buttons and elements in the game, shown in Figure A3.
Without Tutoraa Expert
Wkh Auto Tutorial
E
Further Discussion
Count
E.1 Comparison
with Existing Active Learning Approaches
Active Learning and Automatic Curriculum Learning (ACL) have evolved into well- developed domains with numerous sophisti- cated methods designed to optimize learning trajectories. While we have included a brief introduction to these related works, we would like to further elaborate how our approach compares with existing active learning and cur- riculum learning methods.
amblgi.nus butfie
Iely
doot
Object
Figure A3: Comparison of correct for each object type and condition human study.
From the sample selection paradigm perspective, traditional active learning approaches primarily operate through sample selection based on specific criteria, measuring informativeness through un- certainty or diversity metrics. Recent advances like PORTAL (Wu et al., 2024) attempt to discover task sequences automatically but, unlike our approach, require explicit task features (specifically task similarity and difficulty metrics in PORTAL) and predefined search spaces. While these criteria effectively identify ordering relationships between tasks in controlled environments, they lack the
A9

---

dynamic evaluation and generation mechanisms necessary for more complex settings. Pre-defined policies prove valuable when the task space is constrained or when learning objectives are suffi- ciently intuitive for human designers to create effective switching policies. However, they demon- strate significant limitations in larger, more complex environments where optimal task sequencing becomes less obvious and more context-dependent.
From the feedback loop mechanism perspective, a critical limitation in existing curriculum frame- works is their predominandl unidirectional optimization process, TeachMyAgeni (Romac et al., 2023) implements
curricula but relies on predetermined environment parameteriza- tions rather than adversarially discovering optimal challenges. CurBench (Zhou et al., 2024), while establishing evaluation protocols for curriculum learning, confirms that most methods struggle with dynamic adaptation to learner progress. In contrast, our approach tries to establish continuous bidirectional feedback loop where the teacher's task generation and the student's problem-solving capabilitiess co-evolve, creating system that autonomously identifies and addresses knowledge gaps through their adversarial interaction.
Our idea draws inspiration from the remarkable success of self-play methods in artificial intelli- gence, particularly in complex strategic domains. AlphaGo (Silver al., 2016) and its successors (Silver et al., 2017) demonstrated that self-play creates an emergent curriculum of increasing com- plexity without requiring human examples or explicit task engineering. This automatic adaptation has proven exceptionally powerful because it continually maintains an appropriate challenge level as agent capabilities evolve. More recently, similar adversarial dynamics have appeared in language model training through techniques like RLHF (Christiano et al., 2017; Ouyang et al., 2022) and adversarial prompting (Perez al., 2022), where models improve by addressing increasingly so- phisticated challenges. Our approach extends these principles beyond symmetric self-play to the in- herently asymmetric teacher-student relationship, preserving the beneficial adaptive dynamics while accommodating the different roles in curriculum learning. This allows us to capture the emergent complexity benefits of adversarial approaches while tailoring the process specifically to pedagogical objectives.
E.2
LLMs as the Teacher
Recent research has explored leveraging Large Language Models (LLMs) as curriculum designers and teachers in automated learning systems (Wang al., 2023; Ryu et al., 2024). This emerging paradigm utilizes the extensive knowledge and reasoning capabilities of models like GPT-4 (Ope- nAI, 2023) and PaLM (Chowdhery al., 2023) generate learning tasks, provide feedback, and adapt curricula. LLM-based teachers can potentially draw on broad domain knowledge to create diverse and contextually appropriate challenges without requiring explicit programming of task gen- eration strategies. For instance, Wang et al. (2023) demonstrated that LLMs can effectively design progressively complex tasks in Minecraft environments, while Ryu et al. (2024) showed promising results using LLMs for learning complex robot skills.
Despite these advances, we intentionally excluded LLM--based teaching approaches from current work for several reasons. First, LLM-generated curricula, while impressive, still lack theoretical grounding in optimization principles they operate through heuristic prompting rather than targeted adversarial dynamics. This introduces uncertainties about their ability to maintain optimal challenge levels without human oversight. Second, LLMs currently serve as task generators but typically lack integrated mechanisms to observe and adapt to learner states in real-time, creating a disconnect in the feedback loop essential our approach. Third, the "black-box" nature of L.---based teachers complicates analysis of emergent teaching strategies and makes it difficult to isolate the effects of curriculum design from the model's 's innate capabilities.
E.3 Further extending of HAP: Meta-Learning Perspective
In the current implementation of HAP, we treat the teacher as single neural network that takes the learner's learning performance as input and outputs the curriculum. The framework is actually simplified for easier training. We demonstrate that HAP can be extended through a meta-learning lens, where the teacher itself becomes an adaptive agent that learns optimal teaching strategies. While the original framework establishes adversarial dynamics between teacher and student, this
A10

---

Algorithm 4: Extended HAP with meta-Learning (a simple demo).
Data: Initial 0,6; learning rates o, while not converged do
/* 1. Observe Student's Learning State:
Compute teacher state from student's learning history;
/* 2. Teacher's Meta Taak Generation:
Generate task parameters:
(So(Swcche);
/* Using actor network to generate pedagogically valuable tasks
/* 3. Student's Policy Execution:
Execute r[a|s, C; collect trajectory T; Compute reward: R(T; C) > y'ra:
Update 8 to maximize returns::
8 - 8 + aV. FV
/+ 4. Teacher's s Meta-Learning Update:
Compute teacher reward Fiaccerr based on student progress;
Store transition (Sreacher, C, Tiescher; Steacher) in buffer:
11
Update critic: minimize (Qus (Steahher C) Preacher
Qu (Seatteers Mo ())astee)))
12
Update actor: maximize Q (Seeacher (Suaaceer ));
13
end
extension formalizes how the teacher can systematically improve its curriculum generation through experience, albeit at significantly higher computational cost for meta- peaita-rrtraiinng..
In this extended formulation, model the teacher as operating where states reflect the student's learning trajectory, and actions correspond to task parameters. Unlike the simple network our baseline approach, the teacher now employs more sophisti- cated actor-critic architecture to capture the complex relationship between curriculum decisions and student progress. The teacher's state Staahee comprises observations about the student's learning progress, potentially including:
higher-level eettrennvionment
Swacher Aperformance Rggradients: Ausaeectorics ...
(A3)
where / represents historical windows of various student metrics. The teacher's meta-learning ob- jective becomes:
max Jwacher (ő)
(A4)
where F incorporates pedagogical signals beyond the purely adversarial reward.
The student agent remains similar original formulation, learning policy I[ass, C;0) to maximize expected returns. However, the relationship between teacher and student becomes more nuanced:
max studentt (0) EC pul Steacher)
[E (:0) []]]
(AS)
where of (Seacher)) represents the teacher's actor network that maps the student's 's learning state to task parameters.
The modified training algorithm follows a similar structure to our original approach but incorporates meta-learning elements:
This meta-learning extension creates a teacher agent capable of developing sophisticated teaching strategies through experience. Unlike our baseline adversarial approach, the teacher now aims to (i) identify optimal challenge levels that maintain student engagement; (ii) recognize when to re- visit foundational concepts versus introducing new challenges; (iii) develop understanding of skill transfer and prerequisite relationships; and (iv) create coherent task sequences that build upon previously learned skills.
The original HAP framework can be seen as a rule-based version of meta-learning extension, where we leverage adversarial policy the teacher's intuition. The extended bidirectional learning process also mirrors sophisticated human teaching, where effective educational strategies emerge from repeated interactions rather than being fully specified in advance.
A11

---

F Limitations and Border Impact
Our experiments were conducted on simulated learners with homogeneous skill progression pat- terns, which may not fully capture the complexity of real-world learning environments. Perfor- mance degradation could arise in highly heterogeneous task structures or noisy learning conditions. We cannot do real-world testing in this work due to the absence of high-fidell sigh-fideiity simulators and limited adaptability of baseline learning frameworks currently available. HAP can help build more intelligent AI agents, holding transformative potential for scalable, personalized adaptive Al systems, particularly in resource-constrainet learning settings.
G Complexity Analysis of Algorithm Variants
We provide simple analysis of the algorithmic complexity of our proposed meta-learning extension to HAP, and compare it with a simplified version to establish upper and lower bounds.
Upper Bound: Meta-Learning Framework The meta-learning extension represents upper bound in terms computational complexity. This upper bound reflects the comprehensive nature of our meta-learning extension, which maintains detailed state representations and employs sophis- ticated actor-critic architectures for both teacher and student.
Time Complexity: [[]] |Seecceer |o| H 10] + B 1)] per iteration, where |/ is the history length, |tcacher| is the dimensionality of the teacher's state representation, |ø] and [0] are the parameter counts of teacher and student networks respectively, H is the task horizon length, and B is the mini-batch size for teacher updates.
Space Complexity: O(Ool 10) D M). where D is the dimension of stored transitions and M is the replay buffer capacity.
Sample Complexity: The meta- veta-learning approach potentially requires O(CC?) task explorations the worst case to fully model relationships between tasks in curriculum space C.
Lower Bound: Simplified Algorithm If we replace the actor-critic architecture with basic heuris- ties like task cycling or simple difficulty gradients, we can get a simplified version of HAP, which maintains only minimal state about student performance (e.g., success rate the current task) and uses a predefined based task selection strategy without extensive historization or predictive modeling. simplified variant of approach provides lower bound complexity:
Time Complexity: O + (0l) per iteration, where
small constant representing the
complexity of simple heuristic task selector.
Space Complexity: O(10) + k'), with K' being the minimal state representation needed for basic task selection.
Sample Complexity: A simple approach might require only O(CC) task explorations with linear progression through the task space.
While the meta-learning approach incurs higher computational overhead, it offers significant ad- vantages in dynamic environments with complex task interdependencies. The simplified approach may be sufficient for domains with clear, linear difficulty progression but will likely fail to identify optimal curricula in complex skill acquisition scenarios. Empirically, we observe that the addi- tional computational cost of the meta-lcarning approach is justified by substantial improvements in student learning efficiency, particularly in domains where task relationships non-obvious and student learning dynamics are complex.
A12


Reviews:
[{"review_id": "nVUTveGJKw", "reviewer": "Reviewer_EgTN", "summary": "This paper proposes Heterogeneous Adversarial Play (HAP), which extends the teacher-student framework for curriculum generation. In HAP, the teacher and the student plays a zero-sum game where the teacher selects difficult environment for the student, while the student tries to maximize its reward in that environment. The framework is evaluated in several grid-based environments and is shown to perform comparable or better than standard RL algorithms as well as curriculum learning approaches. The authors also conducted human experiments to evaluate the framework for designing curriculum for Minigrid.", "questions": "1. What exactly is the significance of heterogeneity in *heterogeneous* adversarial play? And how does it differ from \"asymmetric\"?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths:\n- The framework is presented clearly, and the progression from curriculum learning to teacher-student framework and eventually to HAP is well motivated.\n- The experiments cover a wide range of comparisons, from simple grid-based environments to human curriculum design.\n\n## Weaknesses:\n- Characterizing the teacher-student interaction as a zero-sum game seems like an odd choice to me, especially given literature on zone of proximal development. The common intuition is that the teacher is supposed to pick tasks with a goldilock difficulty: not too easy that the student can’t practice anything new, and not too hard either. There wasn’t sufficient justification for why HAP frames the interaction as zero sum, and no discussion or empirical evidence to explain why the training would not collapse to the teacher picking an adversarial curriculum, e.g., oscillating between several most confusing environments.\n- Although the authors did a decent job reviewing the literature of curriculum design methods and included some standard approaches as baselines, the related work section missed out on some of the most relevant frameworks where the teacher and student engage in zero-sum or asymmetric games. \n     - Protagonist Antagonist Induced Regret Environments\n    - Rethinking Teacher-Student Curriculum Learning through the Cooperative Mechanics of Experience: \n    - Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play\n- The experiments are limited. The main quantitative evaluation of HAP is based on three environments: Minigrid (Chevalier-Boisvert 246 et al., 2019), CRAFT (Andreas et al., 2017), and Crafter (Hafner, 2022). All three are grid-based environments with similar visual representations and discrete action spaces. Although the author claimed that the extension to continuous tasks is ease, there is no evaluation on continuous control tasks like robotics, physics simulations. All are visual grid worlds - no language-only or multimodal tasks.\n- Insufficient human evaluation. Although I appreciate it that the authors did human experiments to validate the effectiveness of HAP at creating curriculum for humans, the sample size is very limited and the choice of task is debatable. It’s unclear what aspect of human learning is captured by teaching Prolific workers Minigrid. So it remains hard to predict how HAP would perform in real world curriculum generation.", "was_revised": true, "final_justification": "In my initial review, too much weight was put on my criticism of the framework, which comes from a philosophical disagreement. In the rebuttal, the author provides significant empirical evidence to support their viewpoint, and provided additional experimental results to further support their claims. Given this, I can recommend the paper to be accepted to the conference.", "rebuttal": "Dear Reviewer EgTN,   \nThank you for your thoughtful feedback. We try to answer your questions as follows:   \n\n> Characterizing the teacher-student interaction as a zero-sum game seems like an odd choice to me, .... The common intuition is that the teacher is supposed to pick tasks with a goldilock difficulty ... There wasn’t sufficient justification for why HAP frames the interaction as zero sum, ... why the training would not collapse to the teacher picking an adversarial curriculum.\n\nWe respectfully disagree with your intuition. Adopting an goldilock standard does not affect the zero-sum / adversarial nature of our setup. Letting $\\bar{r} _ {\\text{stu}} = \\frac{1}{n} \\sum _ {i=1}^N r _ {\\text{stu}}^{i}$, the teacher's reward can be simplified as:\n\n$$\nr _ {\\text{teacher}} = 0 \\quad (\\text{if } \\bar{r} _ {\\text{stu}} = 0 \\text{ or } \\bar{r} _ {\\text{stu}} \\leq 1 - \\epsilon) \\quad or \\quad  - \\bar{r} _ {\\text{stu}} \\quad (\\text{otherwise})\n$$\n\n\nYet, the student reward can also be modified such that $r _ {\\text{stu}} = 0 (\\text{if } \\bar{r} _ {\\text{stu}} = 0 \\text{ or } \\bar{r} _ {\\text{stu}} \\leq 1 - \\epsilon)  $, resulting in the zero-sum setup as well. The motivation is that if the model always gets the solution right, it is not worth ``learning'' anymore.\n\nWe believe the core disagreement stems from different philosophical perspectives. Your critique appears to adopt a **cooperative** learning framework, where the teacher helps the student learn by finding the optimal \"Goldilocks zone\" of difficulty. However, from our perspective, we intentionally design HAP from an **adversarial** angle with a dynamic equilibrium where the teacher continuously raises the bar as the student improves, rather than seeking a static \"optimal difficulty\". **The teacher's objective is not only providing data for the student to learn, but also to generate increasingly challenging problems that push the boundaries of what the student can solve, such that the generated problems themselves are hard and valuable, rather than assisting the students to learn only**. The student, in turn, aims to master these progressively difficult tasks. This creates a bootstrapping dynamic where the teacher explores a larger space of challenging problems and the student continuously adapts to solve harder tasks. **The ultimate goal is not cooperative learning assistance, but rather the emergence of a problem proposer capable of generating highly challenging tasks, and a solver that can evaluate and overcome these difficult problems**.\n\nWe acknowledge that this pure adversarial setup introduces training difficulties. Ideally, the zero-sum formulation creates a dynamic equilibrium where the teacher finds tasks that maximally challenge the student's current capabilities. But for teachers like simple probability teacher, there is indeed no such guarantee. We do encounter cases when training collapses due to pathological teacher task selection, so we further introduce entropy regularization and cold-start policies to help avoid these cases. See the table for the ablation results for these modules. These training difficulties are natural in this adversarial setup. Like in self-play of AlphaZero, it's also possible that one agent leverages a weakness spot of its opponent and somehow stuck learning. However, there is exploration and exploitation and MCTS search tree to help making progress even such a weakness spot was found. But these design relate to the techical methods to mitigate training difficulties. Philosophically, they should be adversarial.\n\n| Configuration | Easy | Middle | Hard | General |\n|---------------|------|--------|------|---------|\n| Original model | 0.92 | 0.46 | 0.20 | 0.527 |\n| w/o entropy regularization | 0.91 | 0.38 | 0.11 | 0.467 |\n| w/o cold start | 0.92 | 0.45 | 0.20 | 0.523 |\n| w/o lower bounds | 3/10 converged | 1/10 converged | 0.21 | - |\n\n(For the table: Entropy regularization proves crucial for hard task performance. Without it, we observe significant performance degradation on harder tasks. Cold start primarily affects training efficiency rather than final performance, influencing convergence speed during the initial training phase. Lower bounds are essential for training stability. Without them, models tend to get stuck on hard tasks and suffer from catastrophic forgetting of simpler tasks.)\n\n> The related work section missed out on some of the most relevant frameworks where the teacher and student engage in zero-sum or asymmetric games.\n\nThank you for bringing these important works to our attention. All of these studies will be incorporated into our revised Related Work section.\n\n> All three are grid-based environments with similar visual representations and discrete action spaces. There is no evaluation on continuous control tasks like robotics, physics simulations. All are visual grid worlds - no language-only or multimodal tasks.\n\nRegarding the scope and diversity of our experiments, we further evaluated HAP on the continuous control task parametric-continuous-stump-tracks-v0, based on the ACL benchmark TeachMyAgent [1]. TeachMyAgent leverages procedurally generated environments to assess the performance of teacher algorithms. To accommodate HAP’s implementation, we reduced the continuous teacher action space to 10 sampled tasks. Due to time constraints, our experiments were limited to the \"fish\" embodiment and employed \"ppo\" as the student algorithm. The results are as follows:\n\n| Method | eval_reward_max (5e6) | eval_reward_mean (5e6) | eval_reward_max (1e7) | eval_reward_mean (1e7) |\n|--------|----------------------|------------------------|----------------------|------------------------|\n| Random | 86.5 | -18 | 82.8 | -11 |\n| Self-Paced | 15.3 | -4.63 | 77.4 | -23.4 |\n| GoalGAN | 73.8 | 2.86 | 31.1 | 5.35 |\n| Setter-Solver | 124 | 6.6 | 107 | 24.3 |\n| HAP | 63.8 | 4.34 | 72.4 | 5.46 |\n\nHAP achieves competitive performance and beats ACL methods like GoalGAN and Self-Paced in continuous control tasks with robotics and physics simulations (Setter-Solver has extra pre-knowledge for the solver so the result is significantly better). Tasks with continuous space actually have no difference in the algrithim structure compared with discrete space.  \n\n[1] Romac, Clément, et al. \"Teachmyagent: a benchmark for automatic curriculum learning in deep rl.\" ICML.   \n\nFor \"no language-only or multimodal tasks,\" we are not entirely certain if this refers to tasks where only language is used (without images) or to more complex settings involving multiple modalities (e.g., visual+language). For completeness, we note that HAP has also been evaluated on both CV and NLP tasks under the supervised learning setting (see Section 6.1). We will further clarify this point in the revision.\n\n> Insufficient human evaluation. Although I appreciate it that the authors did human experiments to validate the effectiveness of HAP at creating curriculum for humans, the sample size is very limited and the choice of task is debatable. It’s unclear what aspect of human learning is captured by teaching Prolific workers Minigrid, ...\n\nThank you for your thoughtful feedback. Our human experiment is an initial step toward evaluating HAP's applicability for real-world curriculum design. Our goal in this study was to provide a controlled, proof-of-concept demonstration that HAP-generated curricula exhibit pedagogical properties beneficial to human learners—such as revisiting foundational challenges during plateaus and adapting task difficulty to learner progress. Regarding sample size, we recognize the limitations of our current study. We note, however, that sample sizes of 20–30 participants are common in early-stage psychological and educational research, especially when studies are time-consuming or require careful supervision. The results were statistically significant with p < 0.0001 (compared with w/o tutorial), providing meaningful evidence for our claims. Nonetheless, we acknowledge that larger and more diverse cohorts are important for drawing generalizable conclusions. \n\n> What exactly is the significance of heterogeneity in heterogeneous adversarial play? And how does it differ from \"asymmetric\"?\n\n“Heterogeneity,” especially as commonly defined in multi-agent and federated learning literature (e.g., [1]), emphasizes agents with fundamentally different roles, capabilities, and action/observation spaces. In our framework, the teacher and student are not merely engaged in the same task from different perspectives; rather, they operate in genuinely distinct domains—one generates problems and the other solves them. In contrast, “asymmetry,” as in asymmetric self-play [2], often involves agents with related (sometimes swapped) roles but still share comparable skill sets, experiences, or objectives—for instance, a single agent alternately acting as problem setter and solver in similar tasks. Our design adopts a genuinely **adversarial** perspective rather than **cooperation** in self-play: the teacher’s objective is not only explicitly to help the student learn but to continually propose more challenging tasks, pushing the boundaries of the student’s capabilities. Meanwhile, the student’s objective is to learn to solve tasks of ever-increasing difficulty. By using the term “heterogeneous,” we foreground the structural and functional differences between our teacher and student, which are much more pronounced than in standard asymmetric setups.\n   \n[1] Zhong, Yifan, et al. \"Heterogeneous-agent reinforcement learning.\" JMLR.   \n[2] Sukhbaatar, S., et al. (2018). Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play. ICLR.   \n\nThank you for your insightful review. We sincerely welcome your feedback."}, {"review_id": "TQ9yYys8iL", "reviewer": "Reviewer_wSJi", "summary": "This paper introduces Heterogeneous Adversarial Play (HAP), a novel curriculum learning framework inspired by the human teacher-student paradigm. Unlike traditional self-play or static curriculum learning, HAP features a teacher agent that adversarially selects tasks the student agent is likely to fail at. The teacher is rewarded when the student fails, and vice versa, forming a zero-sum game that naturally drives the student to improve.", "questions": "- Could you clarify the implementation details for the curriculum learning baselines (e.g., TSCL, EXP3)? Without sufficient methodological description, it’s difficult to assess whether the comparisons are fair and reproducible.\n- Why was there no baseline that follows a manually ordered easy-to-hard curriculum? While real-world task difficulty may not be explicit, such a baseline would serve as a useful reference point for evaluating HAP’s effectiveness.\n- The paper briefly mentions that HAP is applied to supervised tasks (e.g., CIFAR-100, RTE), but provides little detail on how the framework is adapted in those domains. Could you elaborate on the methodology used in those experiments?\n- In the experiments, different RL algorithms (DQN, PPO, SAC, etc.) are used as the student across environments. Does HAP consistently perform well regardless of the student’s architecture? How agnostic is it to the underlying RL framework?\n- How does HAP scale with increasing task space size or task granularity? Have you evaluated whether its effectiveness changes as the number of candidate tasks grows?\n- You mention entropy regularization in the teacher’s loss to mitigate underfitting. Is there any ablation study to quantify its impact on training stability and performance?\n- In Figure 2(b), all task difficulties appear to improve almost simultaneously. Could you explain why this occurs?\n- Could you provide further interpretation of the teacher’s task selection behavior and the student’s learning trajectory across Minigrid, CRAFT, and Crafter, similar to the detailed breakdown shown in Figure 2?\n- Is HAP applicable only to pre-defined, static sets of tasks? How might it be extended to open-ended environments?\n- You describe a simplified “Simple Probability Teacher” using student history to guide task selection. Have you conducted any ablation to show whether including history is necessary or beneficial?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "1. Strengths\n- HAP formulates curriculum learning as a zero-sum game between a teacher and student, reflecting real-world pedagogical dynamics, that is realized through an adversarial learning framework.\n- Unlike prior methods requiring handcrafted task orders or heuristics, HAP autonomously adapts task difficulty based on the learner’s capability.\n\n2. Weaknesses\n- While real-world tasks often lack clear difficulty labels, the absence of a manually ordered (easy-to-hard) curriculum baseline limits the evaluation of HAP, as it omits a fundamental reference point for measuring the benefits of automated curriculum learning.\n- In addition, the lack of clear descriptions for how curriculum learning baselines (e.g., TSCL, EXP3) are configured undermines the transparency of the experimental setup, making it difficult to assess whether the comparisons are fair and valid.\n- The application of HAP to supervised learning tasks is only briefly mentioned, with insufficient methodological detail, leaving it unclear how the adversarial setup is adapted beyond the reinforcement learning context.\n- Since the student agent varies across experiments in terms of RL backbone (e.g., DQN, PPO, SAC), it remains uncertain whether HAP’s effectiveness is consistent regardless of the underlying learner, raising questions about its framework-agnostic generality.", "was_revised": true, "final_justification": "All of my questions and concerns have been addressed comprehensively, and the clarifications provided have significantly improved my understanding of the work. I have also reviewed the questions and feedback from other reviewers, as well as the authors’ responses to them, which further reinforced my positive assessment. I will revise my score positively to reflect this improved evaluation.", "rebuttal": "Dear Reviewer wSJi,  \nThank you for your thoughtful feedback. We try to answer your questions as follows:\n\n> While real-world tasks often lack clear difficulty labels, the absence of a manually ordered (easy-to-hard) curriculum baseline limits the evaluation of HAP, as it omits a fundamental reference point for measuring the benefits of automated curriculum learning.\n\nThank you for this important suggestion. In response, we have conducted additional experiments that include a manually ordered (easy-to-hard) curriculum baseline. Instead of training tasks one-by-one, we trained them group-by-group and, finally, trained on all tasks together to help mitigate the forgetting problem (although this issue remains significant). The results are presented in the revised manuscript and are summarized below:\n\n| ENV | Per task step | Easy (Average/Best) | Middle (Average/Best) | Hard (Average/Best) | General (manually-ordered/HAP) |\n|-----|---------------|---------------------|----------------------|---------------------|-------------------------------|\n| Minigrid | 1k | 0.67/0.97 | 0.12/0.32 | 0.20/0.20 | 0.33/0.527 |\n| Craft | 10k | 0.36/0.86 | 0.21/0.57 | 0.25/0.25 | 0.26/0.562 |\n| Crafter | 10k | 0.27/0.94 | 0.16/0.61 | 0.14/0.14 | 0.19/0.723 |\n\nWe didn't include this baseline in the original submission because manually designed curricula introduce their own potential for bias and inequality. For example, the effectiveness of a manual curriculum strongly depends on subjective choices in task ordering, switching policies, and definitions of difficulty, which can vary significantly between designers. We will add these discussions in the revised manuscript.\n\n> the lack of clear descriptions for how curriculum learning baselines (e.g., TSCL, EXP3) are configured undermines the transparency of the experimental setup.  \n> The application of HAP to supervised learning tasks is only briefly mentioned, with insufficient methodological detail, leaving it unclear how the adversarial setup is adapted beyond the reinforcement learning context.  \n\nFor curriculum learning baselines, they are mainly based on their official github repos. For the supervised setting, it is aligned with the standard supervised curriculum learning paradigm, in that it adaptively selects which samples to present to the learner based on their evolving performance. We formulate HAP task selection around label-wise performance: each label or class is considered a distinct sub-task. The teacher observes the student model’s evolving accuracy or loss per label and uses this information to decide which label(s) should be emphasized in subsequent training steps. The implementation of these baselines are based on the benchmark CurBench [1], following the default and standard hyperparameters. We will add further clarification to the baseline implementaitons and hyperparameters in revision.\n\n[1] Zhou, Yuwei, et al. \"CurBench: curriculum learning benchmark.\" ICML.\n\n>  it remains uncertain whether HAP’s effectiveness is consistent regardless of the underlying learner, raising questions about its framework-agnostic generality.\n\nWe have described the RL backbone and model architectures used in HAP for each experiment in Supp Section C. In all three environment settings, HAP demonstrates improved performance compared to baselines using the same RL backbone. The effectiveness of HAP is definitly linked to the choice of learner, as different learning algorithms possess varying capabilities and may benefit differently from curriculum adaptation. Our paper presents results showing that HAP consistently outperforms standard baselines when applied with both A2C and PPO learners. We will include additional experimental results with other baselines and learning algorithms in our revised manuscript.\n\n> Why was there no baseline that follows a manually ordered easy-to-hard curriculum? While real-world task difficulty may not be explicit, such a baseline would serve as a useful reference point for evaluating HAP’s effectiveness.\n\nWith a set of diverse tasks, intuitively, manually ordering all tasks from easy to hard becomes impractical and unlikely to reflect an optimal or even reasonable progression. The complexity and variety of the tasks mean that an imposed manual ordering is unlikely to generalize well or yield competitive performance. That said, based on reviewer feedback, we nevertheless implemented a simplified version of this baseline by grouping tasks in rough difficulty levels and training in group-wise order. See the discussion and results above.\n\n> How does HAP scale with increasing task space size or task granularity? \n\nScaling adversarial frameworks to very large task spaces does present significant challenges. In our work, we specifically address this issue by introducing a Cold Start mechanism and an additional entropy loss term, both of which help constrain the teacher's task selection trajectory and encourage balanced exploration.  \nEmpirically, we observe that our model (as described in Supp Section C) can effectively control the teacher’s task proposals when the candidate task set is around 10 or fewer. This can be further adjusted by tuning the hyperparameter λ in the entropy loss, which regulates the diversity of proposed tasks. However, we note that the optimal setup is influenced by both task complexity and the learning capacity of the student model.\n\n> You mention entropy regularization in the teacher’s loss to mitigate underfitting. Is there any ablation study to quantify its impact on training stability and performance?\n\nWe conducted comprehensive ablation studies to evaluate the impact of different components, running each configuration 10 times with different random seeds. The results on the Minigrid environment are shown below:\n\n| Configuration | Easy | Middle | Hard | General |\n|---------------|------|--------|------|---------|\n| Original model | 0.92 | 0.46 | 0.2 | 0.527 |\n| w/o entropy regularization | 0.91 | 0.38 | 0.11 | 0.467 |\n| w/o cold start | 0.92 | 0.45 | 0.2 | 0.523 |\n| w/o lower bounds | 3/10 converged | 1/10 converged | 0.21 | - |\n\nEasy tasks are consistently learnable across all configurations. Entropy regularization proves crucial for hard task performance. Without it, we observe significant performance degradation on harder tasks. Cold start primarily affects training efficiency rather than final performance, influencing convergence speed during the initial training phase. Lower bounds are essential for training stability. Without them, models tend to get stuck on hard tasks and suffer from catastrophic forgetting of simpler tasks.\n\n> In Figure 2(b), all task difficulties appear to improve almost simultaneously. Could you explain why this occurs?\n\nThe result is in fact an expected outcome given the simplicity of the tasks in this benchmark. As shown in our results, all models eventually converge, indicating that no task poses significant lasting difficulty for the learning agents. In this context, the primary differentiator is data/sample efficiency—that is, how quickly each method enables the learner to master the tasks. With HAP as the curriculum method, the teacher transitions to harder tasks once the easier ones have been sufficiently learned, accelerating progress.  \nThe appearance of concurrent improvement across tasks in Figure 2(b) is also influenced by the sparsity of our evaluation checkpoints for HAP, which may make progress across tasks appear more simultaneous at the sampled points. Figure 2(c) is a more granular view of the learning dynamics where step-wise performance trends are shown in greater detail.\n\n> Could you provide further interpretation of the teacher’s task selection behavior and the student’s learning trajectory across Minigrid, CRAFT, and Crafter?\n\nDue to the much larger number of tasks and the significantly longer training sequences in these environments, presenting the full breakdown at a similar level of granularity becomes impractical and potentially confusing. In the revision we will consider including simplified or aggregated illustrations that highlight key aspects of the curriculum dynamics and learning progress for these more complex environments. \n\n> Is HAP applicable only to pre-defined, static sets of tasks? How might it be extended to open-ended environments?\n\nWe have not yet directly applied HAP to truly open-ended environments, and we acknowledge that significant changes or expansion in task distribution could introduce new challenges, especially for the teacher module—particularly as the number or diversity of possible tasks grows and novel tasks emerge dynamically. A possible solution for such settings is to equip the teacher with mechanisms for continual exploration and adaptation, such as leveraging generative models to propose new tasks or using meta-learning strategies to rapidly adapt the teacher’s proposal policy as the task landscape shifts. \n\n> You describe a simplified “Simple Probability Teacher” using student history to guide task selection. Have you conducted any ablation to show whether including history is necessary or beneficial?\n\nIn all of our experiments, the teacher utilizes student reward history, as it is important not only for assessing the current state but also for understanding learning trajectories and longer-term trends in student performance. We report the result without using student history below:\n\n| ENV | Easy | Middle | Hard | General |\n|-----|------|--------|------|---------|\n| Minigrid (with last 1k history) | 0.92 | 0.44 | 0.18 | 0.510 |\n| Minigrid (with last 100 history) | 0.92 | 0.46 | 0.20 | 0.527 |\n| Minigrid (w/o history) | 0.92 | 0.43 | 0.11 | 0.487 |\n\nThese results show that incorporating a certain amount of history is beneficial; however, using too much history may not accurately reflect the student's current learning status and can lead to decreased performance.\n\nThank you for your insightful review. We sincerely welcome your feedback."}, {"review_id": "cBkwpSYa2J", "reviewer": "Reviewer_fmid", "summary": "This paper brings the idea of adversarial play into curriculum learning and proposes Heterogeneous Adversarial Play. In HAP, the model alternates roles, acting as both a teacher and a student. It selects tasks of appropriate difficulty and then solves those tasks. Experiments show that HAP consistently outperforms relevant state-of-the-art baselines.", "questions": "1. Have you conducted ablation on some key components, such as cold-start, entropy regularization, and task selection lower bound?\n2. What is the teacher’s update frequency vs student steps? Did you try asynchronous updates?\n3. How sensitive is the teacher’s task-selection capability to the data budgets?\n4. Why does HAP perform poorly on the Minigrid task?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "strengths：\n1. The paper incorporates the concept of adversarial play into curriculum learning, enabling more targeted task selection and joint optimization of both teacher and student models.\n2. The preliminary experiment clearly demonstrates the advantages of HAP in a controlled setting.\n\nweakness：\n1. HAP appears to underperform on easy tasks, as shown in Table 1, limiting its generalizability.\n2. Missing reference in Line 256 for the TSCL baseline.\n3. The paper lacks a detailed analysis of learning trajectories, despite claiming (Lines 80–84) that HAP enables phased skill reinforcement and context-aware difficulty scaling. It would be insightful to compare how other curriculum learning baselines adapt task selection over time in Figure 2(c).\n4. Lack a detailed description of the optimization process of teacher and student models.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fmid,  \nThank you for your thoughtful feedback. We try to answer your questions as follows:  \n\n> HAP appears to underperform on easy tasks, as shown in Table 1, limiting its generalizability.\n\nWe don't see this as an weakness. It is indeed expected that HAP may underperform or show no significant advantage on easy tasks, as HAP does not modify the underlying RL backbone or introduce new model capacity; rather, it focuses on adaptive curriculum sequencing to facilitate learning, particularly in more complex or diverse task environments. For simple tasks where standard RL algorithms already learn efficiently without advanced curriculum scaffolding, HAP does not necessarily provide a theoretical guarantee of improvement, but as shown the difference is not that significant.\n\n>  Missing reference in Line 256 for the TSCL baseline.\n\n[1] Matiisen, T., Oliver, A., Cohen, T., & Schulman, J. (2019). Teacher–student curriculum learning. IEEE transactions on neural networks and learning systems, 31(9), 3732-3740.\n\n> The paper lacks a detailed analysis of learning trajectories, despite claiming (Lines 80–84) that HAP enables phased skill reinforcement and context-aware difficulty scaling. It would be insightful to compare how other curriculum learning baselines adapt task selection over time in Figure 2(c).\n\nThank you for this suggestion. We have actually included HAP's task selection trajectory in Figure 2(c), which demonstrates our method's phased skill reinforcement and difficulty scaling. We chose not to include all baseline methods' task selection trajectories in the same figure, as displaying multiple overlapping curves would create visual chaos and make it difficult to discern the key patterns. You can always run baselines and log their task selection with different baselines. We consider adding this part to the Supp in revision.\n\n> Lack a detailed description of the optimization process of teacher and student models.\n\nWe kindly refer you to the Supp Sec B for more details. \n\n> Have you conducted ablation on some key components, such as cold-start, entropy regularization, and task selection lower bound?\n\nYes. We conducted comprehensive ablation studies to evaluate the impact of different components, running each configuration 10 times with different random seeds. The results on the Minigrid environment are shown below:  \n\n| Configuration | Easy | Middle | Hard | General |\n|---------------|------|--------|------|---------|\n| Original model | 0.92 | 0.46 | 0.2 | 0.527 |\n| w/o entropy regularization | 0.91 | 0.38 | 0.11 | 0.467 |\n| w/o cold start | 0.92 | 0.45 | 0.2 | 0.523 |\n| w/o lower bounds | 3/10 converged | 1/10 converged | 0.21 | - |\n\nEasy tasks are consistently learnable across all configurations. Entropy regularization proves crucial for hard task performance. Without it, we observe significant performance degradation on harder tasks. Cold start primarily affects training efficiency rather than final performance, influencing convergence speed during the initial training phase. Lower bounds are essential for training stability. Without them, models tend to get stuck on hard tasks and suffer from catastrophic forgetting of simpler tasks.\n\n> What is the teacher’s update frequency vs student steps? Did you try asynchronous updates?\n\nAs described in Supp Sec C, the update frequency of the teacher is task-dependent and tuned for each experimental setting. We have also experimented with asynchronous updates, but observed no significant improvement in student learning performance. Additionally, updating the teacher typically requires a full evaluation of the student, which can introduce a time bottleneck, especially in more complex environments. As a result, we choose update intervals that balance adaptation speed with computational efficiency. We will further clarify these implementation details and their impact in the revised manuscript.\n\n> How sensitive is the teacher’s task-selection capability to the data budgets?\n\nIn all of our experiments, the teacher utilizes student reward history, as it is important not only for assessing the current state but also for understanding learning trajectories and longer-term trends in student performance. We report the result without using student history below:\n\n| ENV | Easy | Middle | Hard | General |\n|-----|------|--------|------|---------|\n| Minigrid (with last 1k history) | 0.92 | 0.44 | 0.18 | 0.510 |\n| Minigrid (with last 100 history) | 0.92 | 0.46 | 0.20 | 0.527 |\n| Minigrid (w/o history) | 0.92 | 0.43 | 0.11 | 0.487 |\n\nThese results show that incorporating a certain amount of history is beneficial; however, using too much history may not accurately reflect the student's current learning status and can lead to decreased performance.\n\n> Why does HAP perform poorly on the Minigrid task?\n\nNo. HAP outperforms the other baselines on the more challenging tasks (middle: 0.46 vs. 0.34; hard: 0.20 vs. 0.18). HAP only performs slightly worse on the easy tasks (0.92 vs. the best score of 0.98), which, as discussed above, can be explained by its focus on more challenging tasks.\n\nThank you for your insightful review. We sincerely welcome your feedback."}, {"review_id": "Vha0CChWvw", "reviewer": "Reviewer_FFm9", "summary": "This paper introduces Heterogeneous Adversarial Play (HAP), a novel adversarial learning framework inspired by human pedagogical dynamics, where a teacher agent generates tasks to challenge a student agent in a zero-sum game.  The work makes significant contributions to automated curriculum learning by addressing limitations of static curricula and unidirectional adaptation.  The theoretical grounding is solid, and experiments across grid-world and complex environments (Minigrid, CRAFT, Crafter) demonstrate HAP’s superiority over baselines in learning efficiency and task success rates.", "questions": "1. Could the authors discuss HAP’s potential bottlenecks (e.g., teacher network architecture) when applied to high-dimensional state spaces (e.g., pixel-based inputs)?\n\n2. How does HAP handle tasks with non-linear difficulty progression or combinatorial dependencies? An ablation study on CRAFT’s dependency graph could clarify this.\n\n3. The entropy coefficient $\\lambda$ and warm-up duration are critical but empirically set. A sensitivity analysis would strengthen robustness claims.", "limitations": "1. While HAP excels in grid-world and Minecraft-like environments, its performance on Crafter’s open-world tasks (general score 0.723 vs. human 0.85) suggests challenges in scaling to real-world complexity. The paper does not analyze computational costs or memory demands for larger tasks.\n\n2.  HAP assumes tasks can be ordered by difficulty (e.g., Minigrid’s \"simple\" to \"extremely hard\"). Its efficacy in environments with non-hierarchical or interdependent tasks (e.g., multi-modal RL) remains untested.\n\n3. The lack of theoretical guarantee of the  proposed methods.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. HAP creatively formalizes teacher-student adversarial dynamics as a zero-sum game, enabling automatic curriculum generation without human intervention. This addresses a critical gap in adaptive curriculum learning.\n\n2. Experiments across diverse environments (deterministic to stochastic) show consistent improvements over RL and curriculum learning baselines (e.g., +18% on hard tasks in CRAFT). \n\nWeakness\n\n1.The zero-sum game formulation assumes perfect adversarial dynamics between teacher and student. However, in practice, overly aggressive task generation (e.g., tasks with near-zero solvability) could destabilize training. The paper does not analyze conditions for Nash equilibrium or discuss failure modes (e.g., teacher overfitting to generate unsolvable tasks).\n\n2. HAP relies on implicit difficulty estimation through reward signals ($R(\\tau;C)$), but lacks a formal metric to define \"challenging yet learnable\" tasks. This contrasts with prior work (e.g., Graves et al., 2017) that uses explicit difficulty measures. Without this, the teacher’s curriculum may oscillate between trivial and impossible tasks.", "was_revised": true, "final_justification": "After considering the authors' rebuttal and the other reviewers' comments, I maintain my overall rating of 4.", "rebuttal": "Dear Reviewer FFm9,  \nThank you for your thoughtful feedback. We try to answer your questions as follows:\n\n> The zero-sum game formulation assumes perfect adversarial dynamics between teacher and student. However, in practice, overly aggressive task generation (e.g., tasks with near-zero solvability) could destabilize training. The paper does not analyze conditions for Nash equilibrium or discuss failure modes (e.g., teacher overfitting to generate unsolvable tasks).  \n> HAP relies on implicit difficulty estimation through reward signals, but lacks a formal metric to define \"challenging yet learnable\" tasks. This contrasts with prior work (e.g., Graves et al., 2017) that uses explicit difficulty measures. Without this, the teacher’s curriculum may oscillate between trivial and impossible tasks.\n\nWe acknowledge that we do not provide a formal analysis of conditions for Nash equilibrium in this particular teacher-student setting, as the equilibrium structure can indeed be complex and may depend on the properties of the environment, the expressiveness of the teacher and student models, and the specifics of the feedback and update mechanisms. \n\nFrom the formula, the zero-sum objective drives the teacher to find tasks where the student is most likely to fail but can still learn. Ideally, the zero-sum formulation creates a dynamic equilibrium where the teacher finds tasks that maximally challenge the student's current capabilities. For the simple probability teacher, there is indeed no such grantee. We do encounter some cases like you mentioned when training collapses due to pathological teacher task selection, so we further introduce improvements like entropy regularization and cold-start policies to help avoid these cases. See the table below for the ablation results for these modules. Like in conventional RL, there is always exploration and exploitation and the tasks considered happen to be solvable by the agents. But we acknowledge that when scaling to more complex environments, design issues need revisiting.\n\n> Could the authors discuss HAP’s potential bottlenecks (e.g., teacher network architecture) when applied to high-dimensional state spaces (e.g., pixel-based inputs)?\n\nFrom our point of view, applying HAP to high-dimensional state spaces introduces several potential bottlenecks but particularly regarding the teacher network's efficiency. In high-dimensional environments, the teacher must process and reason about complex student observations and performance metrics. A more expressive teacher architecture (e.g., with convolutional or attention-based modules) is often required to capture relevant task and state information, which in turn increases computational cost and may slow down both training and adaptation. The teacher’s signal for what constitutes a good task for the student may become weaker or noisier in high-dimensional settings, requiring additional regularization or sophisticated feedback mechanisms to maintain stable and meaningful curriculum generation. Conventional probability-based methods or teachers relying on handcrafted loss designs may no longer suffice in these scenarios. Therefore, it may become necessary to leverage meta-learning techniques or other advanced approaches to train a more general and capable teacher.\n\n> How does HAP handle tasks with non-linear difficulty progression or combinatorial dependencies? An ablation study on CRAFT’s dependency graph could clarify this.\n\nHAP is designed to adapt to a learner’s evolving abilities by dynamically assigning tasks near the student’s competence frontier, rather than simply following a static or presupposed linear difficulty chain. This means that, in environments like CRAFT where tasks are structured with combinatorial dependencies, the teacher agent can give tutorials following some basic dependencies. We will add a graph in revision showing how HAP scheduling tasks for the student in CRAFT. \n\n> The entropy coefficient and warm-up duration are critical but empirically set. A sensitivity analysis would strengthen robustness claims.\n\nWe conducted comprehensive ablation studies to evaluate the impact of different components, running each configuration 10 times with different random seeds. The results on the Minigrid environment are shown below:\n\n| Configuration | Easy | Middle | Hard | General |\n|---------------|------|--------|------|---------|\n| Original model | 0.92 | 0.46 | 0.2 | 0.527 |\n| w/o entropy regularization | 0.91 | 0.38 | 0.11 | 0.467 |\n| w/o cold start | 0.92 | 0.45 | 0.2 | 0.523 |\n| w/o lower bounds | 3/10 converged | 1/10 converged | 0.21 | - |\n\nEasy tasks are consistently learnable across all configurations. Entropy regularization proves crucial for hard task performance. Without it, we observe significant performance degradation on harder tasks. Cold start primarily affects training efficiency rather than final performance, influencing convergence speed during the initial training phase. Lower bounds are essential for training stability. Without them, models tend to get stuck on hard tasks and suffer from catastrophic forgetting of simpler tasks.\n\n> While HAP excels in grid-world and Minecraft-like environments, its performance on Crafter’s open-world tasks (general score 0.723 vs. human 0.85) suggests challenges in scaling to real-world complexity. The paper does not analyze computational costs or memory demands for larger tasks.\n\nThank you for raising the important point. The overall computational cost of training in our framework is primarily determined by the student policy’s update frequency, batch size, and episode length, as well as the size of the neural networks. Importantly, for each teacher update, the student must complete $N_{evalbatch}$ evaluation batches, which further adds to the computational cost. The total cost per interval of teacher update can thus be summarized as:\n$$\nCompute Cost_{total} ∝ N_{batch} × N_{steps} × P_{student} + N_{evalbatch} × N_{evalsteps} × P_{student} + P_{teacher}\n$$\nwhere $N_{batch}$ is the training batch size, $N_{steps}$ is the episode length, $N_{evalbatch}$ is the number of evaluation batches per teacher update, $N_{evalsteps}$ is the number of steps per evaluation batch, $P_{student}$ is the number of parameters in the student policy, and $P_{teacher}$ is the teacher’s parameter count. Taking the minigrid experiment as an example, the teacher policy, implemented as a lightweight MLP and updated every 100 episodes, introduces minimal overhead compared to student training and rollout collection. For memory usage, key contributors include the student and teacher network parameters, as well as storage for trajectory data and the teacher’s history buffer:\n$$\np_{memory} ≈ sizeof(P_{student}) + sizeof(P_{teacher}) + (N_{batch} × N_{steps} + N_{evalbatch} × N_{evalsteps}) × d\n$$\nwhere d is the storage size per step (observation + action + reward per task, etc.).\n\n> HAP assumes tasks can be ordered by difficulty (e.g., Minigrid’s \"simple\" to \"extremely hard\"). Its efficacy in environments with non-hierarchical or interdependent tasks (e.g., multi-modal RL) remains untested.\n\nNo, HAP has no assumption about task dependency or difficulty (line 131-136). The difficulty shown in tables was for illustrative purposes. To demonstrate that, we further test our model and compare the result with other CL baselines in a continuous motion_control task parametric-continuous-stump-tracks-v0. We kindly refer you to our response to reviewer EgTN.\n\n> The lack of theoretical guarantee of the proposed methods.\n\nWe have to acknowledge that deriving comprehensive theoretical guarantees for adaptive adversarial curricula in complex domains remains challenging. We try to admit a foundational analysis based on properties of stochastic approximation and policy gradient methods. Recall our setting:  \n\n$$ \\min_{\\phi} \\max_{\\theta}\\ J(\\theta, \\phi) = \\mathbb{E}\\left[ R(\\tau; T) \\right]$$\n\nProvided that:\n- The teacher and student policies are updated with sufficiently small learning rates (ensuring stability under stochastic approximation),\n- $J(\\theta, \\phi)$ is differentiable and gradients are unbiased estimators,\n- The parameter space is compact or regularized,\n\nthen the update scheme converges to a local minimax point or egalitarian equilibrium of $J$ where  \n\n$$\n\\nabla_\\theta J (\\theta^\\*,\\phi^\\*) = 0, \\nabla_\\phi J (\\theta^*,\\phi^\\*) = 0\n$$\n\nAt this point, neither the teacher nor the student can locally improve their objective by small, unilateral changes, reflecting a stable adversarial curriculum. The student learns maximally under the hardest (yet solvable) curriculum the teacher can assemble, given both agents' functional capacities, and The teacher cannot further reduce the student's success without stepping outside the feasible task distribution.\n\nHowever, this guarantee pertains to local stationary points, not necessarily global minimax solutions (which is standard in deep adversarial learning). The overall quality and pedagogical value of the equilibrium depend on the capacity of the networks and the granularity of the task space.\n\nThank you for your insightful review. We sincerely welcome your feedback."}]

Author rebuttal:

Meta-review: This paper introduces Heterogeneous Adversarial Play (HAP), a novel curriculum learning framework inspired by the human teacher–student paradigm. Overall, it is a technically solid work that has received consistent positive ratings from the reviewers. One reviewer (fmid) noted that their core concern about the method’s underperformance on easy tasks—potentially limiting its general applicability despite the authors’ justification—remains unresolved. However, since this comment was raised after the reviewer–author discussion period, I have placed less weight on it. In conclusion, this is a technically strong paper whose positive contributions to the field outweigh its limitations. Therefore, I recommend acceptance.
Decision: Accept (poster)