In the ZONE: Measuring difficulty and progression in curriculum generation
Authors: Rose E Wang
Jesse Mu
Dilip Arumugam
Natasha Jaques
Noah Goodman
Venue: Submitted to ICLR 2023 (2023)
Keywords: curriculum learning
multiagent
Bayesian

Abstract: A common strategy in curriculum generation for reinforcement learning is to train a teacher network to generate tasks that fall within a student network's ``zone of proximal development'' (ZPD). These are tasks that are not too easy and not too hard for the student. Albeit intuitive, ZPD is not well understood computationally. We propose ZONE, a novel computational framework that operationalizes ZPD. It formalizes ZPD through the language of Bayesian probability theory, revealing that tasks should be selected by difficulty (the student's success probability on the task) and learning progression (the degree of change in the student's model parameters). ZONE operationalizes ZPD with two techniques that we apply on top of existing algorithms. One is REJECT, which rejects tasks outside a difficulty scope and the other is GRAD, which prioritizes tasks that maximize the student's gradient norm. Compared to the original algorithms, the ZONE techniques improve the student’s generalization performance on discrete Minigrid environments and continuous control Mujoco domains with up to $9 \times$ higher success. ZONE also accelerates the student's learning by training on up to $10\times$ less data.

Under review as a conference paper at ICLR 2023
IN THE ZONE: MEASURING DIFFICULTY AND PROGRESSION IN CURRICULUM GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
A common strategy in curriculum generation for reinforcement learning is to train a teacher network to generate tasks that enable student learning, But, what kind of tasks enables this? One answer is tasks belonging to a student's zone of proximal development (ZPD), a concept from developmental psychology. These are tasks that are not too easy and not too hard for the student. Albeit intuitive, ZPD is not well understood computationally. We propose ZONE, novel computational framework that operationalizes ZPD. It formalizes 7PD through the language of Bayesian probability theory, revealing that tasks should be selected by difficulty (the student's probability of task success) and learning progression (the degree change in the student's s model parameters). ZONE instantiates two techniques that enforce the teacher to pick tasks within the student's s ZPD. One is REJECT, which rejects tasks outside of difficulty scope, and the other is GRAD, which prioritizes tasks that maximize the student's gradient norm. We apply these techniques to existing curriculum learning algorithms. We show that they improve the student's generalization performance on discrete MiniGrid environments and continuous control MuJoCo domains with up to 9x higher success. ZONE also accelerates the student's 's learning by training with 10x less data.
INTRODUCTION
Many reinforcement learning (RL) problems require designing a distribution of tasks to train effec- tive policies for the real-world (Taylor & Stone 2009). However, designing the full space of tasks challenging; the real world complicated and specifying every edge task impractical or even infeasible for certain domains (Wang et al.) 2019 Parker-Holder al. 2022b). These tasks might also be onerous for the agent to solve without the provision of scaffolding. Training the RL agent on all possible tasks is intractable under limited training budget (Schmidhuber 2013 Narvekar Zeng
jet al. 2022 Florensa et al.) 2018a).
The state-of-the-an art approaches to this problem use multi multi-agent curriculum generation algorithms for automating task generation. A teacher agent learns to generate tasks judiciously to train student agent (Dennis et al. 2020 Du al.] 20022] Campero et al., 2020; Florensa et al.. 2018a| Portelas eet al. 2020; Matisen et al. (020). The teacher is rewarded by the difficulty of tasks it generates. Even though prior methods share intuitions on the teacher objective, there no framework for understanding the kind of tasks that best enables student learning and how the teacher should be rewarded to this end. These challenges suggest that we should re- re-assess how we formalize and operationalize the objective for the teacher.
Thus, our work is interested in the following question: What kind of tasks should the teacher gener- ale and how should the teucher be rewarded? From the lens of developmental psychology, an answer that the teacher should be incentivized to generate tasks within the student's 's zone of proximal de- velopment (ZPD) (Vygotsky & Cole 1978; Vygotsky| 2012 Cole et al. 2005 Shabani al. 2010). These tasks have two propertics: They are within the student's 's difficulty and accelerate the student's s learning progression. Albeit intuitive and widely known, ZPD lacks computational framework. This makes operationalizing ZPD in the teaching setting difficult.
Our work proposes ZONE as computational framework that operationalizes ZPD. ZONE isolates the two properties of ZPD tasks into techniques for enforcing the teacher to generate within a student's ZPD. The first technique is REJECT, which omits training tasks that fall outside the student's

---

Under review as a conference paper at ICLR 2023
current ability zone- this is akin to rejection sampling. The second technique is GRAD, which rewards the teacher for generating tasks that maximize the norm of the student network's ' gradient. These are tasks which induce the largest changes in the student's ' current model. We apply these Sasorthhss-PARREn
techniques to two popular curriculum generation
(Dennis et al., (020) and Goal GAN (Florensa al. 2018b)- and show that ZONE accelerates student learning on suite of discrete and continuous environments, We then investigate how these two techniques impact the student's ss and teacher's ' learning.
In summary, our work's contributions are the following:
1. We propose ZONE, a novel computational framework that formalizes the zone of proximal development (ZPD) Vygootsky & Cole, 1978) with Bayesian probability theory.
2. ZONE operationalizes ZPD with two techniques: REJECT, which rejects tasks that fall outside the student's difficulty range, and GRAD, which rewards the teacher for generating tasks that maximize the student's 's gradient norms.
3. We show that REJECT and GRAD improve the student's generalization performance and learning speed across a variety of discrete and continuous environments.
4. We investigate how the ZONE techniques improve the teacher's ability to generate ZPD tasks for the student.
2 RELATED WORKS
Zone of proximal development (ZPD) The objective of teacher is to help student learn effec- tively and to maximize the student's long-term performance. Prior work in developmental psychol- ogy and personalized education argue that teacher should aid students in problems that fall within their zone of proximal development (ZPD) (Vygotsky & Cole 1978). ZPD problems are ones that not too easy (where the student doesn't need the teacher's assistance), and ones that not too difficult (where the student couldn't 't solve even with the teacher's assistance). Aiding students on ZPD problems how students would most benefit from the teacher's scaffolding, such as through a curriculum (Warford 2011 Wass & Golding, 2014). Vygotsky's idea has shaped how we think about teaching human students in different domains, like teaching the sciences (Chounta et al, 2017 Vainas et al.) 2019) foreign languages (Mu et al., (02I). It has also shaped our understanding of how young infants acquire knowledge over time (Dizz et al. 1991 Wass & Golding 2014).
Teacher-student curriculum generation Teacher-student curriculum generation is a long- standing paradigm for accelerating training and generalization RL agents (Matiisen et al. 2020
Sukhbaatar et al. [2018] FFlorensa et al. 2017;2201 2018b] Zhang et al. 20200] Parker-Holder et al. 2022b
Dennis et al. 2020 Jiang et al.] 2022b Portelas et al. 2020) Fang et al. 2020; (ooviayy et al. 2022).
These algorithms reward the teacher based on measures of difficulty, such as the student's ' return.
Despite this similarity, there is little work on formalizing what make up good task generations and how to align the teacher's 's objective to this end. This is contrary to work developmental psychol- ogy that suggest pedagogical interventions via the teacher based on the student's needs and progress.
Our work focuses two popular curriculum generation algorithms: PAIRED (Dennis et al.) 2020) and Goal GAN (Florensa el al. 2018b). PAIRED is a successful regret-based algorithm applied to discrete domains, where adversarial teacher generates tasks that maximize regret between student and a competing student (the anti-student). Goal GAN is a similarly successful algorithm designed for continuous control domains where generative adversarial network (GAN) (Good- fellow et al., (2020) teacher generates 2D navigation goals. We focus on these algorithms because several works build on them as either a competitive baseline or the basis of their own algorithms (Jang et al.) 2021a, Gur et al. 2021 Du et al.) 2022, Paaker- Holder et al.] [022a]] Zhhnn et al.) 2020). Additionally, the algorithms cover different properties prior work, such as domains (dis- crete vs. continuous), difficulty criteria (static vs. dynamic), and teacher objectives (regret vs. non-regret). Prior intrinsic reward methods can alternatively enable generalization where additional reward supplements the extrinsic reward to incentivize exploration of the environment (Campero) jet al. 2020 Zhang et al.] 202T] Raileanu & Rocktäschel] 2019). However, our work focuses on the teacher determining the student's curriculum and considers these methods out of scope.
2

---

Under review as a conference paper at ICLR 2023
Tercher
Medium
(A) REJECT
Easy
GRAD
(c) Training loop
(a) Difficulty (REJECT)
(b) Progression (GRAD)
ZONE consists of components ensuring the teacher generates tasks within the student's zone proximal development. (a) REJECT, where tasks outside a difficulty range are omitted from training. Tasks that are too hard (in red) and too easy (in green) are rejected. It keeps tasks c², c². (b) is GRAD where the teacher selects tasks that induce large changes in the student's 's model. The plot is an illustrative example of the model space in R.. The student's current model is 8. The student's model after updating after updating on task c 0". c induces larger change than c², thus the teacher prefers d² task ca is 8 and situates both student's
Figure 1:
2ONE components in typical teacher-student training loop. REJECT rejects tasks " based reward cun and GRAD encourages the teacher to generate tasks that maximize the student's gradient norm.
ZPD-based teaching teacher acting on 7PD generates tasks with two types of properties. The first property focuses difficulty: curriculum tasks are neither too easy nor too hard. In prior work like PAIRED (Dennis et al., 2020) and Goal GAN (Florensa et al. (018b), a teacher generates a batch of tasks and trains the student on the entire batch regardless of whether the tasks meet the difficulty criterion. Training the student on problems that are too easy and too hard slows down the student's learning. In contrast, our work proposes sample-efficient rejection sampling technique inspired by ZONE: we replace tasks that don't meet the difficulty criterion with those that do, and train on the resulting batch.
The second property focuses on progression: curriculum tasks accelerate the student's ability to generalize. In general, identifying such tasks prior to training the student is difficult. A proxy for identifying such tasks is picking those that induce large model changes in the student, such as the norm of the student's 's model gradients. Prior work in the supervised learning setting have employed the gradient norm as heuristic in active learning (Ash et al.) 2019; Jiang et al., [2019). Prior work intrinsic rewards for RL agents has used the gradient norm for learning better world models (Sun) et al. [201T] Houthoooft et al., 2016). However, to the best of our knowledge, the idea of maximizing the gradient norm student has not been used as a curriculum learning objective.
METHOD
The goal of teacher is generate tasks that lead to the student's success in the long-run while also accelerating the pace of student learning. This is challenging goal to convert into an objective: How does a teacher measure both the long-run performance and progression in selecting the best tasks? Our work is interested in mathematically formalizing the teacher's 's objective motivated by ZPD. The formalism motivates the techniques we use to improve existing curriculum generation algorithms. We first describe the teacher-student's learning setting. Then, we formalize ZONE, framework for describing the ideal teacher objective with the language of Bayesian probability theory. Finally, we describe how ZONE can improve existing curriculum generation algorithms.
3.1 TEACHER-STUDENE FORMALISM
Our work is concerned with agents that endeavor to learn optimal behaviors across multiple tasks
(Kaelbling 1993 Schaul et 2015). a problem we formulate as Contextual MDP (Brunskill & Li 2013 Hallak et al. 2015 Modi et al. 2018) (CMDP) given by M = (S, A,C. C, R.T, M. p. X)- Here, each task of interest is identified by an individual context contained in C while jointly sharing the same set of states S, set of actions A, and discount factor 7 E j0, 1). We will refer to contexts and tasks interchangeably. Each variation in the context induces a new infinite-horins horizon, discounted
Markov Decision Process (MDP) (Bellman) 1957 Puterman 1994) that may differ in the initial state
distribution μ + (SS), the (deterministic) reward function R C x S x A - [0, 1] providing evaluative feedback signals (in the unit interval), or the transition function T C S A + A(S) prescribing distributions over next states. X E ACC) denotes a distribution over contexts, ie. a task distribution.
3

---

Under review as a conference paper at ICLR 2023
The student is a policy parameterized by 0 Q S Rd and is mapping to distributions over actions given a state ne : S - ((A). The teacher is a distribution over contexts parameterized by Ø. Xa E ACC). At the beginning of an episode, the teacher samples a context as the student contends with learning optimal behavior in the resulting MDP (S,A, RR: Te: Me, He, 7)- At each discrete timestep E N, beginning with initial state 8o (e((), the student observes the current state E S, executes an action E A, observes a reward = Rs, az), and transitions to the next state S1+1
xx(() which is held fixed
Te(- St, az).
3.2 FORMALIZING THE ZONE: HOW SHOULD THE TEACHER BE REWARDED?
To help build intuition for ZONE we describe a one-step curriculum learning process using the language of Bayesian probability theory. The goal of the teacher is to learn to construct a curriculum over tasks C such that the student does well in the long run. Because care about generalization, model the student's ' long-run performance as its test success rate. R is a Bernoulli random variable that denotes success: If the student solves the task, R 1, otherwise R 0 (Levine 2018 We use C Re to denote the curriculum task and student's curriculum success, C Nest denote the test task and student's test success. Similarly, @uur Rd denotes the student's policy parameters at the beginning of the curriculum. The teacher's objective is learn task distribution X (C to maximize the student's expected success over a held-out task distribution:
()) 1|Ctest C cour geurr)
max (Crmn)
The objective does not yet reflect the student's learning update after training C To do so, introduce the random variable ue to model the student's 's curriculum success. We also introduce Glest, which is the student's policy parameters after updating. We apply the chain rule to expand Equation as follows:
max E xx [P (R (a(Cn
= 1|Ctest Cai geur))
(2)
max E Ba Xo (Ctam)
Σ
p(ltes r cure Oross (ree cum Oour dd0tes
(3)
yaur ER Rd
max Bam)xa()
p( 1|Ctest Oleat (le))p ( cor grun )p(r cum cum oour d
Xo (Crmn)
Eli
(4)
Σ E ge
E Exmaxo() (C
max
p(R 1| 8²) (p(0p( (o corr gear y( cun gonn d8
(5)
The transition from Equation 3 3oo Equation 4 Jsses conditional independence: The student's s test-time performance Ries conditioned on its updated model parameters glest is independent of Orur Huu. Moving from Equation to Inequality 5 Jsssmms that the teacher only cares about the student learn- ing optimal test parameters 6" O is the set of optimal test parameters that globally maximize the student's 's test performance p( Rlest mizing the student's 's test performance because the teacher's objective is to help the student perform successfully in the long run. We show Appendix (A that we can lower-bound Equation | with 5
((*) We assume the teacher only cares about maxi-
What does the decomposition in the final equation tell us about the teacher's role in the stu- dent's long-run performance? The teacher influences the difficulty of the student's curriculum p(r Jetum (fur). The teacher also influences the student's learning progression good test-time parameters, p(0" curr cour (our). finds tasks that accelerate the student's acquisition of optimal
'In Appendix E we empirically show that the choice of difficulty measure matters. For example, one cannot simply use the environment's 's dense reward function. The dense reward values are not always correlated with task success.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 ZONE (REJECT and GRAD) applied to standard curriculum generation loop.
Initialize student Ta, teacher Xp. and curriculum learning algorithm alg
while not converged do
K
Xx(C)
generate tasks D. evaluate tasks
MA=1
Grgure
K
+rollout (o (er )i=1)
(o 1x
K [ Ti-1 < alg. alg.criterion (fc R 12-1: 1 (15uur) 1:m)) AO <-train_stuuee Errain_student(Ta) (2aun) K (pour 1
K
> REJECT
X + train teacher(Xo. (2t K (Gourr) 1K 1==1)
D train student D. train teacher with GRAD
11 A0)
Ji=1:
end while
test parameters. ZONE elucidates why the teacher might want avoid tasks that are too easy and too hard. Easy tasks (high p P( = -lc (eurr) do not change the student's current model: if the student does not have a good model (0 * 0*), then picking an easy task will not increase the chances of the student learning better test model (low p(0" p( c geur)). The teacher should avoid picking hard tasks too: the probability of failure is high, and the student will likely need pro-
longed interaction with difficult task instances before learning good test model. In Appendix B we provide a more formal reinforcement learning analysis that recovers a similar decomposition to the above for the ZONE framework.
3.3 USING ZONE TO IMPROVE CURRICULUM ALGORITIMS
In general, it is hard to exactly estimate p( p lourr @ear)) and p(0"|r c (eurr) per task instance deliT, Below, we explain two ZONE techniques to make the formalism practical. These techniques
are illustrated in Figure land Algorithm aas simple additions to the teacher-student training loop.
Modeling p(r-urr c geur)) with rejection sampling (REJECT) ) Prior works including Goal GAN and PAIRED measure difficulty as the core objective. They typically approximate problem difficulty with the student's 's return. Broadly speaking, the algorithms reward the teacher for gen- erating tasks where the student achieves cumulative reward r E [r m In other words, the teacher satisfies the algorithm's 's difficulty criterion if the teacher does not generate problems that are too easy (r Tmax) and too hard (r < Tmin). This range may change (as in PAIRED) or be fixed (as in Goal GAN) throughout training. Rejecting tasks that fall outside the range ensures that the teacher generates tasks within a desired level of difficulty. We model REJECT as p(rsurr | geur) because the teacher influences the difficulty of the curriculum and task MDP. The procedure for REJECT is as follows: At every round, the teacher generates a batch of K tasks {ctuT) LK Ji=1- The student runs on the tasks and achieves the cumulative reward of (r:err K -- The rewards are checked against the algorithm's 's criterion; if the reward does not satisfy the criterion, the task is removed from the batch. Otherwise, it remains in the batch. Rather than let the teacher waste generated tasks and use smaller batch, we upsample the accepted tasks to replace the removed tasks") The student is trained on the resulting batch {2."m) K Prior works like Wang et al. (211 and Brant & Stanley heuristic improve evolutionary algorithms. Nonetheless, our work for- malizes REJECT with the ZONE analysis and shows that REJECT can be broadly useful for existing curriculum learning algorithms in the literature.
2017 use rejection
Modeling p(0* (u-urr, coure geurT) with maximizing gradient norms (GRAd) In maximizing the lower bound in inequality5] the teacher wants to generate tasks which move the student's ' parameters to a global maximizer 0*, On the one hand, knowing 0* or the set of global maximizers O' priori is a challenge. On the other hand, updating towards 0* is not possible unless the student's model is moving. Thus, we propose simple proxy that incentivizes the student's 's model to change. Namely, we reward the teacher for generating tasks that induce high model gradient norms in the student. To the best of our knowledge, the idea of maximizing the gradient norm of student has not been explored as a curriculum learning objective. Both Goal GAN and PAIRED use actor- critic algorithms, therefore the gradient norm is the norm of the actor and the critic gradients. In practice, GRAD calculates the norm of the student's gradient and adds it to the teacher's original
We are not running the teacher or the student more often when we do rejection sampling. The computa- tional requirements the same with or without ZONK.
5

---

Under review as a conference paper at ICLR 2023
reward from prior works: Preacher Foriginal + || || 12- The teacher's reward can also be casted as a weighted sum of the original reward and the student's gradient norm; our work does not explore the importance of tuning these weights. For dense reward settings, GRAD multiplies the two terms: Y Mleacher = Fariginal |- In dense reward settings, we found that Fariginal was often negative and much smaller in magnitude than the gradient norm. If we add the two terms, the teacher's s reward would be dominated by the gradient norm.
4 EXPERIMENTS
We now evaluate the ability of ZONE inform and improve existing curriculum generation algo- rithms. Specifically, we aim to answer the following research questions (RQs):
RQ1: How does ZONE impact the student's 's generalization performance and its learning progress?
Section 5.1 investigates impact of ZONE on the student's learning by applying ZONE on the curriculum generation algorithms a suite of discrete and continuous environments. We compare the held-out performance of the students with and without ZONE.
RQ2: How does ZONE impact the teacher's 's learning and task generations? Section 5.2 investigates this question by qualitatively and quantitatively comparing the task generations from ZONE teachers and non-2ONE teachers. We also compare the teachers on metrics that capture whether the teachers generate ZPD tasks and whether they learn generate ZPD tasks more easily.
Algorithms and applications of ZONE As previously mentioned, run ZONE techniques over two existing curriculum generation algorithms Goal GAN (Florensa et al., 2018b) and PAIRED (Dennis et al. 2020). We use each algorithm's default model settings and hyperparameters. Goal GAN's difficulty criterion is that the student achieves returns of E [0.1, 0.9] on the generated goals. This the interval chosen by prior work avoid problems that are too easy and too hard. The teacher is trained on labelled goals of whether the goals are within the difficulty range. Goal GAN+REJECT trains on tasks that are only within that difficulty range. Because the teacher is not trained with rewards, we do not apply GRAD to Goal GAN. PAIRED trains the teacher to maximize regret r'encher max(r- (r-x - 0), where v. is the reward of the student and r-s s is the reward of the competing student. As regret must be positive (Bell) 1982 [oomme & Sugden 1982 Lattimore & Szepesvári 2020). PAIRED's s difficulty criterion is that the student achieves returns where -- Ts > 0 on the generated tasks. In other words, PAIRED+REJECT trains the student on tasks where r, < [0, s] PAIRED+GRAD trains the student on all tasks, but rewards the teacher with Teeacer max(r-, rs, 0) + A on sparse reward settings and lieacber max(r-s - Ts, 0) | on dense reward settings. The student's gradient norm include both the gradients of the student's s actor and critic networks. PAIRED+REJEC
+GRAD trains the student on the tasks within the difficulty range, and rewards the teacher as in PAIRED+GRAD. More details on these algorithms can be found in the Appendix()
Environments We run each algorithm on its original domains. PAIRED's original domains are shown in Figure 2a 2b 2c 2d which are partially observable navigation tasks in the MiniGrid environ- ment Sdevalier-Boisveed et al. 2018). Goal GAN's original domains are shown in Figure 2e 21 2g) We also run both algorithms on additional MuJoCo continuous domains where students are tramedd within an in-domain region, and tested on a held-out region (Figure 2h 21 We use Du et al. 2022)* 's implementation for this setting. Note that in their implementation, the student is trained using dense rewards whereas the original algorithms are trained on sparse reward settings. Nonethe- less, the teacher still rejects tasks based the student's 's (binary) success, regardless of the exact numerical return achieved.
5 RESULTS
5.1 HOW DOES ZONE IMPACT THE STUDENT'S GENERALIZATION AND PROGRESS?
To test whether 2ONE improves prior generation algorithms, we apply the ZONE techniques to PAIRED and Goal GAN and measure the student's test-time performance. We mark the techniques

---

Under review as a conference paper at ICLR 2023
(a) 4 Rooms
(b) 16 Rooms
(c) Labyrinth
(d) Maze
(c) Goal Ant
(f) Maze Ant
(g) Maze
(h) Point
(i) Walker
(j) Reach
(k) Toss
Figure 2: PAIRED transfer environments (a-d) (Dennis et al., 2020), Goal GAN environments (e-g)
Flooresa al. 2018a), and additional MuloCo environments adapted from Du al. (2022) (h-k).
In the MuJoCo environments, the green-shaded area corresponds to the space of in-domain goals the
teacher can generate, and the gray-shaded area corresponds to the out-of-domain goals the student is tested on.
PAIRED
WAIRED+REJBG
PARRED+GRAD
PAIRED+REJECH GRAD
(a) Rooms
(b) Rooms
(c) Labyrinth
(d) Maze
Shur aod EG0
(e) Point mass
(f) Walker
(g) Toss
(h) Reach
Figure 3: PAIRED performance on MiniGrid transfer environments (a-d) and MuJoCo control envi- ronments (e-h). All the methods see the same amount of data per step; REJECT 's dropped samples are included in the learning curves.
REJECT in red, GRAD in blue, REJECT +GRAD in purple. The original algorithms marked in black in the graphs. We test PAIRED on its original transfer environments and MuJoCo environ- ments (Figure 3) and test Goal GAN its original test environments and MuloCo environments (Figure 4
The ZONE techniques propose tasks that improve and accelerate the student's performance majority the domains. the MiniGrid setting (top row of Figure PAIRED+REJECT discards problems that inhibit student learning in regular PAIRED. PAIRED+GRAD is slower find tasks that help the student generalizes, however matches REJECT 's performance over time (eg. in Sixteen Rooms, Labryinth and Maze). On all four domains, PAIRED+REJECT +GRAD performs at least as well as both individual techniques. This is most evident on Maze and demonstrates that REJECT and GRAD interact positively with each other.
In the control environments like the Goal GAN environments or MuloCo environments (bottom row of F and Figure(4). ZONE techniques provide slight improvements over the original algorithm. Goal GAN generally improves with REJECT. most noticeably on Walker, Maze, and Maze Ant in

---

Under review as a conference paper at ICLR 2023
Goal GAN
Goal
(a) Maze
(b) Maze Ant
(c) Goal Ant
New mncermitnii
Vontio Ece
Amaan
(d) Point
(e) Walker
(I) Toss
(g) Reach
Figure 4: Goal GAN performance on Goal GAN environments (a-c), and on the MuJoCo control
environments (d-g).
Figure Figure 3 on the dense reward MuJoCo environments, the student benefits from GRAD across all domains, however does not benefit from REJECT. We hypothesize REJECT is especially helpful in the sparse reward setting as that where learning a good policy is most challenging. REJECT is not helpful in dense reward setting like the MuJoCo environments because the agent learns on every trajectory, or in simple navigation environments like the Goal GAN environments.
Altogether, these results suggest that ZONE aids student' learning, particularly when the task struc- ture and the teacher's 's task generations are complex, such as the MiniGrid environment. In the MiniGrid setting, the teacher generates a task that is a 15 15 grid with up 50 walls. Small variations in its generations can result in large variations of the task difficulty. For example, placing a single wall piece the end of a corridor might block off direct route the goal for the stu- dent. By contrast, in the control environments, small variations the 2D goals would not make large difference the task difficulty.
5.2 HOW DOES ZONE IMPACT THE TEACHER'S ' TASK GENERATIONS AND LEARNING?
This section investigates how the teacher is impacted by ZONE by comparing the tasks gener- ated by the base algorithm and the ZONE variant. We focus on the MiniGrid setting because we see the largest improvements in this setting, and particularly compare between PAIRED and PAIRED+REJECT +GRAD. We include the analysis on the other algorithms and environments in Appendix C The section focuses two items. The first is an analysis of the task generations. The second is an analysis the teacher quality; we use the rejection rate and the student's gradient norms proxy measures for this.
Analysis of task generations We first compare the qualitative differences in generated tasks. Ta- bhows series of tasks generated by PAIRED (top row) and PAIRED+REJECT +GRAD (bottom row). The columns indicate number of updates to the teacher over time. PAIRED generates sim- pler tasks with fewer obstacles than PAIRED+REJECT +GRAD. This is most noticeable within the first 8,000 updates. We confirm this in Figure Sa where we measure the percentage of area covered in obstacles within the first 10% of student training time; Figure | shows that PAIRED+REJECT +GRAD generates more obstacle-filled tasks than PAIRED. Furthermore, columns 1 12-24k indicate that PAIRED+REJECT +GRAD generates tasks that contain more hallways, eg. along the bottom of the grid. These complex tasks are helpful for the student in generalizing human-designed transfer environments, like a labyrinth or a maze. Note that while these tasks are more complex, they are also feasible since apply REJECT Altogether, this provides evidence that ZONE techniques are effective in helping the teacher produce a coverage of difficult and interesting grid worlds that improve the student's generalization.
8

---

Under review as a conference paper at ICLR 2023
4k
12k
16k
20k
24k
(b)
Table
(a) row are tasks generated from PAIRED, and (b) row are tasks generated from
PAIRED+ PAIRED-RRIECTT REJECT-GRAD. The columns mark the number of model updates.
PAIRED
PAIRED-REJECT
GRAD
PAIRED+REJECTGGRAD
Expettin
Groded
66
60
Stgge
(a) Wall percentage
(b) Rejection rate
(c) Student's gradient norms
Figure 5: (a) tracks the percentage of grid space occupied by walls, generated by the PAIRED teacher and the PAIRED+REJECT +GRAD. (b) tracks rejection rate of the teacher based the PAIRED difficulty criterion. (c) tracks the student's ' gradient norms over the course of training.
Analysis teacher quality We now investigate whether the teacher is able satisfy the ZPD criteria of difficulty and progression. Do the ZONE techniques help the teacher generate tasks that meet the difficulty criterion and that maximize the student's gradient norm? Figure compares the rejection rates across the teachers. The rejection rate is calculated as the percentage of tasks that do satisfy PAIRED's ' regret objective in a sample batch. Rejection in PAIRED occurs when the stu- dent does better than or equal to the anti-student. All ZONE variants learn to generate tasks satisfying PAIRED's difficulty criterion much faster than the original PAIRED algorithm. PAJRED+REJECT and PAIRED+REJECT +GRAD are the fastest to do so. Figure 5c compares the student's gradient norms on the teacher's tasks. The ZONE techniques are very useful in guiding the teacher to generate tasks which induce higher gradient norms than PAIRED: PAIRED+REJECT results in higher gra- dient norms than regular PAIRED, and PAIRED+REJECT +GRAD results in higher gradient norms than both PAIRED+R examples that lead to high gradient norms for the student. This also suggests that the measure of progression could still be improvement. Overall, these results indicate that ZONE aids the teacher in learning to generate tasks better which are within the student's difficulty range and lead to faster learning.
and PAIRED+GRAD. This demonstrates that REJECT helps select
CONCLUSION
Our work is interested in the kind of tasks that best lead to student learning and the teacher's ob- jective. We take inspiration from the concept of the "zone of proximal development" (ZPD) in developmental psychology. which asserts that students learn most effectively when given tasks that are not too difficult and accelerate their learning. We propose ZONE as computational framework that operationalizes ZPD. ZONE consists of two simple techniques: rejecting tasks that fall outside of difficulty zone (REJECT) and maximizing the student's gradient norms (GRAD). We show that these techniques improve the student's performance on both continuous and discrete domains. Alto- gether, this work proposes simple techniques that could be useful for practitioners who want to both accelerate the speed of RL training and improve the agent's 's generalization capabilities. The ZONE methods might be most effective in settings with complex task structures (eg. in natural language setting for reasoning domains) the results in MiniGrid suggest.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mabajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22:1-76, 2021.
15
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, 2019.
James Bagnell, Sham M Kakade, Jeff Schneider, and Andrew Ng. Policy search by dynamic pro- gramming. Advances in Neural Information Processing Systems, 16, 2003. T5
David E Bell. Regret in decision making under uncertainty. Operations research, 30(5):911-981, 1982. 6
Richard Bellman. A Markovian decision process. Journal of Mathematics and Mechanics, pp- 679-684, 1957.
14
Jonathan C Brant and Kenneth O Stanley. Minimal criterion coevolution: approach to open- ended search. Proceedings of the Genetic and Evolutionary Computation Conference, pp- 67-74 2017.
Emma Brunskill and Lihong Li. Sample complexity of multi-task reinforcement learning. In Pro- ceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, pp. 122-131, 2013. 14
Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint arXiv:2006. 2020. 2
Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment
for openai gym. https://tt htw.:tttubbboooo.coooooomooe www.c/ommaaaammm max mecb/ http//ggmmmmiiggg wtwta//ammmmmmmmmmii/i/i/o/o/og
2018.
Irene-Angelica Chounta, Patricia Albacete, Pamela Jordan, Sandra Katz, and Bruce M McLaren.
The "grey area": A computational approach to model the zone of proximal development. In European conference on technology enhanced learning, pp. 3-16. Springer, 2017. 2
Michael Cole, Sheila Cole, and Cynthia Lightfoot. The development of children. Macmillan, 2005. 0
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing processsin systems, 33: 1344911113, 2020.
20
Rafael M Diaz, Cynthia J Neal, and Angela Vachio. Maternal teaching in the zone of proximal development: comparison of low-and high-risk dyads. Merrill-Palmer Quarterly (1982-), pp-
83-107, 1991. -
Yuqing Du, Pieter Abbeel, and Aditya Grover. It takes four to tango: Multiagent self play for auto- matic curriculum generation. In International Conference on Learning Representations, 2022.
19
Kuan Fang, Yuke Zhu, Silvio Savarese, and L Fei-Fei. Adaptive procedural task generation for
hard-exploration problems. In International Conference on Learning Representations, 2020, 2
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference on robot learning, pp. 482-495.
PMLR, 2017. 2
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528. PMLR, 2018a.
7
10

---

Under review as a conference paper at ICLR 2023
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic Goal Generation for Reinforcement Learning Agents. In Proceedings of the 35th Internatiional Conference on Machine Learning. pp. 1518-1528. PMLR, July 2018b. URLhttps: wttppeeeoceeggsseeeeeee mllr press/
v8C/florense Ba html]
ISSN: 2640-3498.
19
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the
ACM, 63(11):139-144 2020.
Izzeddin Gur, Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and
Aleksandra Faust. Environment generation for zero-shot compositional reinforcement learning.
Advances in Neural Information Processing Systems, 34:4157-4169 2021. 2
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual Markov decision processes. arXiv preprint arXiv: 05.. 022592 2015.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing sys- tems, 29, 2016. 3
Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary Lipton, et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv: urXiv:/910.000 1910.00762, 2019.
Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay- guided adversarial environment design. Advances in Neural Information Processing Systems, 34:1884-1897, 2021a. 2
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pp.
PMLR, 2021b.
Leslie Pack Kaelbling. Learning to achieve goals. IJCAI, volume 2, pp. 1094-8. Citeseer, 1993.
14
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: 1805.00909 2018. 4
Graham Loomes and Robert Sugden. Regret theory: alternative theory of rational choice under uncertainty. The economic journal, 92(368):805-822 -24, 1982. 6
Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-Student Curriculum Learning. IEEE Transactions on Neural Networks and Learning Systems, 31(9):3732-3740, September 2020. ISSN 2162-2388. doi: 10.1109/N WWWINNN.S.20........ 29199.934996. Conference Name:
IEEE Transactions on Neural Networks and Learning Systems. 12
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937. PMLR, 2016. I6
Aditya Modi, Nan Jiang, Satinder Singh, and Ambuj Tewari. Markov decision processes with con- tinuous side information. In Algorithmic Learning Theory, pp. 597-618, 2018. 3 14
Tong Mu, Shuhan Wang, Erik Andersen, and Emma Brunskill. Automatic adaptive sequencing in webgame. In International Conference on Intelligent Tutoring Systems, pp. 438. Springer,
2021. 2
Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):29--433 1997. [6
Sanmit Narvekar. Curriculum Learning for Reinforcement Learning Domains: A Framework and
Survey. pp. 50. 1
11

---

Under review as a conference paper at ICLR 2023
Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprint arXhv:2203.01302, 2022a.
Jack Parker-Holdde Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design, 2022b. URL
2203. 01302
2
Mark S Pinsker. Information and information stability of random variables and processes. Holden- Day, 1964. 17
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pp. 835-853, PMLR, 2020.
Martin L. Puterman. Markov Decision Processes Discrete Stochastic Dynamic Programming, John Wiley Sons, Inc., New York, NY, 1994.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally-genest environments. In International Conference on Learning Representations, 2019. 2
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference machine learning, pp. 1312-1320, 2015. 3
14
Jürgen Schmidhuber. Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers in psychology, 4:313, 2013. [
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016. 16
Karim Shabani, Mohamad Khatib, and Saman Ebadi. Vygotsky's s zone of proximal development: Instructional implications and teachers' professional development. English language teaching, (4):237-248, 2010.
Petru Soviany, Radu Tudor lonescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, pp. 1-40, 2022. 2
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG Lanckriet. On integral probability metrics, d-divergences and binary classification. arXiv preprint arXiv:0901.2699 2009. 16
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fer- gus. Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play. Technical Report arXiv:1703.05407, arXiv, April 2018. RtL.hhtp:///xxxxti wtt.//o//////oooooo http::/://araa...rrr/////O/OO 05407 2
Yi Sun, Faustino Gomez, and Jürgen Schmidhuber. Planning to be surprised: Optimal bayesian ex- ploration in dynamic environments. International conference on artificial general intelligence, pp. 41-51. Springer, 2011.
Matthew Taylor and Peter Stone. Transfer learning for reinforcement learning domains: survey. Journal of Machine Learning Research, 10(7), 2009.
Oded Vainas, Ori Bar-Ilan, Yossi Ben-David, Ran haaddBaacrrchh Galit Lukin, Meitar Ronen, Roi Shillo, and Daniel Sitton. E-gotsky: sequencing content using the zone of proximal development. arXiv preprint arXiv: /2268, 2019. 2
Lev S Vygotsky. Thought and language. MIT press, 2012.
Lev Semenovich Vygotsky and Michael Cole. Mind in society: Development of higher psychologi-
cal processes. Harvard university press, 1978.
12

---

Under review as a conference paper at ICLR 2023
Rui Wang. Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arX:v: 2019.
S
Mark K. Warford. The zone of proximal teacher development. Teaching and Teacher Education, 27:252-2555 2011.
Rob Wass and Clinton Golding. Sharpening tool for teaching: the zone of proximal development. Teaching in Higher Education, 19:671 684, 2014, 2
Yilei Zeng, Jiali Duan, Yang Li, Emilio Ferrara, Lerrel Pinto, C. Jay Kuo, and Stefanos Nikolaidis. Human decision makings on curriculum reinforcement learning with difficulty adjustment, 2022. URL htthhttss///axiiaa org/ abs//20s//222/// 02932
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuan- dong Tian. Noveld: A simple yet effective exploration criterion. Advances Neural Information Processing Systems, 14422217-252300 2021.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value dis- agreement. Advances Neural Information Processing Systems, 33:7648-7659, 2020.
A LOWER BOUNDING THE TEACHER OBJECTIVE
We are interested in showing that the lower bound used in Section 3.2 holds. As reminder, the lower bound is:
E E xx
Σ
p(H
1|ctest glest )00)))((000))))) ( cur gour gour)) c ourr geur dgge
)
No (Cann)
paren Rd
max Xo (C
E Bamaxao()
Σ
El
Σ
p( 11etest 0² (00pp(0''000000000000000000))))0 (mmm بum @eor cum )000
yan ER cr
To how this is true, do the following:
13

---

Under review as a conference paper at ICLR 2023
max Xa (Chum)
E [x)))
Σ
p(
1|Cusst glest (pe))p((()) (6 poun c geurt p( yuo co geur d
ER Rd
= maxxxxx (e(nn
B B (
Σ T ER Rd
p( = 1|Ctes (lee)) (pee))(((e)) (p(( , ynr cumr y
(7)
1 grest E supp p((tiss), youur court gourr) p( peur )
(8)
max Xe (C
B x
Σ
p( p(Rtest 1|Ctest grost ((ex))(((( ( c
geur)
(9
Spar ER
Rd
1 gres
p( R(R 1|Cues glest)
(01)
S... p( your corr Bann)20
p( Rl 1/Crest / (
glest
(11)
max
Ou pl (ross) you jesun Bee (00)) >
max No (Crmn)
E Beanna xx(
Σ
p(l 1|Ctest Orese (p p.
( - mum
(C))
ER
p(H 1|Ctestt greut) pl pi wo
1 glest
(5))
max Ss p(Oiou poir caur (ear))
2 max Xx (Cam)
E Baa)~xo)) x
Σ
p( 1| 0*) yp(0" (rour comn,
com tnn )pl grnn
gour)
yau ER 0.
(14)
We assume that O" is the set of all model parameters that maximize p( RRest 1/Crest 8*).
B ZONE ANALYSIS
B. PRELIMINARIES
For any arbitrary set X,, we use A(X') to denote the space of all probability distributions with support on X. For any two arbitrary sets X' and V/ we denote the collection of all functions mapping between them as (X + V} & (f - }}.
B.2 PROBLEM FORMULATION
A standard choice for representing single sequential decision-making problem is the infinite- horizon, discounted Markov Decision Process (MDP) (Bellman 1957, Puterman 1994) defined by (S, A, R, T, (4, 7). Here S denotes set of states, A is set actions, R S A + [0, 1] is a deterministic reward function providing evaluative feedback signals (in the unit interval), T S A + A(S) transition function prescribing distributions over next states, μ A(S) is an initial state distribution, and y E [0, 1) is the discount factor. Building on this formalism, our work is concerned with decision-make multiple tasks or goals (Kaelbling) 1993 Schaul et al. 2015), problem we formulate as a Con- textual MDP (Brunskill
agents that endeavor to learn optimal behaviors across
Lij [013] Hallak et al., 2015]] Modi et al. 2018) (CMDP) given by M A, R, T. Y, Y,X). x). Here, each task of interest identified by an individual context contained in C while jointly sharing the same state-action space A and discount factor y mean- while, variations in the context lead to different environment configurations that may differ in tran-
sition structure T : x x A ASS), reward structure R : C x S x + 10, 1), and initial state distribution μ > ASS).
At the beginning of an episode, a single random context is sampled c ~ x(- E A(C) and held fixed as the agent contends with learning optimal behavior in the resulting MDP denoted by
14

---

Under review as a conference paper at ICLR 2023
(S, A, Re: Te: Me, Me:7). At each discrete timestep t N, beginning with an initial state so agent observes the current state 81 E S, execute an action as E A. enjoys a reward ry = (st, at), and transitions to the next state $1+1
(-) the
Tal: St. a.).
A(A)), contextual policy II encodes a pattern of behavior that maps the current context and state to a distribution over actions. For any fixed context c, the perfor- mance of an agent in the resulting MDP when starting in state 8 E A and taking action A is
Defining II {S x C
assessed by the associated action-value function Q (s, a) = E Σ R"Ra(s at) 80 8, 8,00
where the expectation integrates over randomness in the action selections ~ T(- | 8t, c) and transition dynamics S(+1 Te(- St. ac). With the corresponding value function defined as (s) = (a (Q, a)}. we slightly abuse notation and use V. (4) - Beory(-le)) [V (so))
to integrate over the randomness in the initial state. The optimal CMDP policy x' is defined as achieving supremal value sup E Be-x(-) [V_ (n)].
=eIl
Our work operates in general function function-approximen approximation setting where individual policies me II are parameterized by a vector 0 E e C Rd of arbitrary dimension d, for instance representing the weights of a neural network with fixed architecture. Consequently, the optimal policy To- within this policy class II & {no E II 0 θ S II is defined achieving sup E [V (u)). In
=eElle
this setting, the approximation enror associated with particular choice of policy parameterization then given by (x-x(-) [V** V (1)].
A priori, there is no reason suspect that the context distribution x agent is charged with solving will be tailored in any sort of helpful manner to facilitate rapid or efficient learning. Intuitively, CMDPs that arise in application areas of interest will likely consist of rich, expressive context space alongside a distribution of challenging, complex tasks x that can be easily specified by a domain expert. Consequently, the onerous burden of mastering difficult collection of tasks with little scaffolding falls to the agent. This reality motivates the use of a teacher-student framework wherein the agent viewed as student who gradually faces tasks prescribed by teacher. A successful teacher can incrementally synthesize a useful curriculum of tasks for the student to solve, building competency that allows student to ultimately generalize and succeed across the original collection of challenging tasks prescribed by the distribution X-
In the next section, we provide a illustrative analysis that identifies a particular objective function for a teacher to maximize whose corresponding lower bound motivates the two key elements of the ZONE framework. Our proof techniques are inspired by the Natural Policy Gradient regret lemma of Aggawaa et al. I202l) and the performance guarantee for the Policy Search by Dynamic Programming algorithm of Bagnell et al. (2003).
B.3 A TEACHER OBJECTIVE FOR DERIVING THE ZONE
Recall that function f Rd TR is 3-smooth if
Vr, E Rd. consequence of this is that [x) 3 BT, Vx E RRd and so, by Taylor's Theorem,
[z()) - Vf((2 1| S B||x ='1l2
3 - -
ffxx) - f(x) f(x) (x' x)] 2
Va, Rd.
Assumption 1. (Policy Smoothness) We assume that log (To(a) 8, s,))) is 8-smooth function of 0 E +, a) E S x A and c E C.
Let (So, og. $1, ..)...) .) be a random trajectory sampled according a current student policy me under fixed context e E C. Defining the advantage function as
(7 a) = Q (s, a) VTT
we consider an abstract the advantage function and a learning rate 77 E R>u via
method that updates the student policy parameters based on
0' 0 Vo log (To(a s,c)) (8,
15

---

Under review as a conference paper at ICLR 2023
In practice, one would choose suitable estimator of the advantage function (Mnih et al., 2016 Schulman et al. 2016). Suppose that on-policy policy-gradien updates are performed sequentially on the state-action pairs of the sampled trajectory so that the student policy parameters at the beginning of the episode are ((0) 0 and
( = g()) + voVo log (Toco (ot |s,, c)) A7 (8t, az).
By Assumption [
we have that
(a st,c) = log (Toce+ Sts c)) - -log (Tor se. ct)
log
TTco) (as 6)
2 Vo log (Toc)) (at se. c)) (glrt1)- {(())
2
nAA (st, az) | (Tec (at
$2 c)) 11
ATT (st. az) 11) Vo log (To (as I ss, st,c)) 11
2
2 nAA (St, as) (1-21 n
V log (Tares (a: st, c)) 12
2(1 2(1-))
where the final line leverages the fact that rewards are bounded in the unit interval implying value is upper bounded by (T-ン]* Note that this inequality only holds in this exact form for the (random)
state-action pair (st, ax) that led to the update from policy parameters g(t) to ((++1), Let p denote the distribution over trajectories induced by the policy Ty under context e E C. Let my denote the student policy at the start of the episode and गy denote the updated student policy after the episode terminates.
For brevity, we omit the state and context arguments to each policy in the following. For any tra- jectory - = (so, ag; $1, .). we introduce notation to denote a partial trajectory whose start and end are indexed by timesteps j E N respectively: ग/ (s;, ag $i4 I.... :--1: aj- $j. Qz)- With further abuse of notation, we still use p to denote the distribution over such partial trajec- tories sampled while executing policy x in the MDP induced under context e E C. An objective for the teacher A(C) is
max E BCwA() AEA(C)
Σ E Big-ttno" " (+) [Ba 1.91 1) (TT= ((-0
TTco) - DKI. (TT || Tgge+n)) }I]
At a high level, this says that good teacher prescribes a distribution over environments for fixed student such that the resulting policy updates bring the student closer to the optimal policy in Hlg In slightly more detail, this achieved by examining rollouts of increasing lengths generated by the optimal policy πg- and assessing the reduction in KL-divergence between the student policy and the optimal policy before and after the assuming fixed context C, allowing to drop the outermost expectation.
update. For brevity, we continue onward
Let X be an arbitrary set consider any two distributions v, 1/ e A(X). Recall that the total variation distance is an integral probability metric (Müller 1997 Sriperumbudur et al. 2009) defined as
DTv(U v/) sup ( [f()) - (a [f())], JeF
F { X l1 S 1).
Consequently, for any function f X + such that |l/ SC < co, it follows that
Easo() [f(x)] Exru(s [f(x)] C DTv(v l|
In order to apply this fact to induce a distribution shift, we make the following assumption which controls for the variability in log-likelihood ratio between two policies separated by single policy- gradient update:
Assumption 2. (Bounded log-likeldoood ratio) Let 0 be an initial set of policy parameters and O' denote the policy parameters after single policy-grudies update. For any fixed E C, we assume
that there exists a numerical constant C < oc such that log Tal (a/(a)),))
< C. V(s, a) E x A.
3o(a
16

---

Under review as a conference paper at ICLR 2023
Expanding from above, have
E 0 or) (-)
( (KL (To- || Totes) - DKL Tgic+t)]]]
1=0
Toceets> (as C)
B N
2000 (+)
BeinTe 1--1)111)
Bor (Hlor))) log
Tpc 8t, c)
1=0
0C
(atesss (6, c)
St, St,c) ]
log
Toa
(1)
Tacc (at)
Tocc (az
(2,cc)
E BTapp
log
())
(Tg(s) (at
1=
Toce)1) (at
St: c) Ta(te) (ag
E BaNp,0 To" (-)
log
Tocc- (8,, c)
2 B B Tor (
*log
Talt
(as ct:
Tou St; (t:C))
}
C
2 B ()
7' llog
Drv(p." 11)
Tacco (at C)
(1 (1-7)*
1==0
11.3
C Drv(p P!
2 17 1
E Bx ())
Ae (st, az) log (Toi (at) sc,c)) 11
2(1 -7))
(1-
n/3
C
2 17 1
E BANDE" (-)
Σ
72t A (st, a. IV log (Tero (at st, st, l1
v2(1 77) V DKL(po" po
2(1 ((1)))
1=0
where the first inequality follows as Y E [0, 1). the second inequality leverages Assumption 2 and the aforementioned fact to shift trajectory distributions, the penultimate inequality applies earlier policy-gradien norm lower bound, and the final inequality follows as Pinsker's inequality (Pinsker)
1964..
Now accounting for the randomness in the contexts, we have the following lower bound to the teacher (maximization) objective:
oc
Ex-A(-)
1Np (-) [T [Dx
|| TTcc)) - (KL(TT || Toc++ )]]
0=
1733
C 2211-9)
V Dralle"" Pl
BowA()
17
2(1 - 7)
E Bynp7
A (st, a:)ll log (Tgie (ax 8t, c))
Observe that term ① captures notion of learning potential for the current student under tasks sampled by the teacher. Intuitively, this is quantified by looking at the average, Eevvantage-wiightee policy-gradiess norm across trajectories generated by the student policy me each sampled context A(-). Orthogonally, term ② encapsulates notion of problem difficulty as measured by how much the trajectory distribution of the student differs from that of the optimal policy in each sampled context.
When this latter quantity 2 is too large, suggesting an overwhelmingly difficult problem for the student where large number of samples environment interactions will be needed to improve per- formance, this term overpowers any learning potential captured in term ①.. Conversely, tasks that are too easy for the student will lead to scenarios where both ② and the advantage terms encountered in I will be small (or even zero), suggesting little opportunity for improving performance. Naturally, the "sweet spot" ZPD suggested by this lower bound consists of teacher selecting tasks with
17

---

Under review as a conference paper at ICLR 2023
reasonably large policy gradient norms (signaling learning potential) while being within the students means (as measured by the divergence between the student's ' trajectory distribution from that of the optimal policy). Practical approaches automated curriculum design use notion of pseudo-regret in lieu (2, accounting for a lack of knowledge about the optimal policy advance (Florensa et al,
2018h Dennis al. 2020
C ADDITIONAL TEACHER ANALYSIS
Here we include additional analysis on the teacher.
C.I PAIRED ON MuJoCo ENVIRONMENTS
Figure 6 shows the rejection rate and student's ' gradient norms. Interestingly, REJECT tends to have higher rejection rates. In general, GRAD and PAIRED have similar rejection rates.
PATRED
PAIRED+REJECT
PRARED+GRAD
PAIRED+REJEC++GRAN
62
60
20
Point
(b) Walker
(c) Reach
(d) Toss
66
60
(c) Point mass
(f) Walker
(g) Reach
(h) Toss
Figure 6: On the MuJoCo environments. The rejection rate is reported in the first row (a-d), and the gradient norms reports in the second row (e-h).
C.2 GOAL GAN
Figure 7 shows the rejection rates on the Goal GAN environments and the MuJoCo environments. In general, the rejection rates similar across environments.
D ALGORITHM INFO
We provide information on the PAIRED and Goal GAN implementations.
D.I PAIRED ON MINIGRID ENVIRONMENTS
We use the implementation at https: httpa//uww.ccaalllaaaaaaaaaeeeee which is based on Dennis et al. 20200x 's implementation. We do not change any hyperparameters in their algorithm. All hyperparameters are the same as those reported in Dennis et al. (2020). We refer to their paper for more details PAIRED. We run all the variants of PAIRED with ZONE with 10 seeds.
com
18

---

Under review as a conference paper at ICLR 2023
(a) Maze
(b) Maze Ant
(c) Goal Ant
05
200
130
200
200
500
300
400 4M6
200
300
600
(d) Walker
(e) Reach
(I) Toss
Figure 7: Reported is the rejection rate. The rate for Goal GAN on the original Goal GAN environ-
ments is reported in the first row, and the rate on the MuJoCo environments is reported in the last row.
D.2 GOAL GAN ON GOAL GAN ENVIRONMENTS
We use the original implementationn at http:///iihubb.cm//ooreeeess
selab-curriene um We do not change any hyperparameters. We refer to their paper (Flornnsa et al. al.] (2018b) for more details on Goal GAN. We run all the variants of Goal GAN with
ZONE with 5 seeds.
D.3 PAIRED AND GOAL GAN ON MuJoCo ENVIRONMENTS
We use Du al. 2022 implementation of PAIRED and Goal GAN on their MuloCo environ- ments. Their MuloCo setup is designed to test students on out-of-domain goals, that the students have not yet seen. We run all the variants both algorithms with ZONE with 10 seeds. We do not change any of the hyperparameters. We use the default Goal GAN implementation in their work. Goal GAN stores 500 goals and the student is evaluated on the goals 3 times. The mean reward is taken and used to determine the label for the teacher. The label is if the mean reward lies within the difficulty criterion (r E 0.9),. and 0 otherwise. Every 500 steps, the teacher trains on the labelled data.
PAIRED is implemented using the default parameters from Du et al. 2022 's algorithms but with symmetrization turned off (ie. remove the second teacher in their work).
E Do MEASURES OF DIFFICULTY MATTER?
A key component to ZONE is the choice of difficulty measure. Most prior work use reward to model difficulty: The lower of the reward, the more difficulty the problem. If the problem is more difficult, then the student is less likely to succeed on the problem.
However, we find that using dense rewards as proxy measure for difficulty is misleading. Dense rewards are typically used for training students in the MuloCo control setting, as done in Du et al. (002)). Running ZONE on PAIRED in this setting reveals an interesting discrepancy: ZONE can achieve higher episodic reward than the base algorithm (Figure 8a). however achieves low suc- cess (Figure 8b). For example, at 2000 steps, episode reward scores increasingly higher from PAIRED, PAIREDGGRAD, PAIRED+REJECT to PAIRED+REJECT +GRAD. However, success scores increasingly higher from PAIRED+GRAD. PAIRED+REJECT, PAIRED+REJECT +GRADto PAIRED.
19

---

Under review as a conference paper at ICLR 2023
Waller Dpicade Reward
Whiter Dob Suecess Rate
20000
T500
0.6
1000C
0.4
BRIDED--JJECT-GRADD PAIRED CRAD PARED-RREEECT
10000
0.2
PAIRED
0.0
1000
9000)
2002
1500
3000
0
1000
7100)
3000
(000
3000
Stops
Stops
Figure 8: PAIRED on the Walker domain. The figures show the performance of the student on the out-of-domain goals. (a) shows the student's test return over the course of training. (b) shows the student's 8 success rate over the course of training.
This discrepancy reveals that training the teacher based on a dense-reward difficulty measure can be misleading when the reward function does not correlate well with the student's success. ZONE sensitive to this choice of difficulty measure that is not well correlated with success. Thus, for MuJoCo experiments, we choose use the student's s success as a measure of difficulty which is what PAIRED in Dennis et al. (2020) and Goal GAN assume. This gives us the results from the previous section in Figure and Figure
ADDITIONAL IMAGES OF GENERATED PAIRED ENVIRONMENTS
See Figure
10
100
200
300
400
500
(a)
(b)
Table 2: (a) row are tasks generated from PAIRED, and (b) row are tasks generated from
BALRED-REJECTGGER The columns mark the number of model updates. This table includes updates
10-500.
G ABLATION ON GRAD
See Figure
g
20

---

Under review as a conference paper at ICLR 2023
600
700
800
900
1000
1100
Table 3: (a)
tasks generated from PAIRED, and (b) row tasks generated from
PAIRED-REIECTGRAA. The columns mark the number model updates. This table includes updates 600-1100.
1200
1300
1400
1500
1600
1700
(b)
Table 4:
(a) row are tasks generated from PAIRED, and (b) row are tasks generated from
The columns mark the number of model updates. This table includes updates
1200-1700.
1800
1900
2000
2100
2200
2300
Table 5: (a) row are tasks generated from PAIRED. and (b) row are tasks generated PAIREID-RRIFCTGGRAA The columns mark number model updates. This table includes updates
1800-2300.
21

---

Under review as a conference paper at ICLR 2023
2300
2400
2500
2600
2700
2800
(a)
(b)
Table 6: (a) row are tasks generated from PAIRED. and (b) row are tasks generated from
PAIRED+REJER REJECT+GRAD. The columns mark the number of model updates. This table includes updates 2300-2800.
2800
2900
3000
Table 7: (a) tasks generated from PAIRED, and (b) row tasks generated from
BALRED-REJEC++RADD The columns mark number model updates. This table includes updates 2800-3000.
(a) 4 Rooms
(b) 16 Rooms
(c) Labyrinth
(d) Maze
Figure 9: PAIRED performance on MiniGrid transfer environments which includes an ablation of just training the PAIRED teacher with GRAD (ie. just training the teacher with the student's gradient norms). Note that GRAD is run on 4 seeds, and the other methods are run on 10 seeds.
22


Reviews:
[{"review_id": "kGY_0922vVW", "reviewer": "Reviewer_9WXc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study to problem of finding and using the right balance in curriculum learning applied to reinforcement learning (RL). More specifically, at each training iteration of a student network, the proposed method aims to generate an appropriate, i.e. not too easy, not too hard, task for the student to learn. The goal is to achieve faster convergence and better performance. The approach is applied on top of two existing methods and tested on two benchmarks.", "summary_of_the_review": "Due to the mixed results and the missed related works, I am not confident about accepting the paper.", "strength_and_weaknesses": "Strengths:\n- The paper is well written.\n\nWeaknesses:\n- Experimental results are mixed (sometimes better, sometimes worse). Thus, the benefit of the proposed method is not clearly shown.\n- The title should be clearly tied to reinforcement learning. The title makes it look like the idea is more generic, while being tested only on RL. If the authors prefer to keep the title as is, they should demonstrate applicability in other areas, e.g. classification.\n- There are many competing curriculum learning approaches for RL, see [A], missed by the authors in their discussion.\n\n[A] Soviany et al., \"Curriculum learning: A survey\", International Journal of Computer Vision, 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aeu_M-Ibm1", "reviewer": "Reviewer_Go3r", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes ZONE, a computational framework of curriculum task generation for reinforcement learning based on a concept from developmental psychology, i.e., zone of proximal development (ZPD). The target of the framework is to measure and schedule difficulty and progression of a student model. The approach is composed of two techniques, REJECT and GRAD. REJECT is to replace too easy or too hard tasks generated by teacher with proper tasks. GRAD is to maximize the norm of student model’s gradient, which reflects student progression. To evaluate the effectiveness, the authors conduct experiments on two baselines and their corresponding settings.", "summary_of_the_review": "To summarize, the paper is good except for the experiments part. The motivation is interesting, based on developmental psychology. The method is well described and easy to understand. However, the experimental results are not so ideal and the analysis is not so convincing.", "strength_and_weaknesses": "Strengths:\n1.\tThe paper is well written and easy to understand.\n2.\tThe motivation of the method is intuitive and backed by psychology.\nWeaknesses:\n1.\tThe transition from psychological concepts to computational metrics seems not that natural, which causes some problems. For instance, the difficulty needs manual design based on different RL models, i.e., a fixed range for Goal GAN and a changeable range for PAIRED. Besides, the GRAD technique cannot be applied when the teacher is not trained with rewards.\n2.\tThe number of baselines is a bit small, which degrades its universality and generality.\n3.\tThe analysis of the experiment is not convincing. Figure 5a shows that ZONE generates more complex tasks, instead of better tasks matching the learning situation of a student. The more obstacle-filled or hallways-filled tasks are not necessarily better. Figure 5c shows that the gradient norm of student model without GRAD is larger than that with GRAD in the later stage of training, which seems weird and strange.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "85jh59AKMB", "reviewer": "Reviewer_3apF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to improve a known application of curriculum generation with two new techniques. One is to select tasks according to the difficulties to the student. The other one is to reward the teacher according to the student’s gradient norm such that the teacher can generate tasks which enhance the learner more. The authors add these two techniques to existing algorithms, PAIRED and Goal GAN and experimentally test the performances with these two techniques.\n", "summary_of_the_review": "This paper is suggested to be rejected because (1) the algorithm does not propose new measurements on if a task is in the zone of proximal development and does not develop novel techniques to solve the problem, and (2) the paper needs more clarification.", "strength_and_weaknesses": "The paper develops an analytical objective for task generation in Equation (1)-(5), clearly defining the problem. Also, the two techniques are well described, and the proposed algorithm is easy to follow and easy to apply. The experiments are well designed, including comparisons with two representative baselines and analysis of generated tasks.\n\nHowever, there are still some improvements that could be made to the paper:\n\n(1) This paper does not bring in enough novelty. The problem is controlling the difficulty of tasks in the zone of proximal development is already considered in the two baselines. There is no new approach to control the difficulty proposed. The first proposed technique, REJECT, is based on the developed methodologies of the baselines and is already developed in the algorithm POET, proposed by Wang et al. (2019). POET already proposes to reject or delete new environments that are too hard or too easy for the agent.\n\n(2) The motivation of the approach needs more clarification. Neither of the two proposed techniques is proven to maximize the derived objective. Meanwhile, the paper is inspired by generating tasks in the zone of proximal development, which is not too hard nor too easy for the student. However, maximizing the student's gradient norm is unrelated to this main idea of generating ZPD tasks. \n\n(3) More description of the setting is needed. It is unclear whether the paper focuses on a generalization objective to maximize expected returns among a given environment distribution or what.\n\n(4) Modelling the probability in the objective in section 3.3 on page 5 is unclear and technically incorrect. The paper claims that the first technique, REJECT, models the probability $p(r_{curr}|c_{curr},\\theta_{curr})$ but how? Furthermore, this probability is decided by the dynamics of the MDP and cannot be influenced by the teacher. This modelling claim is technically incorrect.\n\n(5) The test on the gradient maximization idea is limited. Some ablation studies to test the gradient maximization idea alone would be interesting, for example, whether gradient maximization is better than regret maximization proposed by PAIRED and difficulty control by Goal GAN.\n\n(6) Some descriptions of experiments are missing. For example, how to set the gradient norm and returns scale when applying the gradient maximization technique? what contexts are all algorithms tested on for tasks (a)-(g) in Figure 2 on page 7? How the rejection rate is computed in Figure 5 on page 9? \n\n(7) I have a few questions about the experimental results:\n\n(a) We see improvements in PAIRED-based algorithms. But PAIRED generates tasks which maximize the current regret. So why are there so many tasks given by the PAIRED rejected? Does PAIRED generate many negative regret tasks?\n\n(b) It is shown in Table 1 on page 9 that PAIRED+REJECT+GRAD gives hard tasks from the beginning. Does it break the increasing difficulty set by the curriculum learning? Is it good?\n\n(c) REJECT does not improve Goal GAN much. Can we say PAIRED is not controlling the difficulty of tasks well as claimed, but Goal GAN already fulfills the job?\n\n(d) Why do PAIRED-based algorithms seem to have a larger variance in the acceptance rate?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2KhzOLUMgHH", "reviewer": "Reviewer_BAdD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies curriculum learning in reinforcement learning. To achieve this, the paper formulates a teacher-student framework, where the teacher is responsible to generate the tasks for the student to learn. The key idea is two-folded. First, the student should not learn from the tasks that are outside of the difficulty scope. Second, the teacher should be rewarded for generating tasks that cause larger parameters update of the policy/value networks of the student. To implement this idea, the paper proposes REJECT, which simply rejects the tasks that are too easy or too hard based on the rollout reward, and GRAD, which makes the amount of the parameters update as one of the rewards for the teacher. Experiments show that the two strategies can improve efficiency.", "summary_of_the_review": "This paper presents some interesting ideas. However, I have concerns about the REJECT. Particularly, it is unclear to me whether the dropped samples are counted when drawing the learning curve. I would like to ask the authors to clarify. If not counted, the comparison could be unfair. The experiments also need improvement.", "strength_and_weaknesses": "Strengths:\n\n1. The proposed ideas are simple and intuitive.\n2. The idea of using the amount of student model updates to guide the teacher is interesting. While there are some similar ideas in curriculum learning, it is nice to see it is effective in RL.\n\nWeaknesses:\n\n1. It is unclear to me why REJECT helps. It simply drops the tasks that are too difficult or easy. However, it still requires generating data with the student model. So it is still expensive. I am also curious whether such dropped samples are counted in the learning curve.\n2. The improvement over the baselines is not significant. It is only comparable with goal GAN.\n3. While the focus of the paper is curriculum learning, the paper should consider baselines that use intrinsic rewards in the experiment. The selected environments are mainly hard-exploration environments, so intrinsic reward methods (such as count-based intrinsic reward) could also perform well in these environments. So I would like to see a comparison.\n4. Various exploration methods (such as intrinsic rewards) are not discussed in the related work. They try to tackle similar challenges in different ways.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject