Proximal Curriculum for Reinforcement Learning Agents
Authors: Georgios Tzannetos
Bárbara Gomes Ribeiro
Parameswaran Kamalaruban
Adish Singla
Venue: Accepted by TMLR ()

Abstract: We consider the problem of curriculum design for reinforcement learning (RL) agents in contextual multi-task settings. Existing techniques on automatic curriculum design typically require domain-specific hyperparameter tuning or have limited theoretical underpinnings. To tackle these limitations, we design our curriculum strategy, ProCuRL, inspired by the pedagogical concept of Zone of Proximal Development (ZPD). ProCuRL captures the intuition that learning progress is maximized when picking tasks that are neither too hard nor too easy for the learner. We mathematically derive ProCuRL by analyzing two simple learning settings. We also present a practical variant of ProCuRL that can be directly integrated with deep RL frameworks with minimal hyperparameter tuning. Experimental results on a variety of domains demonstrate the effectiveness of our curriculum strategy over state-of-the-art baselines in accelerating the training process of deep RL agents.

Published in Transactions on Machine Learning Research (05/2023)
Proximal Curriculum for Reinforcement Learning Agents
Georgios Tzannctos
Max Planck Institute for Software Systems
Bárbara Gomes Ribeiro Max Planck Institute for Software Systems
byomesr@mppisss..og
Parameswaran Kamalaruban
Sperrancssurremmtturrnnn.
The Alan Turing Institute
Adish Singla
adishs@@ppissssoogg
Max Planck Institute for Software Systems
Reviewed on OpenReview: /t openreiie.. net/ forum? id= @WUyeeMaaM
Abstract
We consider the problem of curriculum design for reinforcement learning (RL) agents in contextual multi-task scttings. Existing techniques automatic curriculum design typically require domain-specific hyperparameter tuming or have limited theoretical underpinnings. To tackle these limitations, we design our curriculum strategy, PROCURT, inspired by the pedagogical concept of Zone of Prozimal Development (ZPD). PROCURL captures the intuition that learning progress is maximized when picking tasks that are neither too hard nor too easy for the learner. We mathematically derive PROCuRL by analyzing two simple learning settings. We also present practical variant of PROCURL that can be directly integrated with deep RL frameworks with minimal hyperparameter tuning. Experimental results on variety of domains demonstrate the effectiveness of our curriculum strategy over state-of-tho-art baselines in accelerating the training process of deep RL agents.
1 Introduction
Recent advances in deep reinforcement learning (RL) have demonstrated impressive performance in games, continuous control, and robotics (Mnih et al. 2015 Lillirap el. al., 2015 Silver el. al. 2017 Levine el al.
2016). Despite these remarkable successes, a broader application of RL in real-world domains often very limited. For example, training RL agents in contextual multi-task scttings and goal-based tasks with sparse rewards still remains challenging (Hallak et al.
2015 Kirk et al.
2021 Andrychowicz et al. 2017 Florensal
et al. 2017 Riedmiller et al. 2018).
Inspired by the importance of curricula in pedagogical domains, there is growing interest in leveraging curriculum strategies when training machine learning models in challenging domains. In the supervised learning setting, such as image classification, the impact of the order of presented training examples has
been studied both theoretically and empirically (Weinshall al. 2018 Weinshall & Amir 2018 Zhou &
Bilmes| 2018 Zhou et al. 2021 Elman 1993; Bengio et al. 20099] Zaremba & Sutskever 2014] Recent works have also studied curriculum strategies for learners in siguuenntil-dddiiiinnnnaiinggg settings, such imitation learning (where the agent learns from demonstrations) and RL (where the agent Icarns from rewards). In the imitation learning sel ling, recent works have proposed greedy curriculum strategies for picking the next training demonstration according to the agent's learning progress (Kamalaruban et al. 2019 Yengera et al. 2021). In the RL sctting, several curriculum strategics have been proposed to improve sample efficiency, e.g. by choosing an appropriate next starting state or goal state for the task to train on (Wöhlke al.) 2020:
Florensa et al. 2017 2018 Racanière al. 2020 Riedmiller et al. 2018 Klink al. 2020a
Fimer et al.

---

Published in Transactions on Machine Learning Research (05/2023)
2021 Despite extensive research on curriculum design for the RL setting, existing techniques typically have limited theoretical underpinnings or require domain-specifie hyperparameter tuning.
In this paper, we are interested in developing a principled curriculum strategy for the RL setting that is broadly applicable to many domains with minimal tuning of hyperparameters. To this end, we rely on the Zone of Prozimal Development (ZPD) concept from the educational psychology literature (Vygotsky & Cole 1978 Chaiklin 2003). The ZPD concept, when applied in terms of learning progress, suggests that progress is maximized when the learner is presented with tasks that lie in the prozimal zone, i.e., tasks that are neither too hard nor too easy. This idea of proximal zone be captured using notion of probability of success score (s),, w.r.t. the learner's current policy for any given task s. Building on this idea, we mathematically derive an intuitive curriculum strategy by analyzing two simple learning settings. Our main results and contributions are as follows:
I. We propose a curriculum strategy, PROCURL, inspired by the ZPD concept. PROCURL formalizes the idea of picking tasks that neither too hard nor too casy for the learner in the form of selection strategy arg max, PoSx, (s) PoS,, (s) where PoS*(s) corresponds to the probability of success score w.I.t. an optimal policy (Section 3.1
II. We derive PROCURL under two specific learning settings where we analyze the effect of picking task on the agent's learning progress (Section 3.2
III. We present a practical variant of PROCURL, namely PROCURL-VAL, that can be easily integrated with deep RL frameworks with minimal hyperparameter tuning (Section 3.3).
IV. We empirically demonstrate the effectiveness of PROCURL-VAL over state-of-the-art baselines in accolerating the training process of deep RL agents in a variety of environments (Section 4
1.1 Related Work
Curriculum strategies based on domain knowledge. Early works on curriculum design for the supervised learning setting typically order the training examples in increasing difficulty (Elman 1993 Bengio et al. 2009 Schhidhhueer 2013; Zaremba & Sutskever 2014). This casy-to-hard design principle has been utilized in the hand-crafted curriculum approaches for the RL setting (Asada et al. 1996 Wu
& &Tian 2016). Moreover, there have been recent. works on designing greedy curriculum strategies for the
imitation learning setting based on the iterative machine teaching framework (Liu et al. 2017 Yang et al., 2018 Zhu et al., 2018 Kanalaruban et al., 2019 Yengera et al. (021). However, these approaches require domain-specific expert knowledge for designing difficulty measures.
Curriculum strategies based on ZPD concept. In the pedagogical setting, it has been realized that effective teaching provides tasks that are neither too hard nor too easy for the human learner. This intuition of providing tasks from particular range of difficulties is conceptualized in the ZPD concept (Vygotsky
Oudeyer et al. 2007 Baranes & Oudeyer 2013 Zon et al. 2019). In the RL setting, several curriculum strategies that have been proposed are inherently based on the ZPD concept (Florensa al.) 2017 2018 Wöhlke et al. 2020). A common underlying theme in both Florensa et al. (2017) and Florensa et al. (2018) is that they choose the next task (starting or goal state) for the S {_ma)- Here, the threshold values riin and Tmax require tuning according to the learner's progress and specific to the domain. Wöhlke et. al.) (2020) propose a unified framework for the learner's performance-baase starting state curricula in RL. In particular, the starting state selection policy of Wöhlke al. (2020), P[ s) x G(PoS,.. for some function G, accommodates existing curriculum generation methods like Florensa et al. (2017); Graves et al. (017)) Despite promising empirical results, theoretical analysis of the impact of the chosen curriculum on the RL agent's learning progress is still missing in the aforementioned works.
& Cole. 1978 Chaiklin
2003
learner uniformly at random from the set {s: :min PoS,,
Curriculum strategies based on self-paced learning (SPL). In the supervised learning setting, the curriculum strategies using the SPL concept optimize the trade-off between exposing the learner to all
Github repo: http:///ggttuub..ccaaccciiiaaeee http://aiii.....ta.oooooooom///a -ww.rriuumn-urrrruum--rl
2

---

Published in Transactions on Machine Learning Research (05/2023)
available training examples and selecting examples in which it currently performs well (Kumar et al 2010 Jiang et al. 2015). In SPDI. (Klink et al. 2020b a 2021; 2022) and SPACE (Eimer et al. 2021 the authors have adapted the concept of SPL to the RL setting by controlling the intermediate task distribution with respect to the learner's current training progress. However, SPDL and SPACE differ in their mode of operation and their objective. SPDL considers the procedural lask generation framework where lasks of appropriate difficult levels can be synthesizod, as also considered in Florensa et al. (2017) 2018). In contrast, SPACE considers a pool-based curriculum framework for picking suitable tasks, as popular in the supervised learning setting. Further, SPDL considers the objective of targeted performance w.r.t. a target distribution (e.g., concentrated distribution on hard tasks); in contrast, SPACE considers the objective of uniform performance across a given pool of tasks. Similar to SPACE, in our work, we consider the pool-based setting with uniform performance objective. Both SPDL and SPACE serve as state-of-the-art baselines in our experimental evaluation, In terms of curriculum strategy, SPDL operates by solving an optimization problem at each step to pick a task (Klink et al., 2021]; SPaCE uses a ranking inducod by the magnitude of differences in current/previous critie values at each slep to pick a task (Eimer al. 2021 In the appendix, we have also provided some additional information on hyperparameters for SPDL and SPaCE.
Other automatic curriculum strategies. There are other approaches for automatic curriculum gen- eration, including: (i) by formulating the curriculum design problem with the use of a meta-level Markov Decision Process Narvekar et al. 2017
Narvekar & Stone 2019; (ii) by learning how to generate training tasks similar to a teacher (Dendorfer et al.) 2020 Such et al. 2020 Matiisen et al.] 2019 Turchetta et al,
2020): (iii) by leveraging self-play as a form of curriculum generation (Sukhbaatar et al. 2018); (iv) by using the disagreement between different agents trained on the same tasks (Zhang et al.) 2020); (v) by picking the starting states based
single demonstration (Salimans & Chen 2018 Resnick et al.. 2018); and
(vi) by providing agents with environment variations that are at the frontier of an agent's capabilities, e.g.. Unsupervised Environment Design methods (Dennis et al.] [2200] Jiang et al. 2021a Parker-Holder et al. 2022). We refer the reader to recent surveys on curriculum design for the RL. setting Narvekar et al. 2020 Portelas et al.
2021
Weng 2020
2 Formal Setup
In this section, we formalize our problem sctting based on prior work on teacher-student curriculum learn- ing (Matiienn et al., 2019)
MDP environment. We consider a learning environment defined as a Markov Decision Process (MDP) M: (S,A, H, Smit). Here, S and A denote the state and action spaces, T S S A + [0, is the transition dynamics, H is the maximum length of the episode, and R : S A - R is the reward function. The set of initial states Sinit C S specifies a fixed pool of tasks, i.e., each starting state s E Smit corresponds to a unique task. Note that the above environment formalism is quite general enough to cover many practical settings, including the contextual multi-task MDP setting (Hallak et al. 2015
RL agent and training process. We consider an RL agent acting in this environment via a policy t : S A- + [0, 1] that is a mapping from a state to a probability distribution over actions Given task with the corresponding starting state E Smit, the agent attempts the task via trajectory rollout obtained by executing its policy from in the MDP M. The trajectory rollout is denoted as {= (ls(t),((()) R(s(T), ((f)), (()))))== with s()) = and for some h < H. The agent's performance on task is measured via the value function V*(s) := E'Eo R(s(t), a ((+))) (,,)) M, (0)) = s]. Then, the uniform performance of the agent over the pool of tasks Smit is given by V := En- [s)] The training process of the agent involves an interaction between two components: student component that is responsible for policy update and teacher component that is responsible for task selection. The interaction happens in discrete steps, indexed by t = 1,2...., and is formally described in Algorithm
IIn this setting, for a given set of contexts C, the pool of tasks is given by (Me (S. H., Sma) c C). Our euvironment formalism (MDP M) covers this setting follows: S = S x C; Siatt Smit x C; T(('' ((()),)((ss.)) () and R(((,,), o)) V C S, a C A,A. C.
For general finite- horizon MDPs, including the time step as part of the state is important. However, avoid complicating the notation with additional indexing, we have assumed that the time step implicitly included in the state.
3

---

Published in Transactions on Machine Learning Research (05/2023)
Let Tend denote the agent's final policy at the end of training. The training objective is to ensure that the uniform performance of the policy Tend is e-near-optimal, T.e., (max, V* - S 6. In the following two paragraphs, we discuss the student and teacher components in detail.
Student component. We consider a parametric representation for the RL agent, whose current knowledge policy my : SxA - [o, 1]. At step i, the
is parameterized by E O S TRd and each parameter mapped student component updates the knowledge parameter based on the following quantities: the current knowl- edge parameter f,, the task picked by the teacher component, and the rollout & {(s]'), ai RRss ())). Then, the updated knowledge parameter 0.+1 mapped to the agent's policy given by T441 = #B As a concrete example, the knowlodge parameter of the REINFORCE agent (Sutton al. 1999) is up- dated 01+1 + 0, + Th sh11 D .9 (T) where nt is the learning rate, G. >E R(s ", (")), and 9: (T) = Vo log No (a, (7) )s) A-0."
Teacher component. At step I, the teacher component picks a task with the corresponding starting state 9t for the student component to attempt via a trajectory rollout (see line 3 in Algorithm [
(0)
The sequence
of tasks (curriculum) picked by the teacher component affects the performance improvement the policy
my. The main focus of this work is to develop a leacher component to achieve the training objective in both computational and a sample-efficiet manner.
Algorithm RL Agent Training as Interaction between Teacher-Student Components
1: Input: RL agent's initial policy I1
2: for -= 1,2,.. do
3: Teacher component picks task with the corresponding starting state "
4: Student component attempts the task via a trajectory rollout & using the policy ne from s
Student component updates the policy to TT+1-
6: Output: RL agent's ' final policy Tend <- Te+1-
3 Proximal Curriculum Strategy
In Section 3.1 we propose a curriculum strategy for the goal-based setting. In Section 3.2 we show that the proposed curriculum strategy can be mathematically derived by analyzing simple lcarning scttings. In Section
3.3 we present our final curriculum stralegy that is applicable general settings.
3.1 Curriculum Strategy for the Goal-based Setting
Here, introduce our curriculum strategy for the goal-based setting using the notion of probability of success scores.
Goal-based setting. In this setting, the reward function R is goal-based, i.c., the agent gets a reward of 1 only at the goal states and 0 at other states; moreover, any action from a goal state also leads to termination. For any task with the corresponding starting state s E Smit, we say that the attempted rollout & succeeds in the task if the final state of when the rollout succeeds in task s, and 0 otherwisc. Furthermore, for agent with policy म, we have that (*) = E [succ(E: (T, is equal to the total probability of reaching a goal state by executing the policy x starting from E Smit.
goal state. Formally, suce(f: s) is an indicator function whose value is
Probability of success. We bogin by assigning a probability of success score for any task with the corresponding starting state 8 E Sinit w.r.t. any parameterized policy ma in the MDP M.
Definition 1. For any given knowledge parameter 0 E θ and any starting state s E Smnt, we define the probability of success score PoSo(s) as the probability of successfully solving the task s by executing the policy me in the MDP M. For the goal-based setting, we have PoSa(s) = (s).
With the above definition, the probability of success score for any task s E Sint w.r.t. the agent's current policy ne given by (s) PoSa, (s). Further, we define PoS* (s) max PoSo(s).
4

---

Published in Transactions on Machine Learning Research (05/2023)
Curriculum strategy. Based on the notion of probability of success scores that we defined above, we propose the following curriculum strategy:
arg max (PoS.(6) (s) (+)),
(1)
SESmi
i.e, at step i, the teacher component picks a task associated with the starting state si according to Eq. The term PoS{(s) (PoS* (s)) can be interpreted as the geometric mean of two quantities: the learner's probability of solving the task and the expected regret of the learner on this task. In the following subsection, we show that the above currieulum stralegy can be derived by considering simple learning settings, such as contextual bandit problems with REINFORCE agent; these derivations provide insights about the design of the curriculum strategy.
(0)
3.2 Theoretical Justifications for the Curriculum Strategy
To dcrive our curriculum strategy for the goal-basod setting, we additionally consider independent tasks where any lask (0) picked from the pool Smit at step l only affects the agent's knowledge component corresponding to that task. Further, we assume that there exists a knowledge parameter 0 E such that Wg- aarggma, max, V*, and mg- is referred to as the target policy. Then, based on the work of WWinshall et al.) (2018); Kamalaruban et al. (2019); Yengera et al. (2021), we investigate the effect of picking a task (0) at step / on the convergence of the agent's 's parameter O, towards the target parameter 0" Under a smoothness condition on the value function of the form | V=* S L. ,
v0,0 H' E θ for some L 0, we can
translate the parameter convergence (0; + 0*) into the performance convergence (V - V=**). Thus, we define the improvement in the training objective step as
A,(0++1 (0) Et) 11 0ell, - -10 0++1ll1
(2)
In the above objective, we use the f-norm because our theoretical analysis considers the independent task setting mentioned above. Further, we define the expected improvement in the training objective at step due picking the task 2(0) as follows:
C((o, = sala [Ar(0+1100,.5 "
(3)
Note that the above quantity is an approximation of the expected learning progress measure as defined in Graves et al. (2017). In the following subsection, justify proposed curriculum strategy by analyzing the above quantity for specific agent model under the independent task setting. More concretely, for the specific setting considered in Section 3.2.1] Theorem D implies that picking tasks based on the curriculum strategy given in Eq. D maximizes the expected value of the objective in Eq. 2 Tn the appendix, we provide an additional justification by considering an abstract agent model with a direct performance parameterization.
3.2.1 Reinforce Agent with Softmax Policy Parameterization
We consider the REINFORCE agent model with the following softmax policy parameterization: for any 0 E RISHAA we parameterize the policy as (e(a(s) & asp( Vs S, E A. In the following, we consider problem instance involving a pool of contextual bandit tasks (a special case of independent task setting). Consider an MDP M with g S as the goal state for all tasks, Smit S {g), A= {ai, a2), and H 1. We define the reward function as follows: R(s, a) = 0, Vs E S {g). a E A and R(g.a) = 11,va E A. For a given probability mapping Prand S + [0, 1], we define the transition dynamics as follows: T(gls, a1) Prand(s), Vs E S; T(s|s, a1)
Praud(s), (s), vs E S; and T(s]s, 02) = 1, Vs S. Then, for the REINFORCE agent under the above setting, the following theorem quantifies the expected improvement in the training objective at step:: t:
Theorem 1. Consider the REINFORCE agent with softmar policy parameterization under the independent task setting as described above. Let st be the task picked at stept with PoSe, (s p and PoSg- (s = pp"
Then, we have: C((s) (0) = 2 2.th (1 f where te is the learning of the REINFORCE agent.
5

---

Published in Transactions on Machine Learning Research (05/2023)
For the above setting with (s) 1, Vs S, max Ci(s) is equivalent to max PoS,(s) (1
(s)) This means that for the case of PoS* (s) 1, Vs E Smit, the curriculum strategy given in Eq. can
be seen as greedily optimizing the expected improvement in the training objective at step t given in Eq 3
3.3 Curriculum Strategy for General Settings
Next, we discuss various practical issues in directly applying the curriculum strategy in Eq. for general settings, and introduce several design choices to address these issues.
Softmax selection. When training deep RL agents, is typically useful to allow some stochasticity the selected batch of tasks. Morcover, the arg max selection in Eq. T is brittle in the presence of any approximation errors in computing PoS(-) values. To tackle this issue, we replace arg max selection in Eq. [ with softmax selection and sample according to the following distribution:
P p (0) s] x exx (B (B.Po.((s) (PoS* PoSs(s))),
where 3 a hyperparameter. Here, (s) values are computed for cach s Sma using rollouts obtained via executing the policy n. in M; PoS*(s) (s) values are assumed to be provided as input.
(-) is not known. Since the target policy Tg. is unknown, it is not possible to compute the PoS values without additional domain knowledge. In our experiments, we resort to simply setting (s) Ys E Smit in Eq. 4 the ionale behind this choice is that we expect the ideal ma- to succeed in all the tasks in the pool This brings us to the following curriculum strategy referred to as PROCURL-ENY in our experimental evaluation:
(0) s] exp (B. C.....PSS) PoS.(s) -1- PoS) P P St
Computing (-S is expensive. It is expensive (sample inefficient) estimate PoS,(s) over the space Smit using rollouts of the policy Te- To tackle this issue, we replace PoS,(s) with values Vi(s) obtained from the critic network of the RL agent. This brings to the following curriculum strategy referred to PROCURL-VAL in our experimental evaluation:
(0) IP s
= s]
exp Vi((s -0:-011111 (1- (1- v.(s)).
(6)
Extension to non-binary dense reward settings. The current forms of PROCURL-VAL in Eq. i and PROCURL-ENV in Eq. 5 are not directly applicable for settings where the reward is non non-binary or dense. To deal with this issuc in PROCURL-VAL, we replace Vi(s) values from the critic in Eq. 6 with normalized values given by Ve(s)
Vi(e)-Vaiin clipped to the range [0, 1]. Here, Vman and Vmax could be provided Vinin
as input based on the environment's reward function; alternatively we can dynamically set Vmin and Viax during the training process by taking min-max values of the critic for states Smit at step t. To deal with this issue in PROCURT-ERNY, we replace (o),(s) values from the rollouts in Fq. 5 with normalized values (s)s) as above. Algorithm 2 in the appendix provides a complete pseudo-code for the RL agent training with PROCURL-VAL in this general setting.
4 Experimental Evaluation
In this section, we evaluate the effectiveness of our curriculum strategies on variety of domains w.r.t. the uniform performance of the trained RL agent over the training pool of tasks. Additionally, consider the following two metrics in our evaluation: (i) total number of environment steps incurred jointly by the teacher and the student components at the end of the training process; (ii) total clock time required for the training process. Throughout all the experiments, we use the PPO method from Stable- Baselines3 library for policy optimization (Schulman et al. 2017 Raffin et al. 2021b.
"This simple choice leads to competitive performance in varicty of cuviromments used our experiments. However, the above choice could lead suboptimal strategy for specific scenarios, e.g. when all PoS*(s) are below 0.5. It would be interesting to investigate alternative strategies estimate PoS* (s) during the training process, e.g, using top 5% rollouts obtained by executing the current policy m, starting from 8.
6

---

Published in Transactions on Machine Learning Research (05/2023)
Environment
Reward Context
State Action Pool size
POINTMASS-S
binary
B
R4
T2a
100
POINTMASS-D
non-binary
R
100
BASICKAReL.
binary
24000
{0, 1M
24000
BALLCATCHING
non-binary
R²
R
R
100
ANTGOAL
non- binay
(a) Complexity of the environments
(b) Illustration of the environments
Figure 1: (a) shows complexity of the environments w.r.t. the reward signals, context variation, state space, action space, and the pool size of the tasks used for training. (b) shows illustration of the environments (Trom left to right)): POINTMASS, BASICKAREL, BALLCAATCHING, and ANIGOAL. Details provided in Section 4.1
4.1 Environments
We consider 5 different environments in our evaluation, as described in the following paragraphs. Figure ! provides summary and illustration of these environments.
PointMass-s and PointMass-d. Based on the work of Klink et al. (2020b). we consider contextual POINTMASS enviromment where an agent navigates a point mass through a gate of given size towards a goal in a two-dimensional space. More concretely, we consider two settings: (i) POINTMASS-S environment corresponds to a goal-based (i.e., binary and sparse) reward setting where the agent receives a reward of only if successfully moves the point mass to the goal position; (ii) POINTMASS-D environment corresponds al. (2020b) where the reward values decay in squared
to a dense reward setting as used by Klink exponential manner with increasing distance to the goal. Here, the contextual variable R controls the position of the gate (C-GalePositionn) Position), the width of the gate (C-Gate Width)) and the friction coefficient of the ground (C- (C-Friction). We construct the training pool of tasks by uniformly sampling 100 tasks over the space of possible tasks (here, each task corresponds to a different contextual variable).
BasicKarel. This environment inspired by the Karel program synthesis domain Bunel et al. (2018), where the goal of an agent is transform an initial grid into a final grid configuration sequence of commands. In our BASICKAREL environment, we do not allow any programming constructs such as conditionals or loops and limit the commands to the "basic" actions given by A (move, turnLeft, turnRight, pickMerker putMarker, finish). A task in this environment corre- sponds to a pair of initial grid and final grid configurations; the environment is episodic with goal- base- (i.e., binary and sparse) reward sctting where the agent receives a reward of 1 only if succossfully transforms the task's initial grid into the task's final grid. Here, the contextual variable is discrete, where each lask can be considered as a discrete context. We construct the training pool of tasks by sampling 24000 tasks; additional details are provided in the appendix.
BallCatching. This environment is the same used in the work of Klink et al. (020b)); here, an agent needs to direct. robot to catch ball thrown towards it The reward function is sparse and non-binary, only rewarding the robot when it catches the ball and penalizing it for excessive movements. The contextual vector E 3 captures the distance to the robot from which the ball is thrown and its goal position in a plane that intersects the base of the robot. We construct the training pool of tasks by uniformly sampling 100 tasks over the space of possible tasks.
GntGoal. This environment is adapted from the original MuJoCo ANT environment (ooorov et al. 2012). In our adaptation, we additionally have goal on flat 2D surface, and agent is rewarded for moving an ant robot towards the goal location. This goal-based reward term replaces the original reward term of making the ant move forward; also, this reward term increases exponentially when the ant moves closer to the goal location. We keep the other reward terms, such as control and contact costs, similar the original MuJoCo ANT environment. The environment is episodic with a length of 200 steps. The goal location essentially serves as a contextual variable in R², We construct the training pool of tasks by uniformly sampling 50 goal locations from circle around the ant.
7

---

Published in Transactions on Machine Learning Research (05/2023)
These environments are goal-based and have an implicit way of defining successful trajectory. Typically, success is defined as a reward signal to the agent for approaching the goal, as done by Klink et al. (020b)) for POINTMASS, BALLCATCHING, and ANTGOAL. As future work, it would also be interesting to investigate the effect of our curriculum strategy on RL algorithms designed for the same goal-based setting but without assumiig that a goal proximily function is defined the environment. (Ding al. 2019
Eysenbach et al.
2022
Lin et al. 2019
4.2
Curriculum Strategies Evaluated
Variants of our curriculum strategy. We consider the curriculum stralegies PROCURI-VAL and PROCURL-ENV from Section 3.3 Since PROCURL-ENV uses policy rollouts to estimate (s),(s) in Eq. 5 it requires environment steps for selecting tasks in addition environment steps for training. To compare PROCURL-VAL and PROCURL-ENV in terms of trade-off between performance and sample efficiency, we introduce variant PROCURT-ENY* where x controls the budget of the total number of steps used for estimation and training. In Figure g variants with x E {2,4] refer to a total budget of about x million environment steps when training comprises of 1 million steps.
Statc-of-the-art baselines. SPDL, (Klink et al. 2020b) and SPACE (Fimer et al. 2021 are state-of-the- art curriculum strategies for contextual RL. We adapt the implementation of an improved version of SPDL, presented in Klink et al. (2021), to work with a discrete pool of tasks. We also introduce a variant of SPACE, namely SPACE-ALT, by adapting the implementation of Eimer et al. (2021) to sample the next training task as P(1))) (0) = s] xx exp (B- (Ve(s) - (=-) PLR (Jiang et al. 2021b) is a stale-of-the-arl curriculum strategy originally designed for procedurally generated content settings. We adapt the implementation of PLR for the contextual RL setting operating on fixed pool of tasks and include it an additional baseline.
Prototypical baselines. IID strategy randomly samples the next task from the pool; note that IID serves as competitive baseline since we consider the uniform performance objective. We introduce two additional variants of PROCURL-ENV, namely EASY and HARD, to understand the importance of the two terms PoS:(s) and (1 Ps)) in Eq. 5 EASY samples tasks as P( s] x exp (3 PoS:(s)), and
HARD samples tasks as (P(x = s] xx exp (3 (1 PO)S(s))))
4.3 Results
Convergence behavior and curriculum plots. As shown in Figure 2 the RL agents trained using the variants of our curriculum strategy, PROCURL-ENV and PROCURL-VAL, either match or outperform the agents trained with state-of-the-art and prototypical baselines in all the environments. Figures 4 and 5 visualize the curriculums generated by PROCURL-ENV, PROCURL-VAL, and IID; the trends for PROCURL. VAL generally indicate a gradual shift towards harder lasks across different contexts. The increasing trend in Figure Aaa corresponds to preference shift towards tasks with the gate positioned closer to the edges; the decreasing trend in Figure 4b corresponds to preference shift towards tasks with narrower gates. For BASICKAREL, the increasing trends in Figures Baaandsaaadddd 5b] correspon to preference towards tasks with longer solution trajectories and tasks requiring marker to be picked or put, respectively. In Figures 5c and 5d tasks with a distractor marker (C-DistractorMarker) and tasks with more walls (C- Walls) are increasingly selected while training.
Metrics comparison. In Figure 3 we compare curriculum strategies considered in our experiments w.r.t. different metries. PROCURL-VAL has similar sample complexity as state-of-the-art baselines since it does not require additional environment steps for the teacher component. PROCURL-VAL performs better com- pared to SPDL, SPACE and PLR in terms of computational complexity. The effect of that is more evident as the pool size increases. The reason is that PROCURL-VAL only requires forward-pass operation on the critic-model to obtain value estimates for each task in the pool. SPDL and SPACE not only require the same forward-pass pass operations, but SPDL does an additional optimization step, and SPACE requires a task ordering step. As for PLR, it has an additional computational overhead for scoring the sampled tasks. In terms of agent's performance, our curriculum strategics exceed or match these baselines at different training segments. Even though PROCURI-ANY consistently surpasses all the other variants in terms of perfor-
8

---

Published in Transactions on Machine Learning Research (05/2023)
PROCURI--EVV
VAL
III
SPDL
PLR
SPACE-AIT
SPACE
POINTMASS-S
POINTMASS-D D
BASICKAREL
BALL CALCCING
ANTGOAL
1.0
70-
1.0
70-
0.8
60
0.8
Performance
0.6
50
0.6
50
400
0.4
0.4
0.2
30
30
0.0
0.0
0.0
0.3
0.0
0.5
1.0 Steps
2
x100
110
100
X100P
110
Figure 2: Performance comparison of RL agents trained using different curriculum strategics described Section The performance measured as the mean reward (+1 standard error) on the training pool of tasks. The results are averaged over 20 random seeds for POINTMASS-S and POINTMASS-D, 10 random seeds for BASICKAREL and BALLCATCHING, and 5 random seeds for ANTGOAL. The plots are smoothed across 5 evaluation snapshots happening at over 25000 training steps.
Env
POINTMASS.S
BASICKAREL.
Method
Performance
Steps
Time
Performance
Steps
Time
0.25M
0.5M
1M
1M
1M
0.25M
0.5M
1M
1M
PROCURL-ENY
0.60 + 0.16
0.79 0.13
0.11
17.4 1.7
156
0.10 + 0.02
0.38 = 0.03
0.76 0.04
31.2
191
PROCURL-ENY
+ 0.15
0.15
0.11
+ 0.03
0.38 = 0.01
0.75 0.04
4.6
PROCURL-ENV
0.36 0.17
0.16
0.60
2.0 0.0
0.10
0.03
0.32 0.05
0.73 0.04
2.4
PROCURL-VAL
0.48 + 0.15
0.61
0.71
1.0 1 0.0
20
0.06 + 0.03
0.30 = 0.08
0.71 0.05
1.0 1 0.0
70
SPACE
0.05 + 0.06
0.17 + 0.12 0.12
0.29 0.15
1.0 + 0.0
22
0.04 + 0.02
0.11 0.03
0.30 0.04
1.0 1.0:+00
89
SPACE-AIT
0.22 0.12
D:37 + 0.15
0.46 0.17
1.0 + 0.0
21
0.05 + 0.03
0.18 0.06
0.50 0.08
1.0 0.0
69
SPDL
0.16
0.45
0.12
0.0
0.07 = 0.02
0.29 0.04
0.69 0.05
1.0
PLR
0.12
0.15
0.19
1.0 0.0
20
0.05 = 0.03
0.23 0.05
0.70 0.04
1.0
IID
0.27 0.15
0.34 0.177
0.36 0.19
1.0 1: 0.0
20
0.03 + 0.02
0.15 0.06
0.64 0.08
1.0 0.0
34
EASY
0.37 + 0.13
0.12
0.10
17.1 1: 2.3
154
0.04 + 0.01
0.07 = 0.02
0.11 0.03
22.6 0.9
126
HARD
0.01 + 0.01 0.00 + 0.00
0.01 0.01
37.0 + 0.7
3332
0.01 + 0.00
0.01 0.00 0.01
0.00
35.2 + 3.9
197
Figure 3: Comparison of different curriculum strategies described in Section 4.2 under the following metrics: (i) performance (mean reward + !xstandard error, where /. is the value from the t-distribution table for 95% confidence [Beyer] 2019h) of the RL agent at 0.25, 0.5, and 1 million training steps; (ii) total number of environment steps incurred at the end of 1 million training steps (this captures the sample efficiency of a curriculum stralegy); (iii) lotal clock time in minutes at the end of million training steps (this captures the computational efficiency of a curriculum strategy).
mance, its tcacher component requires a lot of additional environment steps. Regarding the prototypical baselines in Figure we make the following observations: (a) IID is strong baseline in terms of sample and computational efficiency; however, its performance tends to be unstable in POINTMASS-S environment because of high randomness; (b) EASY performs well in POINTMASS-S because of the presence of casy tasks in the task space of this environment, but, performs quite poorly in BASICKAREL; (c) HARD consistently fails in both the environments.
Ablation and robustness experiments. We conduct additional experiments to evaluate the robustness of PROCURL-VAL w.r.t. different values of 8 and different e-level noise in Vi(s) values. The results are reported in the appendix. From the reported results, note that picking a value for 8 somewhere between 10 to 30
9

---

Published in Transactions on Machine Learning Research (05/2023)
PROCURL-ENW PROD
PROCURL-ENV PROCURL-VAL.
PROCURLLENY PAOCURL-VAL
5.3
C-GuseMidhh E
C-Prectin
%
3.5
(d) At 0.25M
(c) At 0.5M
(I) At 0.75M
00
03
1.
05
0.2
0.6 Stags
0.4
0. Stegs
Steps
(a) C-Gate Position
(b) Width
(c) C-Friction
Figure 4: (a-c) Curriculum visualization of PROCURL-ENV, PROCURL-VAL, and IID in the PAINTMASS-S
environment; these plots show the moving average variation of the context variables of every 100 tasks picked by curriculum strategies during the training process (a picked task involves multiple training steps shown on the x-axis of plots). The increasing trend in (a) corresponds to a preference shift towards tasks with the gate positioned closer to the edges; the decreasing trend in (b) corresponds to a shift lowards lasks with narrower gates. (d-f) Illustrative tasks used during the training process for PROCURI-VAL (M is 10%).
PROCCRL-SNY
PROCURI PROCURLLVVE.
PROCoRI-TNY VROCURL.VAL
PRoCURR
VALL
PROCURL-VAL
[TD
IID
6.30
0.75
0.59
6.28
C-Derractr
0.50
C-TngLengh
5.0
0.65
0.54
C-Waade
6.15
0.52
1.8
8.00
6.s
0.55
0.48
0.46
595
Steps
x11
Steps
(a) C- C-rrgiLength
(b) C-Marker
(c) C- G-DistractorMarker
(d) C- Walls
(e) 0.1M
(f) At 0.25M
(g) At 0.5M
(h) At 0.75M
(1) At 1.5M
Figure 5: (a-d) Curriculum visualization of PROCURL-ENV, PROCURL-VAL, and IID in the BASICKAREL environment; these plots show the moving average variation the context variables every 500 tasks picked. The increasing trends in (a-d) correspond to a preference towards tasks: (a) with longer trajectories, (b) requiring marker action, (c) with more distractor markers, and (d) with more walls. (c-i) Illustrative (asks used during the training process at different steps for PROCURL-VAL (M is 105).
leads to competitive performance, and PROCURL-VAL is robust even for noise levels up to = 0.2. Further, we conduct an ablation study on the form of our curriculum objective presented in E.q. [ More specificall we consider the following generalized variant of Eq. I with parameters 7i and 72: (0) arg max (P)S. (71 PoS (s) 72 (o))( In our experiments, {0.6, 0.8, 1.0, 1.4). The results are reported in the appendix. From the reported results, note that our default curriculum strategy in Eq. (corresponding to 72/m1 = 1.0) leads to competitive performance.
consider the following range of 72/m E
5 Concluding Discussions
We proposed a novel curriculum strategy for deep RL agents inspired by the ZPD concept. We mathemati- cally derived strategy by analyzing simple learning settings and empirically demonstrated its effectiveness
10

---

Published in Transactions on Machine Learning Research (05/2023)
in variety of complex domains. Here, we discuss few limitations of our work and outline plan how to address them in future work. First, our experimental results show that different variants of our pro- posed curriculum provide an inherent trade-off between runtime and performance; it would be interesting to systematically study these variants to obtain a more effective curriculum strategy across different met- rics. Second, it would be interesting to extend our curriculum strategy to sparse reward environments with high-dimensional context space; particular, our curriculum strategy requires estimating the probability of success of all tasks in the pool when sampling a new task which becomes challenging in these environments. Third, extending the theoretical analysis of the curriculum strategy from independent task settings to cor- related task settings would be an interesting avenue to explore; this could involve developing generalized version of PROCURL curriculum strategy using a distance metric over the context space (Klink et al. 2022
Huang et al. 2022
Acknowledgments
Parameswaran Kamalaruban acknowledges support from The Alan Turing Institute. Runded/Co-funded by the European Union (ERC, TOPS, 101039090). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
References
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schncider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight Experience Replay. In NewrIPS, 2017.
Minoru Asada, Shoichi Noda, Sukoya Tawaratsumida, and Koh Hosoda. Purposive Behavior Acquisition for Real Robot by Vision-based Reinforcement Learning. Machine learning, 23(2-3):279 -03, 1996.
Adrien Baranes and Pierre- Yves Oudeyer. Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots. Robotics and Autonomous Systems, 61(1):49-73, 2013.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum Learning. In ICML, 2009.
William H Beyer Handbook of Tables for Probability and Statistics. CRC Press, 2019.
Rudy Buncl, Matthew J. Hausknecht, Jacob Devlin, Rishabh Singh, and Pushmect Kohli. Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis. In ICLR, 2018
Seth Chaiklin. The Zone of Proximal Development in Vygotsky's Analysis of Learning and Instruction. Vygotsky's 's Educational Theory in Cultural Contezt, pp. 39, 2003.
Patrick Dendorfer, Aljosa Osep, and Laura Leal-Taixé. Goal-GAN: Multimodal Trajectory Prediction based on Goal Position Estimation. In ACCV, 2020.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design. In NewrIPS, 2020.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned Imitation Learning. In NewrIPS, 2019.
Theresa Eimer, André Biedenkapp, Frank Hutter, and Marius Lindauer. Self-Paced Context Evaluation for Contextual Reinforcement Learning. In ICML, 2021.
Jeffrey L Elman. Learning and Development in Neural Networks: The Importance of Starting Small, Cog- nilion, 48(1):71-90) 1971-99, 1993.
11

---

Published in Transactions on Machine Learning Research (05/2023)
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Ruslan Salakhutdinov. Contrastive Learning Goal-conditioned Reinforcement Learning. In NewrIPS, 2022.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse Curriculum Generation for Reinforcement Learning. In CORL, 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic Goal Generation for Reinforce- ment Learning Agents. In ICML, 2018.
Alex Graves, Mare G Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated Curricu- lum Learning for Neural Networks. In ICML, 2017.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual Markov Decision Processes.
CoRR.
2015.
Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation. In NewrIPS, 2022.
Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. Self-Paced Curriculum Learning, In AAAI, 2015.
Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rock- läschel. Replay-Guided Adversarial Environment Design. In Newr/PS, 2021a.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized Level Replay. In ICML, 2021b.
Parameswaran Kamalaruban, Rati Devidze, Volkan Cevher, and Adish Singla. Intcractive Teaching Algo- rithms for Inverse Reinforcement Learning. In LJCAI, 2019.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A Survey Generalisation in Deep Reinforcement Learning. CoRR, abs/2111.09974. 2021.
Pascal Klink, Hany Abdulsamad, Boris Belousov, and Jan Peters. Self-Paced Contextual Reinforcement Learning. In CORL, 2020a.
Pascal Klink, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Self-Paced Deep Reinforcement Learning. In NewrIPS, 2020b.
Pascal Klink, Hany Abdulsamad, Boris Belousov, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning. Journal of Machine Learning Research, 22:182-1, 2021.
Pascal Klink, Haoyi Yang, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum Reinforcement Learning via Constrained Optimal Transport. In ICML, 2022.
M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-Paced Learning for Latent Variable Models. In NewrIPS, 2010.
Sergey Levine, Chelsea. Finn, Trevor Darrell, and Picter Abbeel. End-to-end Training of Deep Visuomotor Policies. Journal of Machine Learning Research, 17(1):1334-18) 1373, 2016.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Sil- ver, and Daan Wierstra. Continuous Control with Deep Reinforcement Learning. CoRR, abs/1509.02977 2015.
X. Lin, H. Baweja, and D. Held. Reinforcement Learning without Ground-Truth State. ICML'19 Workshop on Multi-Task and Lifelong Reinforcement Learning, 2019.
Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg, and Le Song. Iterative Machine Teaching. In TCML, 2017.
12

---

Published in Transactions on Machine Learning Research (05/2023)
Tambet Matissn,, Avital Oliver, Taco Cohen, and John Schulman. Teacher- Ceacher-Student Curriculum Learning. IEEE Tronsactions on Newral Nebworks and Learning Systems, (9))33332-3377 2019.
Volodymyy Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-Level Control Through Deep Reinforcement Learning. Nature, 518(7540):529--33) 2015.
Sanmit Narvekar and Peter Stone. Learning Curriculum Policies for Reinforcement Learning. In AAMAS, 2019.
Sanmit Narvekar, Jivko Sinapov, and Peter Stone. Autonomous Task Sequencing for Customized Curriculum Design in Reinforcement Learning. In IJCAI, 2017.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey. Journal of Machine Learning Research, 21:1-50, 2020.
Pierre-Yvv Yves Oudeyer, Frdric Kaplan, and Verena Hafner. Intrinsic Motivation Systems for Autonomous Mental Development. IEEE Transactions Evolutionary Computation, 11(2):265-286, 2007.
Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving Curricula with Regret-Basse Based Environment Design. CoRR,, abs//203.011m 2022.
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre- Yves Oudeyer. Automatic Curriculum Learning for Deep RI: A Short Survey. In IJCAT, 2021.
Sébastien Racanière, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated Curricula Through Setter-Solver Interactions. In ICLR, 2020.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-Baselines:: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Re- search, 22(268):1-8, 2021.
Cinjon Resnick, Roberta Raileanu, Sanyam Kapoor, Alexander Poysakhovich, Kyunghyun Cho, and Joan Bruna. Backplay: Man muss immer umkehren".. CoRR, abs/ hte/18007.0699,, 2018.
Martin A Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by Playing Solving Sparse Reward Tasks from Scratch. In ICML, 2018
Tim Salimans and Richard Chen. Learning Montezuma's Revenge from Single Demonstration. CoRR, abs/1812 A11/01111,3338, 2018
Jürgen Schmidhuber. Powerplay: Training an Incrcasingly General Problem Solver by Continually Scarching for the Simplest Still Unsolvable Problem. Fronliers in Psychology, 4:313, 2013.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Olog Klimov. Proximal Policy Opti- mization Algorithms. CoRR, abs/1707.06347, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the Game of Go Without Human Knowlodge. Nature, :334 359, 2017.
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeffrey Clune. Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data. In ICML, 2020.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Robert Fergus, Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play. In ICLR, 2018.
13

---

Published in Transactions on Machine Learning Research (05/2023)
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In NewrIPS, 1999.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A Physics Engine for Model-basee Control. In TROS, 2012.
Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe Reinforcement Learning via Curriculum Induction. In NeurIPS, 2020.
Lev Semenovich Vygotsky and Michael Cole. Mind in Society: Development of Higher Psychological Pro- cEsses. Harvard University Press, 1978.
Daphna Weinshall and Dan Amir. Theory of Curriculum Learning with Convex Loss Functions. CoRR, abs/1812.03472,, 2018.
Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks. In ICML, 2018
Lilian Weng. Curriculum for Reinforcement Learning. tiflanwengggthuub.o 2020.
URL https://
lilianweng. Lilianveng.github..o http://ppat//00000-------rr----e
Jan Wöhlke, Felix Schmitt, and Herke van Hoof. A Performance-Bate Based Start State Curriculum Framework for Reinforcement Learning. In AAMAS, 2020.
Yuxin Wu and Yuandong Tian. Training Agent for First-Person Shooter Game with Actor-Critic Curriculum Learning. In ICLR, 2016.
Scott Cheng Hssn Yang, Yue Yu, arash Givchi, Pei Wang, Wai Keen Vong, and Patrick Shafto. Optimal Cooperative Inference. In AISTATS, 2018.
Gaurav Raju Yengera, Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Curriculum Design for Teaching via Demonstrations: Theory and Applications. In NewrIPS, 2021.
Wojeioch Zaremba and Ila Sutskover. Learning to Exccute. CoRR, abs/1410.4615 2014.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic Curriculum Learning Through Value Disagree- ment. In NewrIPS, 2020.
Tianyi Zhou and Jeff Bilmes. Minimax Curriculum Learning: Machine Teaching with Desirable Difficulties and Scheduled Diversily. In ICLR, 2018.
Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Curricluum Learning by Optimizing Learning Dynamics. In ATSTATS, 2021.
Xiaojin Zhu, Adish Singla, Sandra Zilles, and Anna N. Rafferty. An Overview of Machine Teaching, CoRR, abs/1801.05927, 2018.
Xiaotian Zou, Wei Ma, Zhenjun Ma, and Ryan S Baker. Towards Helping Teachers Select Optimal Content for Students. In A ALED, 2019.
14

---

Published in Transactions on Machine Learning Research (05/2023)
A Table of Contents
In this section, we give a brief description of the content provided in the appendices of the paper.
Appendix B provides proof for Theorem and an additional theoretical justification for our curriculum
strategy. (Section 3.2
Appendix
C provides additional details and results for experimental evaluation. (Section 4
B Theoretical Justifications for the Curriculum Strategy - Proof and Additional Justification (Section 3.2)
B.1 Proof of Theorem
1
Proof. For the contextual bandit setting described Section 3.2.1 the REINFORCE learner's update rule
reduces to the following: 0c+1 + Oz + qe 1(8) = g} [V log T 1st )) In particular, for St =
(0)
and of
(0) 01, update:
0 0c+1 + 0.(s, ail + : 1(s = =g} (1 Te. (a [s)
04418,,021 f 02(s,a2l .l (( (1) =g} (1 To, (s)
and we set 0s+1ls, + 01(3, when it # or di # a1. Let st s, and consider the following:
(1(4 0,, Et)
100 0elli - 0c+1lli
|00[s, 0, [10 0.+1 -1l1
(01) 0,10, 0, + 0,(s,02) 02] 0" [s. 02]}
[0"(s, [s, 0,41(s, ai] + 0041(8,a2) az) 0 (s [s,,a])
(s,1 - 0, ail + 0,10, 02) - 0 00-10 02)
= 2 m/ (a (0) = s1 (1) = g} (1 - ne. T (ails)).
For the contextual bandit sctting, the probability of success is given by PoSo(s) = V=* Prand (s) Ta(ails), Vs E S. We assume that 3 H* such that To- (ails) above definition, the probability of success scores for any task associated with the starting state E Sinit w.r.t. the target and agent's 's current policies (at any step t) are respectively given by PoS*(s) PoSp- (s) = Prand(s) = p" and (s) = PoSe, (s) Praud (s) To, (ails) = p. Now, we consider the following:
here, т&- is the target policy. With the
Ci(s) Egcls [A (110+4100,,s50 s.E)
Bella 2 U1 1aa, (0) = =1, = g) (1 To. (ails))
= 2 The (s) Te, (ails) (1 no, (alls))
P
= 2. 2 pp
p"
B.2 Abstract Agent with Direct performance Parameterization
We consider abstract agent model with the following direct performance parametorization: for any E θ [0, |Smal we have (sS {[s), Vs Smit Under this model, the agent's 's current knowledge A, at step is
vector of "performance tasks" PoSg. Then, choose the parameter space as [0, Siui (where Smit) and define PoSş Thus, an update in the "parameter" equivalent update in the "performance on tasks" PoSg-
9In this setting, abstract the policy no and directly map the "parameter"
15

---

Published in Transactions on Machine Learning Research (05/2023)
encoded directly by its probability of success scores (PoSo, (s) is given by {PoSp- (s) 8 E Smit)- Under the independent task setting, we design an update rule for the agent to reflect the characteristics of the policy gradient style update. In particular, for s s. E Snits we update
Smit). The target knowledge parameter 0*
1 0.[s] + a suco(Er; (0*[s) 0,(s)) 3 (1- -1- succ(Ef 8)) (0*(s) 0cls)),
where 3 E [0, 1J and > 3. For Smit and + s!"), we maintain 0c+1[s] + 0c[s]- Importantly, > B implies that the agent's 's current knowledge for the picked task is updated more when the agent succeeds in that task compared to the failure case. The update rule captures the following idea: when picking a task that is "too easy", , the progress in 8g towards & is minimal since (0*[s] (s(s)) is low; similarly, when picking a lask that is "too hard", the progress in O, lowards 0" is minimal since B (0*[s] - 0c(sl) is low for B << 1. This idea aligns with the ZPD concept in terms of the learning progress (Vygotsky & Cole, 1978 Chaiklin 2003). For the abstract agent under the above setting, the following theorem quantifies the expected improvement in the training objective at step t:
Theorem 2. Consider the abstract agent with direct performance parameterization under the independent task setting as described above. Let ( be the task picked at step with PoSo, (8( = p and PoSo- (s, ( = p="
Then, we have: Cr(o))) (0)) a p.pp.pp. (p" p) + 3 (1 - p) (p' p))
Proof. Let st = s E Smit, and consider the following:
(0.11 Et) &,) = |0- 01111 110- 0 0141ll2
0c+1(s] [,(s]
a- succ(Eri s) (0"[s] - 0.(s)) 3- suucc((((;s)) (0* 0c(s)).
For the abstract learner model defined in Section B.2 we have PoSo(s) VT 0(s), for any E Smitt Then, the probability of success scores for any task E Sintt w.r.t. the target and agent's 's current policies (at any step t) are respectively given by (s) = PoSg-(8) = 0*[s] = p" and (oS,(s) = PoSa, (s) = 0c[s] = Now, we consider the following:
(s) Eg,ls [ si)]
Lg (ao-suos succ(Er;:))
0,())) + BB- suco(f);))) (0 (,(+))
Oc(o] (00 [00[s-00p0)) 0,(s)) 3 (1 0,(s)) (0'1s) 0,(s))
p- (p" pp) + 3 p) (p" -p).
For the above setting with a = 1 and 3 = 0, max Cs(s) is equivalent to max,es.... (sS,(s) (PoS" (s) PoS,(s)). This, in turn, implies that the curriculum strategy given in Eq. can be seen as greedily optimizing the expected improvement the training objective at step given in Eq. 3
C Experimental Evaluation Additional Details (Section 4
C.1 Environments
BasicKarel. This environment is inspired by the Karel program synthesis domain Bunel al. (2018), where the goal of agent is to transform an initial grid into final grid configuration by sequence of commands. In the BASICKAREL environment, we do not allow any programming constructs such as conditionals or loops and limit the commands to the "basic" actions given by the action space A = (move, turnLeft, turnRight, pickMerker putMarker, finish). A task in this environment corre- sponds to a pair of initial grid and final grid configurations. It consists of an avatar, walls, markers, and empty grid cells, and cach element has a specific location the grid. The avatar characterized by its current location and orientation. Its orientation can be any direction (North, East, South, West), and its location can be any grid cell, except from grid cells where wall is located. The state space S of
16

---

Published in Transactions on Machine Learning Research (05/2023)
BASICKAREL is any possible configuration of the avatar, walls, and markers in pair of grids. The avatar can move around the grid and is directed via the basic Karel commands, T.e., the action space A. While the avatar moves, if it hits wall or the grid boundary, it "crashes" and the episode terminates. If pickaaree is selected when no marker is present, the avatar "crashes" and the program ends. Likewise, the putMarker action is laken and a marker already present, the avalar "crashes" and the program terminates. The finish action indicates the end of the sequence of actions, i.c., the episode ends after encountering this action. To successfully solve a BASICKAREL task, the sequence of actions must end with a finish, and there should be termination via "crashes". Based this environment, we created multi-task dataset that consists of 24000 training tasks and 2400 test tasks. All the generated tasks have grid size of 4 x 4.
C.2 Evaluation Setup
Hyperparameters of PPO method. We the PPO method from HabbleBaseliness library with basic MLP policy for all the conducted experiments (Schulman et al.] 2017: Raffin et al.) 2021 For the POINTMASS-S, POINTMASs-D, and BALI.CATCHING environments, the MLP policy has shared layer with 64 units and a second layer with separate 64 units for the policy and 64 units for the value function. For the BASICKAREL environment, we use two separate layers size [512, 256] for the policy network and two layers of size [256, 128] for the value function nelwork. For the ANTGOAL environment, we use two separate layers of size [512, 512] for the policy network and two layers of size [512, 512) for the value function network. For all the experiments, ReLU is the chosen activation function. In Figure F we report the PPO hyperparameters used in the experiments. For each environment, all the hyperparameters are consistent across all the different curriculum strategies.
Hyperparameters
POINTMASS-D POINTMASS-S
BASICKAREL
BALLCATUHING
ANTGOAL
Natope
1024
1024
20-88
5120
1024
0.99
0.95
0.99
0.99
0.99
Nopocts
10
learning rate
3e-4
3e-4
3e
3-
2e-5
batch size
64
64
64
64
32
ent coef
0
0
0
5e-7 7
dlip range
0.2
0.2
0.2
0.2
0.1
gae lambda
0.95
0.95
0.95
0.95
0.8
max grad norm
0.5
0.5
0.6
noef
0.5
0.5
0.5
0.5
0.7
Figure 6: Different hyperparameters of the PPO method used in the experiments for each environment.
Compute resources. All the experiments were conducted on cluster of machines with CPUs of model Intel Xeon Gold 6134M CPU @ 3.20GHz.
C.3 Curriculum Strategies Evaluated
Variants of the curriculum strategy. Algorithm 2 provides a complete pscudo-code for the RL agent using PPO method when trained with PROCURL-VAL in the general setting of non-binary or dense rewards (see Section 3.3) In Eq. I and Algorithm [ we defined t at an episodic level; however, in Algorithm 2 denotes an environment step (in the context of the PPO method). For PROCURL-ENV, in line 24 of Algorithm
2 we estimate the probability of success for all the tasks using the additional rollouts obtained by executing the current. policy in M.
To achieve the constrained budget of evaluation steps in PROCURI-ENV (with x E (2,4)), we reduce the frequency of updating PoS, since this is the most expensive operation for PROCURL-ENV requiring additional rollouts for each task. On the other hand, PROCURL-VAI. updates PoS, by using the values
17

---

Published in Transactions on Machine Learning Research (05/2023)
obtained from forward-pass on the critic model this update happens whenever the critic model is updated (every 2048 training steps for BASICKAREL). This higher frequency of updating PoS, in PROCURT.VVAL is why it is slower than PROCURL-ENV* (with x {2, 4)) for BASICKAREL. Note that the relative frequency of updates for POINTMASS different in comparison to BASICKAREL because of very different pool sizes. Hence, the behavior total clock times is different.
Algorithm 2 RL agent using PPO method when trained with PROCURL-VAL in the general sctting
1: Input: RL algorithm PPO, rollout buffer D
2: Hyperparameters: policy update frequency Nstops: number of epochs Nopochs, number of minibatches
Nbatch, parameter 3, Vmin: and Vmax
Initialization: randomly initialize policy 21 and critic Vi; set normalized probability of success scores Vi(s) and PoS* PoS*(s) 1, Vs Sinit
4: for = , do
5: // add an environment step to the buffer
6: observe the state st, and select the action as ~ TT(st)
7:
execule the action de in the environment
8: observe reward rt, next state $t41, and done signal do+1 to indicate whether 8141 terminal
store (st, ot, St, St+1, d++1) the rollout buffer
10:
// choose new task when the current task/episode ends
11:
if d++1 true then
12:
reset the environment state sample next task 81+1 from P[s = s] x exp (B Ve(s) (1- ((1-V((s)))
13:
14:
// policy and (s))) update
15:
1%Nss = 0 then
16:
set n' + nT and V' + Ve for E = L. Nopochs do for b 1..... Nbatchh do sample b-th minibatch of Nsteps/Naatch Nbatch transitions B {(s, r, s &'d)) from D update policy and critic using PPO algorithm n', V' <- PPO(T', V)
17:
18:
19:
20:
21:
set Te+1 + n' and V++1 + V
22:
empty the rollout buffer D
23:
// normalization for the environments with non-binary or dense rewards
update V V++1(s)+ + Vi41(4)-Vmia Vs E Smnt using forward passes on critic
24:
Vuna Vusi
25:
else
26:
maintain the previous values TH11 + Tr, V411 + V1, and V + V.
27:
Output: policy øт
Hyperparameters of curriculum strategies. In Figure 7 report the hyperparameters of each curriculum strategy used in the experiments (for each environment). Below, we provide short description of these byperparameters:
1. 3 parameter controls the stochasticity of the softmax selection.
2. N parameter controls the frequency at which V, is updated. For PROCURL-ENV, we set Npos higher than Nsteps since obtaining rollouts to update (*))) is expensive. For all the other curriculum strategies, we set Npos= = Nsteps- For SPACE, Noos controls how frequently the current task dataset is updated based on their curriculum. For SPDL, Npos controls how often we perform the optimization step to update the distribution for selecting tasks.
3. Crollouts determines the number of additional rollouts required compute the probability of success score for each task (only for PROCURL-ENV).
4. (Vmin, Viax) are used the environments with non-binary or dense rewards to obtain the normalized values V(s) (see Section 3.3 In Figure {Vmin,t, V denote the min-max values of the critic for states Smit. at step f.
18

---

Published in Transactions on Machine Learning Research (05/2023)
5. 1y and и parameters used in SPACE [Eimer et al. 2021
6. VI.B performance threshold used in SPDL KKlink et al.
2021
7. p staleness coefficient and BPLR temperature parameter for score prioritization used in PLR Jiang
et al. 2021b
Method
Hyperparameters
POINTMASS-S
BASICKAREL PASNT
BALLCATCHING
ANTCOAL
20
10
10
10
10
PROCURL-ENV
Nocs
5120
5120
102400
20480
81920
Coollouts
20
20
20
20
20
(Vmmn Vinax
n/a
(Vist.ee kmax.r]
n/a
n/a
(0. 300)
20
10
10
10
10
PROCuRL-VAL
Npcs
1024
1024
2048
5120
1024
(Vmin- V
n/a
(Vmin,t, Vinse.
n/a
{0, 60)
(0. 300)
0.1
0.1
0.5
0.1
0.1
SPACE
1
64
Npw
1024
1024
2048
5120
1024
20
10
10
10
10
SPACE-ALT
Npos
1024
1024
2048
5120
1024
VLB
0.5
3.5
0.5
30
100
SPDL
Nyw.
1024
1024
2048
5120
1024
0.5
0.9
0.9
0.7
0.3
PLR
9PLa
0.1
0.3
0.1
0.3
0.1
Figure 7: We present the hyperparameters the different curriculum strategies for all five environments. For SPDL, we choose the best performing VLB the non-binary environments from the following sets: set
{1, 3.5, 10, 20, 30, 40} for POINTMASs-D. D; set {20, 25, 30, 35, 42.5) for CALCCATCHIG;; set (50, 100, 200, 300, 4003 for ANTGOAL. For PLR, choose the best performing pair (3PLR, p) for each environment from the set {0.1, 0.3, 0.5, 0.7, 0.9} x {0.1, 0.3, 0.5, 0.7, 0.9).
C.4 Additional Results
A bllation and robustness experiments. We conduct additional experiments to evaluate the robustness of PROCURL-VAL w.r.t. different values of B and different e-level noise in Vi(s) values. The results are reported
in Figure g
Further, we conduct an ablation study on the form of our curriculum objective presented in Eq. More specifically, we consider the following generalized variant of Eq. ! with parameters ?1 and 72:
arg max (PoS,(s) (71 Po)S(ss - PoS,(s)))
(7)
.ESm.
In our experiments, we consider the following range of 72/m E {0.6, 1.0, Our default curriculum strategy in Eq. essentially corresponds to 72/71 = 1.0. The results are reported in Figure g
Performance on test set. In Figure 10 we report the performance of the trained models in the training set and test set for comparison purposes. For POINTMASS-S, we constructed separate test sot of 100 tasks by uniformly picking tasks from the task space. For BASICKAREL, we have train and test dataset of 24000 and 2400 tasks, respectively.
We sought to assess the effectiveness of PROCURL-VAL. on tasks where IID does not perform well. To demonstrate this, we construct. a more challenging set of tasks for the POINTMASS- s environment. We generate half of these tasks by uniformly sampling over the context space. The remaining tasks are sampled from a bi-modal Gaussian distribution, where the means of the contexts [C-GATEPOSITION C-GATEWIDTHI are [-3,1] 1] and [3, 1] for the two modes, respectively. In Figure I1 present the results for PROCURI-VAL and IID, and in Figure 12 the different distributions.
Pool of harder tasks.
19

---

Published in Transactions on Machine Learning Research (05/2023)
Env
POINTMASS-S
BASICKAREL.
Method
PROCeRI.VVAL
Performance
Performance
0.25M
0.5M
1M
0.25M
0.5M
IM
= 10
0.48 + 0.17
0.58 + 0.19
0.70 + 0.18
0.06 0.03
0.30
0.08
0.71 0.05
0.42 1: 0.17
0.64
0.17
0.74 0.15
0.04
0.38
0.04
0.71
0.48
0.17
0.18
0.06
0.75
0.45
0.19
0.21
0.03
0.04
0.62
30
0.54 0.18
0.64 0.20
0.74 0.19
0.20 0.06
0.36 0.07
0.67 0.07
0.15
0.64 0.17
0.18
0.03
0.30 0.08
0.71
0.53
0.19
0.20
0.02 0.30
0.69
0.05
0.16
0.60 0.17
0.70 0.20
0.06 0.02 0.31 = 0.06
0.72 0.04
0.1
0.17 0.59 + 0.16
0.67 0.18
0.06 0.03 0.30 0.07
0.69 0.07
0.2
0.49 0.16 0.61 + 0.18 0.68 + 0.18
0.04 + 0.02 0.26 0.08
0.74 + 0.03
Figure 8: Robustness of PROCURL-VAL w.r.t. different values of 3 and different e-level noise Vi(s) values. We present the results for the POINTMASS-S environment and BASICK AREI environment. We report the mean reward (+ txstandard error, where the value from the t-distribution table for 95% confidence) 0.25, 0.5, and million training steps averaged over 20 and 10 random seeds, respectively.
Env
POINTMASS
BASICKAREL.
Method
PROCURL-VAL
Performance
Performance
0.25M
0.5M
1M
0.25M
0.5M
1M
7a/71 0.6
0.33 0.14
0.50 = 0.14
0.55 + 0.13
0.08 = 0.04
0.21 0.07
0.41 = 0.08
7a/71 0.8
0.26 -
0.43 0.17
0.55 0.20
0.11 = 0.03
0.36
0.05
0.64 = 0.07
7a/7 1.0
0.48 0.15
0.64 0.17
0.71 + 0.18
0.06 + 0.03
0.30 = 0.08
0.71 = 0.05
72/71 1.2
0.42
0.55 = 0.19
0.65 0.65-0.11 0.17
0.07 = 0.04
0.30 0.11
0.72 = 0.08
72/71 1.4
0.39
0.59 0.17 0.59 0.15
0.04 + 0.02 0.21 0.08
0.71 + 0.06
Figure Performance comparison of the generalized form of curriculum strategy presented in Eq. w.r.t. different values of 72/11 We present the results for the POINTMASS-S environment and BASICKAREL environment. We report the mean reward (+ txstandard error, where is the value from the t-distribution table for 95% confidence) at 0.25, 0.5, and 1 million training steps averaged over 20 and 10 random seeds, respectively.
20

---

Published in Transactions on Machine Learning Research (05/2023)
Env
POINTMASS-S
BASICKAREL
Method
Performance (1M)
Performance (2M)
Thain Set
Tist. Set.
Train Set.
Test: Set
PROCURLAENY
0.84
0.78
0.92
0.90
PROCURL-VAL.
0.71
0.65
0.91
0.90
SPACE
0.34
0.28
0.65
0.64
SPACE-AIT
0.47
0..0
0.82
0.81
SPDL
0.55
0.48
0.88
0.87
PLR
0.69
0.60
0.88
ITD
0..32
Figure 10: Performance of the curriculun strategies, discussed in Section 4.2 in the training set and a test set. We report the performance, i.e., expected mean reward, of the best model obtained during training for all the methods. The training steps to achieve this performance is shown in parenthesis for each environment (M is 10 steps). We present the results for the MASS- environment and BASICKAREL environment and report the mean reward averaged over 20 and 10 random seeds, respectively.
Env
POINTMASS-S
Method
Performance
0.25M
0.5MI
IM
1.5M
2M
PROCURL-VAL
0.07 = 0.07
0.19 0.11
0.40 # 0.15
0.46 = 0.16
0.49
0.19
HID
0.01 = 0.01
0.03 0.03
0.05 # 0.06
0.04 = 0.04
0.03
0.03
Figure 11: Performance comparison of our eurriculum strategy, PROCURL-VAL, and IID pool of harder tasks for the POINTMASS-S environment. We report the mean reward (+: Ixstandard error, where t is the value from the t-distribution table for 95% confidence) at 0.25, 0.5, 1, 1.5 and 2 million training steps averaged over 20 random seeds.
8
8
0.014
-..150
6.5
-0.013
6.5
C-Gate C-GateWidthh Width
0.012
C-Gate C-GateWidth Width
0.125
5
0.011
5
0..00
3.5
0.010
3.5
0.075
-0.009
0.055
2
0.008
2
0.025
0.007
0.5 -4
0.5
0.000
2
-4
2
C-GatePosition
C-GatePosition
(a) Distribution for uniform pool of tasks
(b) Distribution for harder pool of tasks
Figure 12: (a) shows the distribution of context values used to generate the uniform pool of lasks for the main experiments (b) the distribution of context values that is used generate a harder pool of tasks.
21


Reviews:
[{"review_id": "Lct5xwwrgu", "reviewer": "Reviewer_F5vg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Using the idea of a ZPD is well motivated.\n* Empirical evaluation compares a number of variants of the proposed method (Monte Carlo estimates of success vs a function approximators) and relevant baselines across a decent number of environments. The proposed methods performs well.\n\nWeaknesses:\n* I have concerns about the theoretical results that serve as motivation. Specifically, it seems to me that there is no guarantee in general (and certainly not in discrete environments) that the set $\\mathcal{D}_t(p^*, p)$ is non empty for most p. Certainly the differentiability of $C_t(p^*, p)$ depends on $\\mathcal{D}_t(p^*, q)$ being non-empty values of q neighbourhood of $p$. \n* I find the theoretical results hard to read and the notation for e.g. $\\theta$ confusing. I think the writing in this section could be improved. I would also not call $\\theta$ a knowledge parameter. \n* I'm also not convinced that the theoretical results sufficiently elucidate the method to be worth including even if correct. Eg. the method drops $PoS*$ entirely. This also raises the question of what happens in situations where some tasks are very hard or impossible.\n* In the empirical section I wonder about the set of baselines. What is the reason for not comparing to prioritized level replay [1] adapted to this setting? To my knowledge PLR is a state-of-the-art curriculum learning method that has been shown to scale to quite large domains [2]\n\n[1] Jiang, Minqi, Edward Grefenstette, and Tim Rocktäschel. \"Prioritized level replay.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Team, Adaptive Agent, et al. \"Human-Timescale Adaptation in an Open-Ended Task Space.\" arXiv preprint arXiv:2301.07608 (2023).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new method for curriculum learning that selects tasks (in this work initial states) such that are neither too easy nor too hard. The proposed strategy roughly amounts to choosing tasks that maximise ```current success rate * (best success rate - current success rate)```.  The paper is inspired by the concept of \"zone of proximal development\" from the educational psychology literature. \n\nThe paper provides some theoretical results to motivate the proposed method as well as an empirical evaluation on a number of different environments. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Please clarify why $C_t(p^*, p)$ is differentiable in your examples (or in general)\n* Please justify the choice of baselines.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8DtI4W1QZw", "reviewer": "Reviewer_Brna", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and easy to follow. The authors discuss the influence of hyperparameter choices and provide proof for their theoretical results in the appendix. The authors even evaluate an additional \n\nRegarding weaknesses, I have a few minor points:\n * I already reviewed this paper for ICLR 2022 (where it did not get accepted). Back then, a reviewer proposed a different implementation of SPDL that improved performance. While the results mostly agree with the results reported by the reviewer back then, there seems to be a rather strong performance difference for SPDL in the PointMass-S environment. The reviewer reported an average performance of $0.74 \\pm 0.03$ computed from 10 seeds whereas the results here show only a performance of $0.52 \\pm 0.08$. Can the reviewers explain where this strong difference comes from?\n * The extra time parameter $H$ is a bit confusing to me. For finite-horizon MDPs, it is typically important to provide the time step as part of the state. I cannot find this time index being passed to e.g. the policy in the notation. Is this because the time step is assumed to be contained in the state implicitly? If so, I'd like to ask the authors to clearly state this detail.\n * An inherent weakness of the theoretical investigations is that the authors do not allow parameter (i.e. information) sharing between contexts. While I understand why this is done, I would like the authors to discuss this limitation a bit more in the paper. What could be the next steps for incorporating the aspect of information sharing in their theory? Such a discussion would greatly help people in pushing the theoretical investigations further.\n * Before Equation (2), the authors could provide a reference to the paper by Graves et al. (already in the references) and make clear that Equation (2) is an approximation to what Graves et al. refer to as expected learning progress.\n * I am wondering whether Section 3.2.1 is beneficial to the overall paper. What I dislike about this section is that a hypothesis is made about the learning rule and the agent parameters are detached from the actual behavior. Section 3.2.2 seems to offer the same insights while having a much clearer motivation (as a contextual bandit problem) and using the REINFORCE algorithm. Could the authors discuss why they would like to keep Section 3.2.1? Or would they be willing to move this section to the appendix and instead provide more details for the results in Section 3.2.2?\n * An improvement for Section 3.2.2. could e.g. be to visualize the quantities that are shown in Theorem 2 and further discuss how $\\max_p C_t(p_{\\text{rand}}(s), p)$ begins to differ from $\\max_p p \\cdot (1 - p)$ when $p_{\\text{rand}}(s)$ increasingly differs from 1.\n * The caption of Figure 2 left me a bit puzzled as to how many seeds were used for which environment. Could the authors edit this caption to clearly say how many seeds were used for which environment?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces an approach for the automated generation of curricula based on the idea of a Zone of Proximal Development, intuitively aiming to select tasks that are neither too hard nor too easy. The authors formulate this selection as the product between current agent performance and regret. This particular choice is motivated by theoretical investigations in two examples, in which the authors can show that their strategy leads to an optimal improvement of overall success.\n\nThe authors then carry on to compare their proposed methods against different baselines, showing good performance. \n\nAll in all, I agree with the contributions that the authors claim to make in this paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to ask the authors to clarify open questions (performance difference of SPDL for PointMass-S, finite-time horizon $H$, importance of Section $3.2.1$, number of seeds used for evaluation). Apart from this, I would like to see a discussion of ways to extend the current theoretical analysis. \nFinally, I think that making the results of Theorem 2 more intuitive (e.g. via visualizations) as well as discussing the influence of $p_{\\text{rand}}(s)$ would give additional insights.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VHCmopoOB6", "reviewer": "Reviewer_jtz6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Simplicity -- It seems straightforward to implement the method. I like the idea of using the learned value function to estimate the probability of success.\n* The writing is generally clear, and the explanation of the prior methods is good.\n\n\nWeaknesses\n* The tasks in the experiments are pretty easy; it'd be great to include at least one really compelling task that cannot be solved without the curriculum. \n* I wasn't able to understand the theoretical justification. I was hoping to see a result that said something like \"selecting tasks using the proposed method decreases the cumulative regret from O(stuff) to O(stuff), as compared with iid task selection.\" As is, Theorem 1 doesn't mention any objective criterion; Sec. 3.2 mentions differences in parameters (Eq. 2), but that is never precisely related to differences in values (the Lipschitz constant L isn't assumed) nor to the update rule in Sec. 3.2.1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method to use curriculum learning to accelerate reinforcement learning (RL). Intuitively, the proposed method samples tasks that are more difficult than those that the RL agent has currently mastered, but not too much more difficult. More precisely, tasks are defined by different initial states, each of which is evaluated using a learned value function. The proposed method samples initial states that both (1) have a large value, and (2) have a large regret relative to the optimal policy. The proposed method is straightforward to implement and achieves good results on benchmark goal-reaching tasks from prior work. The paper also provides some analysis arguing why the proposed method is theoretically justified.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "High-level comments\n* Can we interpret the selection criterion PoS(s) x (PoS*(s) - PoS(s)) as the geometric mean of (1) the probability of solving the task and (2) the expected regret on this task? These are both natural quantities to maximize (w.r.t. curriculum learning). If so, it might be good to include this intuition.\n* \"at the goal states\" -- How does this work in continuous environments, where \"at\" isn't well defined?\n* Eq. 6 -- Does this amount to a Gaussian distribution over _values_? In expectation, do the sampled tasks have a probability-of-success of $\\frac{1}{2}(PoS(s) + 1)$?\n* Given that all the tasks are sparse goal-reaching problems, I'd recommend using an RL algorithm designed for that setting (e.g., [1,2,3]). This will _decrease_ the assumptions needed by the method, as it will no longer require manually-defining a function to say whether you've gotten close enough to the goal. [See comment about \"at the goal states\"]\n\nMinor writing comments\n* \"Recent works have\" -- Cite.\n* Check citep vs citet. E.g., \"in both (Florensa\" should use citet.\n* \"The authors in (...) propose\" -> \"... propose\"\n* Notation -- I'd recommend using $s^{(i)}$ instead of $s^{(\\tau)}$, as some readers will expect that $\\tau$ is the trajectory ($\\xi$) rather than an index.\n* Footnote 1 -- I'd recommend writing this out in words, using an example. For readers not familiar with this construction, the notation can seem obtuse.\n* sample efficient manner\" -- Check grammar here. I think \"manner\" should be plural, or an article should be added.\n* Fig 1b -- I'd recommend showing a real rendering of the BallCatching environment.\n* Fig 3 -- Tables with many numbers are hard to read. I'd recommend replacing this with a bar plot, and putting the table in the appendix.\n* Fig 4, 5 -- I found it hard to interpret these figures. I'd recommend expanding the captions to provide some intuition. E.g., what exactly do the Y axes correspond to, what's the intuition for why they should be large at some time and small at other times?\n\n-----------------\n\n[1] https://proceedings.neurips.cc/paper/2019/file/c8d3a760ebab631565f8509d84b3b3f1-Paper.pdf\n\n[2] https://arxiv.org/pdf/2206.07568.pdf\n\n[3] https://arxiv.org/pdf/1905.07866.pdf\n", "broader_impact_concerns": "This paper doesn't have direct broader impact concerns. That said, it might be good to at least mention the potential for RL to have large positive/negative societal impacts", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers had an open discussion with the authors, and any issues were addressed there. The recommendations from the reviewers had no additional information, simply stating that the authors had answered their questions and supported the work being accepted.

Let me suggest one comment to minorly improve the work.

It would be useful to more clearly highlight at least in the table, in Figure 3, if the differences are significant. Standard errors can be a bit misleading, and it is likely better to at least include 95% confidence intervals for a student-t distribution. Here that effectively means multiplying the standard error by 2. I understand that this does not change much (and so does not seem to be pointful), but it does at least reflect the number of runs used for each environment. For example, an experiment with only 3 runs would have a notably larger multiplier. 

I am accepting this paper without this as a required revision, since it can mostly be inferred from the current table. But nonetheless I encourage the authors to consider 95% student-t intervals as a bit more informative than simply standard errors, and to justify why it is reasonably to report student-t intervals (already, those are also making strong assumptions too).
Decision: Accept (poster)