Safety-Prioritizing Curricula for Constrained Reinforcement Learning
Authors: Cevahir Koprulu
Thiago D. Simão
Nils Jansen
ufuk topcu
Venue: ICLR 2025 Poster (2025)
Keywords: curriculum learning
constrained reinforcement learning
Primary area: reinforcement learning
TL;DR: We propose a safe curriculum generation method that reduces safety constraint violations during training while boosting the learning speed of constrained RL agents.

Abstract: Curriculum learning aims to accelerate reinforcement learning (RL) by generating curricula, i.e., sequences of tasks of increasing difficulty. 
Although existing curriculum generation approaches provide benefits in sample efficiency, they overlook safety-critical settings where an RL agent must adhere to safety constraints.
Thus, these approaches may generate tasks that cause RL agents to violate safety constraints during training and behave suboptimally after. 
We develop a safe curriculum generation approach (SCG) that aligns the objectives of constrained RL and curriculum learning: improving safety during training and boosting sample efficiency.
SCG generates sequences of tasks where the RL agent can be safe and performant by initially generating tasks with minimum safety violations over high-reward ones.
We empirically show that compared to the state-of-the-art curriculum learning approaches and their naively modified safe versions, SCG achieves optimal performance and the lowest amount of constraint violations during training.

Published as a conference paper at ICLR 2025
SAFETY-PRIORITZING CURRICULA FOR CONSTRAINED CEINFORCEMET LEARNING
Cevahir Koprulu'- Thiago D. S Nils Jansen
Ufuk Topeu'
'The University of Texas at Austin Findhoven University of Technology
(Ruhr-Univerye Bochum
ABSTRACT
Curriculum learning aims to accelerate reinforcement learning (RL) by generat- ing curricula, i.e., sequences of tasks of increasing difficulty. Although existing curriculum generation approaches provide benefits in sample efficiency, they over- look safety-critiek settings where RL agent must adhere to safety constraints. Thus, these approaches may generate tasks that cause RL agents to violate safety constraints during training and behave suboptimally after. We develop safe cur- riculum generation approach (SCG) that aligns the objectives of constrained RL and curriculum learning: improving safety during training and boosting sample efficiency. SCG generates sequences of tasks where the RL agent can be safe and performant by initially generating tasks with minimum safety violations over high- reward ones. We empirically show that compared to the statc-of-the-an curriculum learning approaches and their naively modified safe versions, SCG achieves optimal performance and the lowest amount of constraint violations during training.
INTRODUCTION
Curriculum learning for reinforcement learning (RL) aims generate task sequences that boost the performance and speed of convergence of RL agents (Narvekar et al. 2020). A common strategy in curriculum generation is to start with easy tasks and adjust the difficulty toward the target tasks as the RL agent improves. Automating curriculum generation increases sample efficiency in wide-ranging environments (Baranes & Oudeyer 2010 Jiang et al., 20216) with minimum human effort.
The typical exploration by trial-andeeree medorrror in RL may cause unsafe behaviors during training, making such techniques unsuitable for safety-critical scenarios (Kendall et al. 2019). In addition to improving sample efficiency, curriculum learning can potentially mitigate this issue by prioritizing tasks with no or low potential for harm so that an RL agent can learn how to accomplish a task without behaving unsafely (Turchetta et al. 2020). For example, curriculum can start by proposing to agent learning how to drive a traffic scene without cars and pedestrians, minimizing the risk of accidents.
Constrained RL addresses safety-critich scenarios where, given a safety threshold, constraint on cost function characterizes safe behavior (Altman) 1999). A constrained RL agent aims to maximize its reward while satisfying such constraint (Achiam et al. 2017). A standard metric for safety violations during training is the constraint violation regret, i.e., accumulated excess cost over the safety threshold (Efroni et al.) (2020). Curriculum learning approaches overlook constrained RL and fail to consider the cost constraint. They cannot distinguish unsafe behaviors, and propose tasks that, while yielding high rewards, also incur high costs, leading to high constraint violation regret.
A standard curriculum learning method aims to help an RL agent achieve higher rewards faster. Thus, a naive combination of off-the-shelf curriculum learning approach and constrained RL algorithm fails to minimize constraint violation regret due to their misaligned objectives. In comparison, a constrained RI. algorithm searches for policies that primarily satisfy the cost constraint while maximizing reward as much as possible. Given such a combination, the curriculum generator can propose a task that allows the agent to collect high rewards and simultaneously costs higher than the safety threshold, which violates the constraint. Therefore, we argue that a safe automated curriculum generation method should actively prioritize tasks aligned with both objectives.
"Correspondence Cevahir Koprulu ccevahir.koruuut utexas.edu). Code

---

Published as a conference paper at ICLR 2025
Yee
Ves
Performmnt?
Conteat dittitution
No
Prioritios Innestott
Prioriitoo Highrewwnl
Appineac baygtt distcilaatin
Sumple coatlecia
Constrained RI.
Artion
Contect
Polley update
Obscrvation Eoward and
Figure 1: SCG initially prioritizes low-cost contexts to minimize safety violations, then high-reward contexts to boost performance, and finally approaches target distribution by treating them equally.
We propose a safe curriculum generation method (SCG) that improves performance, accelerates learning, and minimizes safety violations during training (Fig. T. Inspired by CURROT (Klink et al. 2022), given a distribution over target tasks, SCG generates a sequence task distributions that allows the current policy to collect higher rewards than performance threshold and lower costs than a safety threshold. Initially, SCG prioritizes safety over performance by proposing tasks where the agent satisfies cost constraint. Once the agent behaves safely in all possible tasks under the current distribution, SCG shifts its focus to satisfying a performance constraint. After the agent becomes performant in all contexts the current support, SCG generates task distributions that approach the target distribution by equally treating safety and performance until the end of the training.
Contribution. Our contribution three-fold: We describe how existing curriculum learning approaches fail to learn an optimal behavior in constrained environment safely in Section 4.2 2) pro- pose Safe Curriculum Generation (SCG), an automated curriculum learning approach developed for constrained RL to boost learning speed and minimize constraint violation regret in Section 5 and lastly 3) our empirical results provide evidence that, compared to the state-of-the-art curriculum gen- eration approaches and their naively modified versions that account for safety, SCG achieves optimal behavior with the lowest constraint violation regret in constrained RL environments Section
RELATED WORK
Curriculum learning for RL. Automated curriculum generation in RL aims to accelerate conver- gence to optimal policies by changing the environment configuration according to agent performance. typical curriculum a sequence of distributions over such configurations, Florensa et al.) (2017) propose generating distributions over initial states where, early on during the training, the agent starts nearby the goal. Other works focus on goal states by optimizing for value disagreement (Zhang et al., 2020), intrinsic motivation (Baranes & Oudeyer 2010 Portelas et al. 2020). fcasibility and coverage of goal states (Racaniere et al. 2020). and intermediate task difficulty (Florensa et al. 2018 Tzan netos et al. 2023). Dennis al. (2020) propose unsupervised level design
alternative curriculum learning paradigm and an approach that adversarially generates environment configurations while avoiding infeasible ones. Jiang et al. (202 a) study generating distributions over levels, namely, environment instances, that allow the agent to have high learning potential. Due to its recent success, study self-paced RL, a method adopted from supervised learning to order training samples in increasing complexity (Kumar et al. 2010 Jiang et al. 2015). Eimer et al. (202)) generate sequences of tasks with high capacity for value improvement. Ren et al. (2018) minimizes coverage penalty by generating sequences of environment interactions. Klink et al. (2020a b 2021 2022); Koprulu &
Topcu (2023): ; Koprulu et al. (2023); Huang et al. (2022) formulate the curriculum generation problem interpolations between task distributions. In contrast, Chen al. (2021) proposes gradient-basee exploration method via variational inference to expand the task distribution to the entire task space.
Curriculum learning for safety during training. Most curriculum learning approaches overlook the safety aspect of RL. Nevertheless, there are methods akin to curriculum learning to increase safety during training. [Song & Schneider (2022) propose genetic algorithm to generate curricula that improve robustness but does not consider an explicit notion cost nor study environments that highlight the misalignment between performance and safety. Wang al. (022) develop curriculum-
2

---

Published as a conference paper at ICLR 2025
guided RL approach for real-time bidding systems that relaxes cost constraints to incentivize safe policies during training. Eysenbach et al. (2018) learn a reset policy that interferes with the training to prevent the agent from entering dangerous states. Similarly, Turchetta et al. (2020) learn curriculum policy that chooses an intervention that takes the agent to a safe state if it enters a trigger state. EEysebbach et al. (20T8)'s 's approach trains the reset policy and the RI. agent together, whereas Turchetta et al. (2020) consecutively train multiple students to learn an optimal curriculum policy. In comparison, existing curriculum learning methods do not interfere with the interactions between the student and the environment but only assume that a teacher can set the environment configuration for which the agent learns an optimal behavior (Florensa et al., 2017] 2018 Portelas et al. 2020)
Jiang et al. 202 la b Klink et al. 2020a 2021 2022). Similarly, SCG does not assume control over environment dynamics, even when the student violates the constraint, [Tomilin et al. (2025) propose a new benchmark for constrained RI. and shows the benefits of using handcrafted curriculum.
Constrained RI.. Constrained RL studies safety-critical settings where errors during exploration may cause constraint violations (Kendall et al. 2019 Roy et al. 2022 Kamran et al. 2022). Therefore,
constrained RL approach aims to achieve safe behavior during and after training (Müller et al. 2024
Simão et al. 2021). Constrained RL approaches that guarantee zero safety violation during training
propose using Gaussian processes as transition models (Sui et al.] [2013: [Berennkmmp et al., 2017
Turchetta et al. [019 Wachi & Sui 2 Lyapunov functions for ensuring global constraints CCow
et al.] 2018), or formal methods (Junges et al. 2016 Alshiekh et al.] 2018 Jansen et al. (020). To
address high dimensional state and action spaces, Achhiam et al. (2017); Tessler et al. (2019): Yang
iet al. (2020): Hogewind et al. (2022) develop safe policy search algorithms with soft guarantees of not violating the constraints, whereas Achiam & Amodei (2019) integrates a Lagrangian approach into popular RL algorithms. Similar to our setting of interest, [in et al. (2024) addresses constrained contextual MDPs, where the context determines the dynamics, yet the reward and cost functions are fixed. Furthermore, Lin et al. (2024) focuses on the safety aspect of offline RL and does not study curriculum learning. Another line of work introduces a framework where dynamics, reward, and cost functions remain the same, yet safety thresholds vary (aao et al. 2023 Günster et al. 2024)..
3 BACKGROUND AND PROBLEM STATEMENT
We formulate the environments of interest contextual constrained Markov decision processes to model a constrained multi-task setting given a distribution over target contexts.
3.1 CONTEXTUAL CONSTRAINED MDPS
Definition 3.1. We define a contextual constrained Markov decision process (CCMDP) M (S, A, X, M, r) with a state space S, an action space A, a context space X C R" for n E Zz, mapping from context space to constrained Markov decision process parameters M, a safety threshold D E R2o, and a discount factor E [0, 1].
CCMDP M represents a family of constrained MDPs parameterized by their contexts x E X. A context x provides a constrained MDP M(x) = AS Px, Tx; Cx Po,x, 7), where S, A, and ? are Ax,
the same as in M, but its probabilistic transition function px: S x A + A(S), reward function rx: S x A + R. cost function cx S x A + 20 and initial state distribution Po.x E A(S) depend on its context x. A policy x: S x A x X + ((A) defines the behavior of an agent in a CCMDP M as a probability simplex over the action space given S E and x E X. Note that the agent observes the context x. Following policy an agent collects a trajectory Tx = [(st, Tt, ct) T of length with an initial state So ~ Po.x; states S1+1 ~ Px( St, at), actions a; ~ x x-||s, x). rewards Cx (St, ax) at time steps t E [T'., resulting in a discounted cumulative
r; (S; az), and costs e;- reward and cost Gx(Tx) Et 7r,, and Gx(Tx) Σt 7/0,, respectively.
Given CCMDP M and a target context distribution yp, i.e., probability simplex A contextual constrained RL aims to maximize expected return subject cost constraint:
- arg max E.
VV" (x)).
s.t. E. [V" $ D,
(1)
where -.- F. po. G. and V -E E., po, , [G, (Tx are the expected discounted cumulative reward and cost, respectively, induced by policy π in context x drawn from y.
3

---

Published as a conference paper at ICLR 2025
Fig. shows safery-maze, constrained version of maze (Klink al. 2022
with simple dynamics on 2D state and action spaces, namely, the agent' s coordinates and its displacement. respectively. Starting from the bottom left corner (green), the agent must reach goal while avoiding the hazards (red), where it collects cost. The context specifies the goal position and tolerance, i.c., the Euclidean distance to the goal for success. The agent can move freely over the white areas but cannot access the walled sections in black. Note that the goal can be positioned over the walls. Safety threshold D determines how much the agent needs to avoid the hazards. An episode terminates when the agent reaches the goal. The target context distribution uniform over the top white horizontal area.
Figure 2: Safety-maze.
3.2 CONTEXTUAL CONSTRAINED RL
Contextual-constrainee RL (CCRL) is online multi-task-constrriine RL framework that does not assume access to the transition, reward, and cost functions. As the optimal policy maximizes the expected discounted return while satisfying cost constraint, CCRL algorithm should focus sample efficiency as well as safety. One metric to measure safety is constraint violation regret, the difference between the safety threshold and the value of learned policy (Efroni et al. (020). Given that a CCRL algorithm A runs for I--any episodes during training, we define the training regret as
(ethee,, D)
Σ LE[L]
Ee [V. [x)] - -D
where [u]+ max 0}, xı refers to the policy at the 1th episode, and @: is the context distribution from which x is drawn episode 1. The regret non-zero only when the expected discounted cumulative cost exceeds the safety threshold D. Thus, training regret Reg" only considers the safety violations an algorithm A with respect to context distributions {er) Once converges to an optimal policy, its training regret also converges. Our problem is learning an optimal policy and achieving with the minimum constrained violation regret. Section 4.2 describes how naive designs of A fail in this problem due to misaligned objectives of curriculum learning and constrained RL.
Problem statement. Given a CCMDP . to describe the parameterization of a set of constrained tasks and target context distribution yo to specify their probability of occurrence, generate sequence of context distributions {er } that allow an RL agent to sample-efficientl learn an optimal policy (1) with minimal constraint violation regret ( by taking the misalignment phenomenon into account.
Traditionally, a curriculum learning approach generates a sequence of context distributions {ei)er while a non-curriculum approach draws contexts directly from the target context distribution. Thus, curriculum learning approaches can choose a context distribution prioritizing contexts where the current policy has low expected cost V (x) minimize constraint violation regret.
CURRICULUM LEARNING AND CONSTRAINED RL
We present state-of-the-art curriculum learning method and discuss its limitations in CCRL.
4.1 CURRICULA VIA OPTIMAL. TRANSPORT
Curricula via Optimal Transport (CURROT, Klink et al. 2022 given a target context distribution yo creates sequence of context distributions {ek R k= to obtain optimal policy for contextual MDP M (S, A, X', M, 7) (Hallak et al. 2015). Compared to contextual constrained MDP, a contextual MDP M does not have cost function, as M(x) = (S, A, Px, Tx, Po.x; 7). An optimal policy x" in contextual MDP M only maximizes the expected return, i.e., n -== arg max, [V (x)).
At curriculum iteration [K]. CURROT draws contexts M from context distribution @k-1+
and collects trajectories D* (Tx, LM where Tx, = {( a Ti,,, St.c1)) |Tx Then, an RL
algorithm updates policy m:-1 via Dr. CURROT generates the next context distribution via
arg min Wz(e,)) v) s.t. ø(x) > 0 + v**(x) 2 Ç, VX E X, and W2(e.e. Se. (3)
g

---

Published as a conference paper at ICLR 2025
where (2(:.-) is the Wasserstein distance and 2+ is particle-baset based distribution based contexts with return (Tx) higher than performance threshold Ç, which a buffer successful contexts B_ keeps. A failure buffer B in parallel maintains the remaining contexts. CURROT estimates V based on B_ and 4 The constraint on V ensures support over contexts with sufficiently high rewards. The Wasserstein distance constraint avoids diverging from the current successful contexts. We investigate CURROT due to its favorable properties: 1) It poses curriculum generation as constrained optimization problem, enabling a natural extension to CCRL ( 2) Interpolating context distributions based on Wasserstein distance allows non- paraameetriic distributions. 3) The performance constraint enforces robustness. We refer the reader to Klink et (2022) for more details.
4.2 FAILURE OF CURRICULA TO ENSURE SAFETY
The state-of-the-arr curriculum learning methods, e.g., CURROT focus on the standard multi-task RL problem, i.e., maximizing E, [..V." (x)). They fail to address CCRL due to prioritizing contexts x ~ 2k where policy тę achieves high V (x) but violate the constraint on V (x). Imagine safety-maze with safety threshold D = 0 and the initial easy contexts are around the bottom left corner. As the agent improves, CURROT will generate a context distribution closer the target distribution. However, CURROT minimizes Wasserstein distance, it will move its contexts over the hazards. Although they set goals closer to the initial position, hence high V they can cause high V and constraint violations. The agent will choose to pass through the hazards or stay out. The former scenario leads to unsafe behavior with high costs, whereas the latter yields failed conservative behavior.
Fig. demonstrates currie- ula generated by CURROT and SCG in safety-maze. The marked points visu- alize the contexts, deter- mining the goal location and tolerance, drawn from context distributions at different curriculum itera- tions/epochs. The color of a point refers to the goal tol- erance. CURROT moves contexts from the bottom row toward the target context distribution. As CURROT ignores the cost, places the goals mostly in the red region early on, causing suboptimal behaviors. In comparison, SCG enables goals centered on the right column, which does not yield any cost or goals with high tolerance. Such scenarios overlook the constrained nature of safety-critical setting. Therefore, to prioritize safety, a curriculum learning algorithm should have its objective aligned with the constrained RL problem. However, by construction, existing approaches suffer from misaligned objectives in constrained RL.
Epoch 10
Epoch 30
Epoch 10
Epoch
3.25
2.25
Epoch 50
Epoch 150
Epoch 100
Epoch 450
-125
0.25
Figure 3: CURROT (left) and SCG's (right) curricula.
unique to CURROT. They occur under curriculum learning algorithms that
5 SAFE CURRICULUM GENERATION
We develop Safe Curriculum Generation (SCG), curriculum learning method that minimizes constraint violation regret and sample-efficiently learns an optimal policy I Algorithm Tis
pseudocode for SCG. At curriculum iteration k, SCG samples contexts {x; = from context distribution 2h-1 (Line 5), and collects trajectories Dj {Tx; Ri= JEE
with transitions including the cost (Line 6). Then, a constrained RI. algorithm updates policy u (Line 7). Next, based on Dr. the UPDATESUCCESSFUL CONTEXTSO function determines successful contexts (B4) according to SCG's three phases (Line 8): 1) prioritizing safcty, 2) prioritizing performance, and 3) safely approaching the target context distribution. Finally, SCG updates V and V based B and B_
(Line 9) and generates the next context distribution @ (Line 10) via
2 Sca (TT, 2+, D.C) arg min W2(
s.t.
e(x) > 0 V (x) 2 V X,
D X. Wa(e.e-)5G
(4)

---

Published as a conference paper at ICLR 2025
Algorithm 1 Safe Curriculum Generation (SCG)
Input: Target and initial context distributions yp and 2o
Parameters: Safety threshold D, cost threshold D, performance threshold Wasserstein distance bound 6, number of curriculum iterations K, number of rollouts per iteration M, buffer size N
Output: Final policy тк
1: Initialize policy Tо ()
2: B_ 4-
t- initialize buffers of size N D to search safe and performant contexts
3: ISSAFE, ISPERF 4- False, False
4: for k 1 K do
D sample contexts > collect rollouts via policy Tk-1 6: De = = (Bi.e. Bi,,, 81,441, Fit policy update via constrained RI. algorithm A
5: X; ~ ~ [M]]
7:
FA + A-(DE, TR-1, D)
B ISSAFF, [SPERF +- UPDATESUCCESSFFU UDDATTSUCCESSUUUUCOTTXXCC... CON.SPXSCCTTTTTTBB.,.... B_, ISSAFE, ISPERF, 'D.)
8:
9:
Update value functions V,"* and V." with B., and B_
10:
@k $500 (TT: 0+; D,C)
new context distribution
where, in contrast to CURROT, SCG imposes constraint on V to ensure that the support of the next distribution will be over low-cost contexts to minimize safety violations. Note that constrained RL algorithm A utilizes safety threshold D for the constraint on the expected cumulative cost { SCG uses cost threshold Đ for constraint on individual contexts under the support of @x. For the remainder of this section, we describe SCG's s three phases (see Appendix C for more details). SCG
does not sample contexts from 2+ but uses it source distribution to approach y (4)
1) Prioritizing safety. Early in training. an RL agent likely collects high costs or low rewards during exploration. In safety-critical setting, this period can rapidly increase constraint vio- lation regret until the agent discovers how to behave safely. Therefore, SCG initially proposes easy contexts where the agent can behave safely without much exploration. To achieve that, WWW.IESSUUCCSSUUUCCOTNE
labels safe contexts as successful. A context x is safe if the discounted cumulative cost Ge(Tx) ) is less than the median cost Ciedd of B4. SCG updates B4 with safe contexts and generates e+, Gaussian mixture model (GMM), via EINT E (B+) ().
EINIT (B-)
SAFE(BB)
Σ W:N((x(x; (AATE, X.EB_
(5)
where W; anwy + (1 an)w,, w/ x [Cmed Tool-Golle GG(TT,))++ W/ & [G,(Tx,) Rmea] +. and SAFF. = max Tmin: 2 Ge(Ts,)-
Can - J. Caax is the maximum cost until curriculum iteration i: and Rmed is the median reward in B4, A weight Wi of this Gaussian mixture model is a weighted average of of and w/ for context x; with ag E E 1]. The idea is tune whether a context with low cost should have higher priority over context with high reward. At the beginning of the search for safe contexts, SCG sets ag to and linearly anneals to lower pre-determined ratio unless that ratio
2) Prioritizing performance. Once Cmed is less than the cost threshold D. SCG focuses on per- formant contexts. A performant context has discounted cumulative reward G,,(Tx) greater than the median reward Ried of B4. In the first two phases, B+ and B_ get updated cyclically. SCG gener- ates 2+ to be a GMM centered in contexts from B Performance prioritizing 2+ differs from the previous one in terms of two factors: 1) SCG linearly anneals an to 0 unless it is 0. 2) The standard f (-G)((T;;) }. Inspired by CURROT, deviation parameter of this GMM is d'Erf, max F Ran
SCG generates 0+ in the first two phases as a GMM to allow for exploration in the context space.
3) Safely approaching the target context distribution. When Rmed exceeds Ç. SCG moves the final phase. Here, CEDDATTSSCCCSSUUUUCONTTTTTT
labels a context x as successful if the policy Tk-1 collects discounted cumulative reward greater than or equal to ( and a discounted cumulative cost less than or equal to D. Similar CURROT, to update full success buffer, SCG generates particle-base based context distribution e+ (x) | ΣxEEB 8g. (x+). where ō,, is a Dirac delta. Next, it replaces contexts in B1 with new ones from distribution that minimizes the Wasserstein distance W2(e+, .p). In contrast, B_ gets updated cyclically. SCG models 2+ as particle- particle-based distribution since exploration is not critical, and B, well represents where the agent is safe and performant.

---

Published as a conference paper at ICLR 2025
Remark:
There are three novel algorithmic features that differentiate SCG from CURROT: 1) The cost constraint in (4) accounts for safety in any context. 2) Safety-prioritizationn via generating
source distributions 2+ over safe contexts (5) avoids high constraint violation regret. 3) Annealing mechanism to tune the prioritization ratio of safe or performant contexts in (5) and (6) allows for a
smooth transition between safety and performance prioritization phases. Section 6.3 demonstrates
how these features makes SCG a well-balanced, safe curriculum learning approach.
6
EMPIRICAL ANALYSIS
Our experiments in constrained RL domains investigate the following questions:
6.1
Can SCG obtain optimal policies while reducing safety violations?
6.2
Does SCG prioritize safe and performant contexts during training?
6.3
What is the contribution of SCG's components to its overall performance?
6.4
How do the final policies perform in each context of the target distribution?
To assess the quantitative questions, we consider 3 metries: constraint violation regret to evaluate safety during training, 2) expected discounted cumulative cost, and 3) expected success to assess safety and performance after training, respectively. For the qualitative questions, we visualize the evolution of curricula across the curriculum iterations.
We consider 3 constrained RL domains:
safety-muze, safety-goal, and safety-push (Figures 2 and 5a In all domains, the agent aims to avoid hazards and reach a goal in the presence of misalignment phenomena. We study safety-maze to showcase that a simple modification to an existing domain (see Section 3.Ip can trigger misaligned objectives. In comparison, safety-goal and safety-pus are navigation tasks with realistic sensory observations in Bafety-Gymmasium (Ji et al. 2023). ) a framework extensively used for constrained RL. In safety-goal, the agent has to navigate high dimensional observations, namely, LIDAR outputs for the surrounding objects. Safety-push introduces (i) larger observation space and more complicated dynamies, as the agent must push a box to a goal given additional LIDAR outputs, (ii) a layout favoring multi-modal distributions that draw goals on the left and right corridors, and (iii) dead-end, trapping the agent and causing high costs. In all environments, the context determines the goal's location and tolerance. The target context distributions are uniform distributions over goals placed in the free space on the top row inside the walls/pillars. Appendix D.2 provides more details. We set the safety threshold D O in our environments of interest, which necessitates safe paths to goals under şo Nevertheless, SCG is not limited to such a constraint. The safety threshold in any other environment may be positive.
given
We compare SCG to statc-of-the-art curriculum learning methods: CURROT (Klink et al. 2022). SPDL (Klink et al. 202T), PLR (Jiang et al., (0211), GOALGAN (Florensa et al. 2018), and baselines provide perspective. To assess the impact of curricula, DEFAULT draws contexts from the target context distribution and, hence, does not generate curricula. To evaluate curricula that only consider the cost, CURROT4COST replaces
ALP-GMM (Portclas et al. 2020) (details in Appendix[B]
the performance constraint of CURROT with a cost constraint V (x) < D. To examine the need for an additional cost constraint in the curriculum update (4), AFIVESAFECURROT penalizes reward with cost and has an augmented performance constraint (V-T(x) (x) > Q. All approaches the same constrained RL algorithm, PPO-Lagrangian (Achiam & Amodei 2019), to update the agent's 's policy. Appendix F4 analyses other constrained RI. algorithms with a DRFAULT curriculum.
6.1 CAN SCG OBTAIN OPTIMAL POLICIES WHILE REDUCING SAFETY VIOLATIONS?
Figures 4b and 4c show final policies' average cost and success in target contexts. while Fig. illustrates the constraint violation regret (2) in training. Table [ summarizes the results showing in which environments the most promising methods satisfied the criteria: 1) whether yields an optimal policy, i.c., policy achieving zero cost and the highest success rates in target contexts in the median. If so, it assesses 2) safest training, i.c., lowest constraint violation regret, and 3) sample efficiency against DEFAULT in terms of number of interactions to reach zero cost or the highest success rate (Appendix F.I provides the corresponding learning curves).
SCG yields zero cost and the highest success rate in all environments. In detail, SCG and CURROT achieve the lowest costs, zero, in all domains, whereas the rest of the approaches achieve higher costs in multiple training runs in at least one domain. Note that the safety threshold / (T) is zero
7

---

Published as a conference paper at ICLR 2025
Table 1: Condensed analysis evaluates a method in safety-man maze (M), safety-goal (G), and safety- push (P) based on three criteria. We strike through domains where method violates a criterion, and *** indicates failure in a run(s). We check the second and third criteria only the first is satisfied.
Method
Optimal policy
Safest training
Sample-efficien DEFAULT
SCG
M
G
P
M
G
CURROT
M
G
P
AAIRESAFECURRO
P
CURROT4ÇOST
M
Comubatve
0.75-
0.50-
025-
0-
Comutaiie
0.5-
0.00-
150-
9.3
100
0.59
0.09
600
GRT DEP APP (UR
9000 C20T
N05 CIR R1S7 CITT
ATP
PER 5791 Roat
K5
SPOC.Gi GAN
CRT
GAM
RAN
CAT
40
(b) Final Figure 4: Safety-man (top), safety-gen goal (middle) and safety-push (bottom) from 10, 5, and seeds, respectively. a) Constraint violation regret in training. b-c) Average discounted cumulative cost and success of the final policies in target contexts, respectively. Box plots show the min, first quartile, median, third quartile, and max, from bottom to top. We use CRT for CURROT, NSCRT for NAIVESAFECURROT DEF for DEFAULT, ALP for ALP-GMM, and GGAN for GOALGAN.
CV regret in training
(c) Final
for all domains; hence any method yielding non-zero cost is considered unsafe. Although CURROT attains 100% success in safety-mazz fails to compete in other domains. NAIVES and CURROT4COST either cannot achieve the highest median success have high variance among training runs. In all environments, at least one curriculum learning method outperforms DEFAULT, no curriculum baseline, in terms of final cost and success, evidencing that automated curricula can boost final performance/safitty. However, ALP-GMM, PLR, SPDL, and GOALGAN fall behind.
In all domains, SCG reaches the lowest constraint violation regret among methods that achieve zero cost and highest success rates. Although some seem safer during training, they fail to learn op- timal policies. For example, in safety-maze (top row), WAIVESAFECURAL yield similar or lower constraint violation regret than SCG. but they have varying success rates. CURROT achieves zero cost and 100% success, yet it yields high constraint violation regret in safety-maze, as it suffers from misaligned objectives. One can make similar arguments for the other domains. On side note, DEFAULT either attains the lowest (safety-maze) or one of the highest constraint violation regrets while failing to learn optimal policies consistently. This may be because DEFAULT either causes the agent to be conservative yet suboptimal or to collect high costs.
and CURROT4COST
SCG consistently yields optimal policies. In safety-maze, CURROT and achieve the highest success rates in the median, yet they fail to learn optimal policies in other domains. Both sample-efficient compared to DEFAULT, but CURROT causes higher constraint violation regret so it is not the safest. CURROT4COST and the rest of the approaches learn suboptimal policies. Thus, do not evaluate them in terms of safety or sample efficiency.
6.2 DOES SCG PRIORITIZE SAFE AND PERFORMANT CONTEXTS DURING TRAINING?
Fig. and Fig. show CURROT's and SCG's curricula in different epochs. Overall, SCG prioritizes safe and performant contexts as it resolves the misalignment between curriculum learning and constrained RL.. In safety-maze, SCG initially prioritizes goals with high tolerance over the
8

---

Published as a conference paper at ICLR 2025
Epoch 5
Epoch 20
Epoch
Epoch 20
30000990
Epoch 50
Epoch 120
Epoch 50
Epoch 120
0.75
0.5
Epoch 5
Epoch 40
Epoch
Epoch 40
0.25
Epoch 75
Epoch 150
Epoch 75
Epech 150
788980980
(a) Environments
(b) CURROT
(c) SCG
Figure 5: SCG and EERRROT'curricuu in safety-goal (top) and safety-push (bottom). The first column shows environment snapshots. The second and third columns demonstrate the curricula generated by CURROT and SCG, respectively. In safety-goal and safety-push, we draw contexts from distributions ex at iterations/epochs & {5, 20, 50, 120} and k E {5, 40, 75, 150}, respectively.
100-
2.06
900
S00 Nofau
SCG NoPP
SCG SoPS
500s NoPPPS
0.00
0.69
Nosua
SUPP
NOPS
NUPPPS
Nolan
NCPP
NuPS
NePPPS
Final CV regret
(b) Final
(c) Final success
Figure 6: Ablation study in safety- push to investigate the impact of SCG's main components: Phase 1, which prioritizes safety (PS), Phase 2, which prioritizes performance (PP), and annealing (Ann).
bottom white row since they are easy. Eventually, SCG moves goals over the right white column with decreasing tolerances. The goals SCG places in red and black areas have high tolerance and can be reached outside both regions. CURROT moves contexts through the hazardous area, causing safety violations. Similarly, in safety-goal SCG prioritizes contexts the right side, as they allow the agent to learn how to reach the target contexts by avoiding the hazards. In contrast, safety-pust map favors multi-modal distributions with the hazards in the middle. SCG recognizes that and prioritizes contexts on the left and right corridors. However, CURROT moves contexts through the hazards, causing high constraint violation regret (see Fig. 4ap. See Appendix F.5 for the progression of curricula generated by the rest of the approaches.
6.3 WHAT THE CONTRIBUTION SCG's COMPONENTS TO ITS OVERALL PERFORMANCE?
Fig. demonstrates the results in safety-push obtained by four variations of SCG: 1) SCG-NOANN does not do annealing, 2) SCG-NoPP prioritizes safety first and then moves on to Phase 3 by skipping performance prioritization, 3) SCG-NoPS begins training Phase 2, hence does not prioritize safety initially, and lastly 4) SCG-NoPPPS starts the training at Phase 3, hence does not prioritize safety performance individually.
SCG and SCG-NoPP yield the lowest constraint violation regret during training (in the median, Fig. (a)) whereas SCG-NoPS and SCG-NoANN closely follow, and SCG-NoPPPS yields

---

Published as a conference paper at ICLR 2025
Cost
1.0
0.8
0.6
Success
SucTesS
0.4
0.2
0.0
(a) Safety-maze
(b) Safety-goal
(c) Safety-push
Figure 7: In-distribution generalization capability of SCG. A marker's color and position demonstrate the median cost/success and the goal location. We set the tolerance to the minimum value. We divide the supports of target context distributions into grids and pick the goal locations in their centers.
the highest CV regret. In terms of final performance, all except SCG-NoPPPS reach the highest success rate in median (see Fig. Fcg. Although all achieve zero cost in target contexts in the median, SCG-NoPS yields unsafe policies in multiple runs at test-time (see Fig. 6b). Also, SCG is the most sample efficient, followed by SCG-NoPP (see Appendix F.2). Overall, PS provides lower constraint violation regret, yet the learning progress slows down when not combined with PP and annealing, In
addition, Appendix F3 provides an ablation study several SCG parameters.
6.4 How DO THE FINAL POLICIES PERFORM IN EACH CONTEXT UNDER THE TARGET DISTRIBUTION?
The CCRL problem ① averages the expected return and cost across the target contexts according to the target distribution (p). Thus, we evaluate the policies learned in the individual contexts in terms of success and cost (Fig. 7). We set the goal tolerance of these contexts to the minimum value to escalate difficulty. The results evidence that SCG yields policies that act safely, that is, they receive zero cost, and succeed in all target contexts except for some goals placed just above the hazardous area in safety-goal. In summary, although we consider an average across the contexts, the policies learned still safe and performant in individual contexts.
CONCLUSION
In this work, we study safe automated curriculum generation multi-tast cost-constrained settings with distributions target tasks. We propose safe curriculum generation approach (SCG) developed for constrained RI to minimize constraint violation regret and accelerate learning. SCG initially prioritizes tasks with low costs over high-reward ones, ensure that the agent learns a policy that satisfies the cost constraint. Next, SCG proposes tasks where the agent can collect high rewards. Finally, SCG takes safety and performance jointly into account. Our empirical evaluation evidences that ttate-of-the the--hhrrtt art curriculum learning approaches fail to learn optimal behavior in a safe and stable way, these approaches suffer from misaligned objectives with constrained RL. In contrast, SCG obtains optimal behavior with the lowest constraint violation regret in domains with low or high dimensional state spaces and complicated dynamics and task structures.
L Limitations and future work. While SCG can reduce the constraint violation regret while preserving the benefits of curriculum learning in boosting learning speed, it does not provide any guarantees for the constraint violation regret that is achieved at the end of the training. Combining SCG with constrained RL algorithms that guarantee safety could mitigate this issue while keeping a good learning speed. Furthermore, SCG inherits dependency on the initial context distribution from automated curriculum generation. Thus, if the support the initial context distribution unsafe, might take multiple iterations find safe contexts.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
For all the hyperparameters and detailed settings of the experiments, please refer to Appendix D We put the core code of SCG in the supplementary details. The code includes instructions install necessary software, reproduce the experiments and target contexts where we evaluate trained policies obtain the experimental results.
ACKNOWLEDGMENTS
This work is supported by the Office of Naval Research (ONR) under grant numbers N00014-24- 2797 and N00014-22-1-2254, and the European Research Council (ERC) under the starting grant 101077178 (DEUCH).
REFERENCES
Joshua Achiam and Dario Amodei. Benchmarking safe exploration deep reinforcement learning. 2019. URL https: /tttps:/cdn. openai. com/ safexp-
wwwps//wwpcoocapppoaaa/aaaaea.ee wwwaaap--orrttpreeeeeeeee short.pdf
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In ICMI., pp. 22-31. PMIR, 2017.
Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topeu. Safe reinforcement learning via shielding. AAAI, volume 32, 2018.
Eitan Altman. Constrained Markov decision processes. Routledge, 1999.
Adrien Baranes and Pierre-Yves Oudeyer. Intrinsically motivated goal exploration for active motor learning in robots: A case study. In IROS, pp. 1766-1773, 2010.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. NeurlPS, 30, 2017.
Jiayu Chen, Yuanxin Zhang, Yuanfan Xu, Huimin Ma, Huazhong Yang. Jiaming Song, Yu Wang, and Yi Wu. Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems. In NeurlPS, pp. 9681-9693, 2021.
Yinlam Chow, Ofir Nachum, Edgar Shenez-Guzman, and Mohammad Ghavamzadeh. A Lyapunov- based approach safe reinforcement learning. NeurIPS, 31, 2018.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. NeurlPS,
2020.
Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exxploiaatoo in constrained MDPs. arXiv preprint arXi:2003.002189, 2020.
Theresa Eimer, André Biedenkapp. Frank Hutter, and Marius Lindauer. Self Belf-paced context evaluation for contextual reinforcement learning. In ICML, pp. 2948-2953 2958, 2021.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. In ICLR, 2018.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In CoRL, pp. 482-495. PMLR, 2017.
Carlos Florensa, David Held, Xinyang Geng, and Picter Abbeel. Automatic goal generation for reinforcement learning agents. In ICML, pp. 1514-1523, PMLR, 2018.
Jonas Günster, Puze Liu, Jan Peters, and Davide Tateo. Handling long-term safety and uncertainty in safe reinforcement learning. In CoRL, 2024.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual Markov decision processes. arXiv preprint arXiv: 00..//02.2222222 02259, 2015.
11

---

Published as a conference paper at ICLR 2025
Yannick Hogewind, Thiago D Simão, Tal Kachman, and Nils Jansen. Safe reinforcement learning from pixels using a stochastic latent representation. In ICLR, 2022.
Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum reinforce- ment learning using optimal transport via gradual domain adaptation. NeurIPS, 35:10656-10670, 2022.
Nils Jansen, Bettina Könighofer, Sebastian Junges, Alex Serban, and Roderick Bloem. Safe rein- forcement learning using probabilistic shields. In CONCUR, 2020.
Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In NeurIPS, pp. 18964-18993. Curran Associates, Inc., 2023.
Jiaming Ji, Jiayi Zhou, Borong Zhang. Juntao Dai, Xuchai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. OmniSafe: infrastructure for accelerating safe reinforcement learning research. JMLR, 25(285):1-6, 2024.
Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G. Hauptmann, Self-paced curriculum learning. AAAI, pp. 2694-2700. AAAI Press, 2015.
Minqi Jiang, Michael Dennis, Jack Parker- Holker, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. NeurIPS, pp. 1884-1897, 2021a.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In ICML, pp. 4940-4950. PMLR, 2021
Sebastian Junges, Nils Jansen, Christian Dehnert, Ufuk Topcu, and Joost-Pieter Katoen. Safety- constrained reinforcement learning for MDPs. TACAS, pp. 130-146. Springer, 2016.
Danial Kamran, Thiago D. Simāo, Qisong Yang, Canmanie T. Ponnambalam, Johannes Fischer, Matthijs T. Spaan, and Martin Lauer. modern perspective on safe automated driving for different traffic dynamics using constrained reinforcement learning. In ITSC, pp. 4017-4023. IEEE, 2022.
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Danicle Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In ICRA, pp. 8248 8258-8254. IEEE, 2019.
Pascal Klink, Hany Abdulsamad, Boris Belousov, and Jan Peters. Self-paced contextual reinforcemnt learning. CoRL, pp. 513-529. PMLR, 2020a.
Pascal Klink, Carlo D' Eramo, Jan Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. In NeurlPS, pp. 9216-9227. Curran Associates, Inc., 2020b.
Pascal Klink, Hany Abdulsamad, Boris Belousov, Carlo D' D'Eramo, Jan Peters, and Joni Pajarinen. A probabilistic interpretation of self-paced learning with applications to reinforcement learning. JMLR, 22:182:1-121 12:18222182::2 2021.
Pascal Klink, Haoyi Yang. Carlo D' D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforcement learning via constrained optimal transport. In ICML, pp. 11341-11358 PMLR, 2022.
Cevahir Koprulu and Ufuk Topcu. Reward-machine--uue UAI, volume 216, pp. 1121-1131. PMLR, 2023.
self- reinforcement learning. In
Cevahir Koprulu, Thiago D. Simão, Nils Jansen, and Ufuk Topcu. Risk-aware curriculum generation for heavy-tailed task distributions. In UAI, volume 216, pp. 1132-1142. PMLR, 2023.
M. Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In NIPS, pp. 1189-1197. Curran Associates, Inc., 2010
Haohong Lin, Wenhao Ding, Zuxin Liu, Yaru Niu, Jiacheng Zhu, Yuming Niu, and Ding Zhao. Safety-aware causal representation for trustworthy offline reinforcement learning autonomous driving. IEEE Robotics Autom. Lett. 95))4639 2024.
12

---

Published as a conference paper at ICLR 2025
Adrian Müller, Pragnya Alatur, Volkan Cevher, Giorgia Ramponi, and Niao He. Truly no-regret learning in constrained MDPs. ICML, pp. 16605-36653, 2024.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey, JMLR, 21: 181:1-181::00 2020.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yves Oudeyer. Teacher algorithms for curriculum learning of deep in continuously parameterized environments. CoRL, pp. 353.
PMLR, 2020.
Sebastien Racaniere, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated curricula through setter-solver interactions. In ICLR, 2020.
Zhipeng Ren, Daoyi Dong, Huaxiong Li, and Chunlin Chen. Self-paced prioritized curriculum learning with coverage penalty in deep reinforcement learning. IEEE TNNLS, pp. 2216-2226, 2018.
Julien Roy. Roger Girgis, Joshua Romoff, Pierre-Luc Bacon, and Christopher J. Pal. Direct behavior specification via constrained reinforcement learning. In ICML, pp. 18828-18843. PMLR, 2022.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In ICLR, 2016.
Thiago D. Simão, Nils Jansen, and Matthijs T. J. Spaan. AlwaysSafe: Reinforcement learning without safety constraint violations during training. In AAMAS, pp. 1226-1235, 2021.
Yeeho Song and Jeff Schneider. Robust reinforcement learning via genetic curriculum. In ICRA, pp- 5560-5566. IEEE, 2022.
Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause. Safe exploration for optimization with Gaussian processes. In ICML, pp. 997-1005. PMLR, 2015.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In ICLR, 2019.
Tristan Tomilin, Meng Fang, and Mykola Pechenizkiy. HASARD: A benchmark for harnessing safe reinforcement learning with Doom. ICLR, 2025.
Mattco Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration for interactive machine learning. NewrlPS, 32, 2019.
Matteo Turchetta, Andrey Kolohov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe rein- forcement learning via curriculum induction. In NewrIPS, pp. 12151-12162, 2020.
Georgios Tzannetos, Bárbara Gomes Ribeiro, Parameswaran Kamalaruban, and Adish Singla. Proxi- mal curriculum for reinforcement learning agents. TMLRR 2023. ISSN 2835-8856.
Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained Markov decision processes. In ICML, pp. 9797-9806. PMLR, 2020.
Haozhe Wang, Chao Du, Panyan Fang, Shuo Yuan, Xuming He, Liang Wang, and Bo Zheng. ROI- constrained bidding via curriculum-guided bayesian reinforcement learning. In SIGKDD, pp. 4021-4031, 2022.
Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter Ramadge. Projection-basee constrained policy optimization. In ICLR, 2020.
Yihang Yao, Zuxin Liu, Zhepeng Cen, Jiacheng Zhu, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constraint-conddtionee policy optimization for versatile safe reinforcement learning. In NeurIPS, pp. 12555-12566 Curran Associates, Inc., 2023.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. In NewrIPS, pp. 7648-7659, Curran Associates, Inc., 2020.
13

---

Published as a conference paper at ICLR 2025
A NOMENCLATURE
Context
M
Contextual constrained MDP
S, A.
State, action, and context spaces
M
Mapping from contexts to constrained MDPs State, action, reward, cost
S, a. T, C
Px. Txx
Transition, reward and cost functions
Po,x
Initial state distribution Policy
Discount factor
D
Safety threshold
V,, Ve
Value function for reward and cost
0
Context distribution and target context distribution
((T,5))
CCRL objective function
Tx
Trajectory in context
G,,, Ge
Discounted cumulative reward and cost a trajectory
Reg"
Constraint violation regret during training
A
CCRL algorithm
Number episodes during CCRL training
T,
A CCRL episode and its length
D,
Cost, performance and divergence thresholds in SCG
D
Trajectory set
SCG function
GMM weight
Sapee, PEREF
GMM's standard deviations for safe/performant context x;
Omin
Minimum standard deviation in GMM
w", w
GMM weights for discounted cumulative reward and cost
GMM weight ratio
Cmed: Crux
Median and maximum cost in buffer
Rmeds Rmin
Median and minimum reward in buffer
EINIT IINT EMAIN
Phase 1, 2 and 3 update functions in SCG
B4, B_, TTEMP N
Success, failure and temporary success buffers and its size Distribution over successful contexts
0+
UNDATESICEESFFIL.COWEEYYS
Update function for successful contexts
K
Curriculum iteration and total number of iterations
M
Number of rollouts per curriculum iteration
14

---

Published as a conference paper at ICLR 2025
B AUTOMATED CURRICULUM GENERATION ALGORITHMS
In this section, we provide short descriptions of the state-of-the-art curriculum learning methods evaluated in the experiments.
CURROT (Klink et al. ) We propose SCG based on Curriculum RL via Constrained
Optimal Transport, which we describe and discuss in Sections 4.1 and 4.2
SPDL (Klink et al., 2021): Self-paced Deep Reinforcement Learning formulates the auto- mated curriculum generation problem similarly to CURROT, except that SPDL generates context distributions that minimize the KI. divergence to the target context distribution. The constraints in the optimization problem solved in SPDL are minimum expected discounted cumulative reward and maximum KI. divergence to the previous context distribu- tion. SPDL does not include an initial search procedure and generates context distributions as Gaussian distributions. SPDL generates parametric context distributions and does not provide the robustness that CURROT achieves via a performance constraint on individual constraints. Therefore, we can highlight why CURROT is better for building SCG.
PI.R (Tiang et al.] 2021b): Prioritized Level Replay a curriculum learning method de- veloped for procedural context generation environments, where level corresponds to task, i.c., an enviromment instance. PLR prioritizes levels that have a high average mag- nitude of generalized advantage estimate (Schulman al. 2016), namely, the discounted sum of cemporal--diferenee errors. PLR is a popular curriculum learning approach, yet it is unsuitable for continuous context spaces, it assumes that contexts are discrete and unidimensional. In addition, PLR assumes uniform target context distributions.
GOALGAN FFlorensa et al. 2018): Goal Generative Adversarial Network is a curriculum learning approach developed for goal-conditioned RL. GOALGANN trains goal discrimina- tor classify goals that are the intermediate difficulty for the policy of the RL agent and goal generator to generate goals at that difficulty to boost learning performance. GOALGAN is among the first automated curriculum generation approaches that pose curriculum as a sequence of distributions. Yet, GOALGAN does not interpolate in CURROT and SPDL. Similar to PLR, GOALGAN assumes uniform target context distributions.
ALP-GMM (Portelas al. 2020): Absolute Learning Progress with Gaussian Mixture Models uses the absolute learning progress of task to measure whether a task would improve the learning process of an RL agent. ALP-GMM learns a Gaussian mixture model over the absolute learning progress where a multi-armed bandit samples Gaussian as an arm based on its utility, which is the absolute learning progress. The Gaussian distribution that the arm corresponds to draws a task, namely, the context in our setting. ALP-GMM also generates a sequence of context distributions, but similar to GOALGAN, does not interpolate them. In addition, it also assumes that the target context distribution is uniform.
C DETAILS OF SCG
To support Section here we provide a closer look into how the UUDATESSUCEESFPUUCCONEEES function in SCG works (See Algorithm 2 for pseudocode).
Prioritizing safety, Initially, SCG sets FOUNDSAFEXS and FOUNDPERFXS to false to enable the WWWFEESSUCCSSFUU.CCNEEE
function to search for safe contexts first. Lines 2-3 indicate that a successful context this phase yields a discounted cumulative cost less than or equal the median cost Cmedt in success buffer B., Cyclically, B+ gets updated with the successful contexts in trajectory set DR. as B_ gets updated with the rest the contexts. Then, UPDATESUCCESSEE SFF..LLOTTETTOOO generates e+ Gaussian mixture model using B+ (Line 4) (see (5) in Section 5p. SCG searches for such safe contexts until Candd is less than equal cost threshold D (Line 5).
2) Prioritizing performance. Once the contexts in Dr satisfy the safety condition, CEOATTESUCESFUU.OOTEETSS switches its focus to finding performant contexts. Similarly, SCG begins by updating B, with contexts where the discounted cumulative reward is greater than or equal to the performance threshold (Lines 7-8). Then, SCG uses =pai (B+) generate e+, which
15

---

Published as a conference paper at ICLR 2025
Algorithm 2 UTDATTESUCCESSFUUCONNEEE
Input: B+, B_, ISSAFE, ISPERF, 2₁
Parameters: Cost threshold D, performance threshold
Output: B- 2+ DR, ISSAFE, ISPERF
1: if not FOUSDSAFEXS then
2: Add {x [o Txx,
> Cmed)] to B_
3: Add {x; Ge (xx
S Ceed to B+
4: @+ INNI (B+)
prioritize safety
5: FOUNDSAFEXS
Caot D
6: else if not FOUNDPERFXS then
7: Add {x; Gr (Tx
Rmed] to B_
8: Add (x; Gr Txx 9: @+ PERK (B+)
2 Reed to B+
b prioritize performance
FOUNDPERFXS Rmot 2 (
11: else
Add {x, (Txa)) < Rmed or Ge(Tx.) ) Cmod] to B_
13:
BTTEMP - Gr(x) Rmed and Ge(Tx.) S mood]
14:
B+, 0+ t- +EMAIN EMAIN (B (p)
b main phase
15:
return 14,B... 2+, ISSAFE, ISPERF, D/
differs from E (B+) in terms of GMM weights w" and standard deviation OPERE, (Line 9). -AAFE
EINI (B+)
Σ
waN (xxxxxx,, o SAmR, I), where wi axelo (1 oxk)u%
(6)
E
wi xx max (0, Cied - Ge(Tx;)) w/ x max G. GTx;)) Rmed), and
OVERF = max min 25 (- Rmin
Gr(Tx.))
Note that Rmio is the minimum reward until curriculum iteration k. SCG prioritizes performant contexts until Rmod greater than or equal (Line 10). During the initial search, SCG updates B., and B in a eyclic fashion.
3) Safely approaching the target context distribution. Section 5 already provides information about how the last phase of SCG works. This phase operates similarly to the main phase of CURROT. For detailed description, we refer the reader to Klink al. (2022).
D EXPERIMENTAL DETAILS
We discuss the process hyperparameter selection for the curriculum learning approaches evaluated in this work and additional details about the constrained RL environments the experiments.
D.I ALGORITHM HYPERPARAMETERS
SCG has five main parameters: performance threshold Ç. cost threshold D, Wasserstein distance threshold e, number of curriculum iterations K and number of rollouts per curriculum updates M. CURROT and NAIVESAFECURROT share the same parameters except the cost threshold D, whereas CURROT4COST shares all except the performance threshold Ç. We chose Ç to be approximately the midpoint between the minimum and maximum possible discounted cumulative reward or success rate. To select the Wasserstein distance threshold c, we ran grid search over (0.25, 0.5} for safety-goal, and over [1.0, 1.25) for safety-maze. For the number of rollouts per curriculum updates M, we ran grid searches over {20, 403 for all settings, Although SPDL shares Ç, K, and M, it has KL divergence threshold éKL. for which we grid search over {0.25, 0.5) in all environments. Table 2 provides all parameter values. The initial search procedure in SCG has three hyperparameters: ratio ag and minimum standard deviation Fmia for the Gaussian mixture model, and lastly the number steps to anneal ag. In all settings, we examined linearly annealing Og from I to 0.75 for searching safe contexts and 0.75 1.0 for performant contexts in 10 or 20
16

---

Published as a conference paper at ICLR 2025
Table Parameters used for SCG, CURROT, NAIVESAFECURROT, and CURROT4COST.
Environment
S
D
6
K
M
Safety-mazz
0.6
0.25
0.25
1.25
500
40
Safety-good
0.6
1
0.25
0.5
150
20
Safety-pace passty-
0.6
1
0.25
0.5
200
20
Safety-push
0.6
1
0.25
0.25
300
40
Table 3: Selected values for parameters of PLR, GOALGAN and ALP-GMM
Environment p
Qnoise
GG 7 Pollout
Psnceess
Prand
n Pollont
AG
Safety-maze 0,45
0.15
100
0.1
200
0.2
0.2
200
500
Safety-goal 0,45
0.15
100
0.1
200
0.2
0.2
200
500
Safety-passage
0.45
0.15
100
0.1
200
0.2
0.2
200
500
Safety-push 0.45
0.15
100
0.1
200
0.2
0.2
200
500
curriculum iterations. We did not anneal ag for safety-maze and safety-goal, but annealed ag in 10 iterations for safety-push and pastty-. We fix min 0.001 as in CURROT.
As parameters to tune, PLR has the score temperature 3, the staleness coefficient p. and the replay probability p. We ran a grid search over (p. 3. E {0.15, 0.45) x {0.15, 0.45] x {0.55, 0.85). GoALGAN has three parameters: the number of rollouts between curriculum updates "yollour the random noise on drawn contexts Onoise, and the percentage of contexts to draw from the success buffer Psoceess We ran a grid search over (Enoise. P'ollout Psuccess) E {0.05, 0.1} X {100, 2003 {0.1, 0.2). ALP-GMM has three parameters: the buffer size Sbuffer. the number of rollouts between curriculum updates n Proliouts and the probability of randomly sampling contexts Prand- We ran a grid search over AG
.G
(Prand, rollout Sbuffer) {0.1, 0.2) x (50, 100} x (500, 1000}. Table 3 shows the final parameter used for PLR, GOALGAN, and ALP-GMM.
D.2 ENVIRONMENT DESCRIPTIONS
Safety-mane environment. Inspired by the maze environment (Klink et al. 2022) we design safety- maze, where the agent receives rewards from of 1 until it reaches the goal, and costs of 0.25 when enters the hazardous area (Figure 2). The context space & [-9, 9) x 1-999 9) [0.25, 5.0] is over and y positions and tolerances of the goal, respectively. We train a constrained RI agent using the PPO-Lagrangian algorithm Achiam & Amodei (2019). The implementation we integrate into our codebase is from OmniSafe Ji et al. (2024). The parameters the PPO-Lagrangian are fixed to their default values in OmniSafe, except the number of steps to update the policy is 4000 and the number of iterations to update the policy is 12.
Safety- gootl environment. To evaluate SCG in more complex dynamics, create environment with a high- sigh-dimensional state space in columns, and hazards, blue circles, objects with which the agent, car, interacts in the environment (see Figure(5a) The rewards and costs come from the -afett-gyymmaiiu implementation. The context space X Similar to safety-maze, we only change the parameters of PPO-Lagrangian in OmniSafe by setting the number of steps and iterations to update the policy 10000 and 15, respectively.
Di et al. (2023). We use pillars, purple
1.5, 1.5] [-1,,, 1.5] x [0.25, 0.75) is goal positions and tolerances, respectively.
Safety-passape environment. Similar to safety-goal, we create an en- vironment with a high-dimensional state space in Safety-Gymnasium Ji et al. (202)). We use pillars, purple columns, and hazards, blue circles, as objects with which the agent, car, interacts in the environment (see Figure 8). The rewards and costs come from the aafety-gymnasium imple- mentation. The context space = [-1.5, 1.5) x [15,,5, 1.5] x (0.25, 0.75] goal positions and tolerances. respectively. Similar to safety-maze, we change the parameters the Lpo--agrangian implementation niSafe by setting the number of steps and iterations to update the policy to 10000 and 15, respectively. In addition, the architecture of a policy is two-layered perception with 256 neurons in each.
Om-
Figure 8: Safety-passage.
17

---

Published as a conference paper at ICLR 2025
Safety-push environment. In addition to the objects in safety-gor goaal safety-push includes box for the agent to push to goal while avoiding hazards (see Figure 5ap. The rewards and costs come from the aaffey-gyymasiinn of the PPO-L IPO-Laagrangian inplementation in OmniSafe by setting the number of steps and iterations update the policy to 10000 and 15, respectively, and the batch size to 256. The context space X' = [-1.5, 1.5] x [-1.5, 1.5] x [0.25, 0. 0.75] is goal positions and tolerances, respectively. As in safety-pasee palsay- the architecture of policy is two-layered perception with 256 neurons in each.
implementation. Similar to safety-maze,
only change the parameters
E COMPUTATION RESOURCES
We run our experiments on cluster with NVIDIA RTX A5000 GPUs and an Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz. We utilize Omnisafe Ji al. (2024) as our RL framework, which uses 16 torch threads and no parallel environments in our experiments. A training rum in the experimented environments approximately takes the following number hours: (i) safety-maze: 5.5 hours (6000 gradient steps in 2 million interactions), (ii) safety-goal: : 7 hours (2250 gradient steps in 1.5 million interactions), (iii) paaseae- hours (3000 gradient steps in 2 million interactions), and (iv) sasety-push: 12 hours (4500 gradient steps in 3 million interactions).
F DETAILED ANALYSIS OF RESULTS
F.I QUANTITATIVE RESULTS
In this section, we present figures that complement Fig. which demonstrate the constraint violation regret at the end the training and the average cost and success of final policies in target contexts.
F.1.1 SAPETY-MAZE
Figure demonstrates additional plots that provide detailed information about safety and per- formance during and after training. We observe that SCG, CURROT, NAIVESAFECURRE CURROT4COST, and DEFAULT achieve optimal behavior at least one run out of 10. However, SCG and CURROT consistently get optimal policies with converged constraint violation regret during training and with respect to the target context distribution. We highlight that, as SCG paces the curriculum according to how safely the agent behaves, its constraint violation regret in o con- verges the last. Nevertheless, it achieves the lowest constraint violation regret in training out of all approaches that more GOALGAN achieve similar success rates throughout the training and they all fail in target contexts. However, the constraint violation regret of ALP-GMM and PLR in training increases very rapidly, with GOALGAN following behind. In contrast, the rest of the approaches have converged constraint violation regret in training.
reliably learn optimal policy. ALP-GMM, PLR, SPDL., and
F.1.2 SAFETY-GOAL
The results in Figure 10 demonstrate that SCG generates curricula that achieve the highest success rates and the lowest costs during training time and when deployment after. CURROT4COST and DEFAULT can achieve similar success rates, but not as reliably, in target contexts. DEFAULT has its best performance out of all three constrained environments we study because safety-goal has dense reward function, which eases learning without a curriculum. NAIVESAFECURROY and CURROT4COST also yield as low constraint violation regret as SCG has at the final iteration of the training. However, AAIVESAFECURROT is less stable at learning policies that accomplish the task both during and after training. CURROT, ALP-GMM, SPDL, and GOALGAN follow NAIVESAFECURROT in terms of success in target contexts. An important interpretation to make in all three settings is that, in contrast learning directly in target contexts, curriculum learning approach can cause an agent to behave unsafely if the approach does not consider the constrained nature of the task.
18

---

Published as a conference paper at ICLR 2025
Consratns Valattin Negget
SP31.
OGAN
CRT
CR240
ALP
SPEML
ORT
CAT4C
ALP
SPOL
Qunilaato Cast
1.00
36-
0.75
-20
0.0
10
0.00
0.25
0.50 1,00
1,55
1.75
Nanker of cuntrvomennt antevctioon x10r
Number of eavivvoment interoctions
xio*
Namber of oonsioonen interactions x12
(a) Constraint violation regret
(b) Cost target contexts
(c) Success in target contexts
CRT
CR340
ALP
SPEL
CATEE
ALP
SPDL
Oumulartre Cosk
1.00
1.0
0.75
0.50
50
0.00
0.25
0.50
1,00
1.25
1.75
CRT
40
GAN
Number eevvivvmmnnt Interonions
x10*
Namber of oonsimmnent tnteractions XI2*
(d) Final regret in 4p
(e) Cost training
(f) Success in training
00AN
Camolitee Cust
1.00
Conatruni: Valktan
00
40-
20-
022
8.0
0.26
0.50
0.75
1.00
125
1.50
SPEC
0.00
$60
N5
SPOL Goal
Number of aasttromeent Internations X104
aC
GAN
CRT
46
BAN
(g) CV regret y
(h) Final cost training
Final success in training
Figure 9: Safety-man results in 10 seeds: Evolution of constraint violation regret during. b) Progression of expected discounted cumulative cost target contexts. c) Progression of expected success rate in contexts drawn from the target context distribution. d) Constraint violation regret with respect to the target context distribution at the final curriculum iteration. e) Progression of expected discounted cumulative cost in contexts sampled during training. f) Progression of expected success rate in contexts sampled during training. g) Evolution of constraint violation regret with respect to the target context distribution. h) Expected discounted cumulative cost of the final policies in contexts sampled during training. i) Expected success rate of the final policies in contexts sampled during training.
F.1.3 SAFETY-PASSAGE
Similar to safety-goat the results in Figure I1 demonstrate that SCG generates curricula that achieve the highest success rates and the lowest costs during training time and when deployment after. In target contexts, NAIVESAFECURRO and DEFAULT can achieve similar success rates but less reliably. CURROT and CURROT4COST can get close but still fail some runs. ALP-GMM,, SPDL, PLR, and GOALGAN fall behind in safety and performance by a big margin.
F.1.4 SAFETY-PUSH
Safety-push results in Figure 12 support our observations the rest the environments. SCG successfully aligns the objectives of constrained RL and curriculum learning, thus yielding the lowest constraint violation regret while achieving the lowest cost and highest success target contexts. Although DEFAULT, CURROT and its naively modified safe versions can get close to SCG in terms of final success, they are not as safe during training and do not consistently yield safe agents at the end. As in the other environments, PLR, SPDL, and GOALGAN fall behind in terms of safety and performance, whereas ALP-GMM seems to be as performant as CURROT, but not as safe.
F.2 ABLATION STUDY SCG COMPONENTS
Fig. 13 provide progression plots for average success, and return in contexts generated during training and sampled from the target context distribution. As mentioned in the main document, SCG the first reach 100% success in target contexts, where SCG-NoPP closely follows. Although
19

---

Published as a conference paper at ICLR 2025
Conataun Valaton Hlgens
CRTRC
AL7
SPD2
LEF
I.R
QGAN
800
NOCRT
usF
R.
O2AN
Ounitatae Coet.
10.0
1.00
40
9.79
R0
9.50
0.23
0.0
04
Namber of sovinommenn Interactions x10
Number internation x1
Number internations x1
(a) Constraint violation regret
(b) Cost target contexts
(c) Success in target contexts
SPOL
OGAN
Cansdaaae Cod
9.50
20
0.23
3.00
PER SPAL
0.0
GAN
x10
(d) Final CV regret in
(c) Cost training
(f) Success in training
OCAN
CHURL
AL
SPOL
Ouma wumalativ Ouss
1.00
Constoue Netdto
20
10
0.25
8.0
0.00
Namber
Indoractions x20
40
GAN
GAN
(g) CV regret in
(h) Final cost training
(i) Final success in training
Figure 10: Safety-goal results from runs in seeds: Evolution of constraint violation regret during. b) Progression of expected discounted cumulative cost in contexts drawn from the target context distribution. c) Progression of expected success rate in contexts drawn from the target context distribution. d) Constraint violation regret with respect to the target context distribution at the final curriculum iteration. e) Progression of expected discounted cumulative cost contexts sampled during training. f) Progression of expected success rate in contexts sampled during training. g) Evolution constraint violation regret with respect to the target context distribution. h) Expected discounted cumulative cost of the final policies in contexts sampled during training. i) Expected success rate of the final policies in contexts sampled during training.
SCG-NOANN and SCG-NoPS do not fall behind in terms of pace of learning, observe that SCG-NOPPPS the slowest among all. addition, SCG-NoPPPS yields high constraint violation regret. The last four subfigures visualize the progression of curricula generated by SCG variations.
F.3 ABLATION STUDY FOR SCG PARAMETERS
We run ablation study for three parameters of SCG: 1) target GMM weight ratio i.e., the value SCG anneals ag to, 2) number of annealing iterations Kann, and 3) cost threshold D in the evaluate how SCG performs in safety-push, more specifically, how optimality at test time and safety during training change with respect to these parameters.
SCG update. Our objective
F.3.1 TARGET GMM WEIGHT RATIO &
Fig. 14 demonstrates the impact of target GMM weight ratio a. We experiment with a E (0.75, 0.5, 0.875, 1). which determines to what degree SCG prioritizes safe or performant con- texts as the annealing progresses. distributions e+ over safe contexts only in anfety-prioritization phase (5) and over performant con- texts only in performance-pricn priiunnicepioiiiizationn phase (6). In comparison,
1 corresponds to no annealing, i.c., SCG constructs source
0.5 anneals the ratio to give phase begins with safe phase
equal importance to safety and performance. Hence, the aafett-prioritizatte contexts only and then anneals O; to give equal importance. The irfffrmannce-rioritiinn operates in reverse, first giving equal importance to both and then annealing it for performant contexts 0.75 is what report in the main document. Although
only. Note that lower CV regret than a = 0.75 in multiple runs, a = 0.75 achieves zero-cost and 100% success in target contexts, also yields CV regret with low variance. 0.5 yields the highest CV regret.
{0.875, 1) provides
20

---

Published as a conference paper at ICLR 2025
Consinut Valttte Regies
CRT
CRTTO
ALP
5001.
N065
DEF
CUR
GuaNANN
SPOL
60
Ortobato Deet
10.0
1.00
9.75
40
0.50
26
0:23
800
0.0
2:5
Number of cnsirommrt Interactions X10*
Namber ooatoomeen interotioos x2pe
Nunter of Woaatouzeen introotocs xion
(a) Constraint violation regret
(b) Cost target contexts
(c) Success in larget contexts
SPOL.
GeaIGAN
GOaIGAN
Connstiteee Gost
DV
2.73
2.50
9.25
0.0
CRT
4C
GAN
Number enstromment
x100
Nember cnntruzment
w000
(d) Final CV regret in
(c) Cost training
(f) Success in training
Coustrite Steaein Regin
Cumulausve Cost
80
0.75
80
R0
866
9.25 035
0.75
1.25
1.50
1.75
50C
ALP PLR SPDL
Number of enstromment interactions
X104
4G
BAN
CAT
4C
GAN
(g) CV regret sp
(h) Final cost training
Final success in training
Figure 11: safety-passage results from runs in 5 seeds: a) Evolution of constraint violation regret during. b) Progression of expected discounted cumulative cost contexts drawn from the target context distribution. c) Progression of expected success rate in contexts drawn from the target context distribution. d) Constraint violation regret with respect to the target context distribution at the final curriculum iteration. e) Progression of expected discounted cumulative cost in contexts sampled during training. f) Progression of expected success rate in contexts sampled during training. g) Evolution of constraint violation regret with respect the target context distribution. h) Expected discounted cumulative cost of the final policies in contexts sampled during training. i) Expected success rate of the final policies in contexts sampled during training.
F.3.2 NUMBER OF ANNEALING ITERATIONS Kann
Fig. 15 showcases the ablation results for the number of annealing iterations K This parameter specifies how many curriculum iterations the annealing of a takes. Note that we anneal linearly. We experiment with Kann (10, 5, 20), where a = 10 reported in the main document. K Kann- 10 achieves zero-cost and 100% success rates in target contexts and provides low variance in CV regret. Decreasing or increasing the annealing pace causes a lower success rate and higher variance in CV regret.
F.3.3 COST THRESHOLD D
Fig. 16 illustrates the ablation results for the cost threshold D. This parameter determines the maximum expected cost SCG allows context under the context distribution @% (see equation 1 is reported in the main document. Kann 1
4). We experiment with Ú E {1, 0.5, 2}. where achieves zero-cost and 100% success rates in target contexts and provides low variance CV regret. Although Kann 0.5 achieve lower CV regret, it significantly increases the variance and causes non-zero cost in target contexts in multiple runs.
F.4 COMPARISON OF CONSTRAINED RL APPROACHES
In our experiments, all baselines, including DEFAULT, which does not generate curricula, and existing curriculum learning approaches PPO- LPg--aaranngin the constrained RL algorithm of choice. To further support our claim that SCG provides safer training than DEFAULT, we experiment with three other constraint RI. approaches in safety-push: Constrained Policy Optimization (CPO),
21

---

Published as a conference paper at ICLR 2025
Ortstaite Valitnn Rogms
000
01044
A12
SHOL
GoalGAN
ALL'
STOL
Oonstiative Ciet
100
7.5
2.79
0.0
8.0
05
1.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
3.00
0.0
0.5
1.5
2.0 inforrctioss
(a) Constraint violation regret
(b) Cost in larget contexts
(c) Success in target contexts
GraIGAN
200
MLP
8PDL
AlP
5906.
Cunobiiie One
1.00
2.5
1.00
2:00
500 CRT
NS oRT DEF AIP PLR saba Geal
0.0
0.3
10
2.0
0.0
2.0 Number anntrusment InBomaitoos
0.5
CILL
Number anstrumment
(d) Final CV regret in y
(c) Cost training
(f) Success in training
Consiraia Siitaing Regnes
20110
SYDL
Cunubatrve Chas
1.00
0.75
0.50-
R0
0.25
0.5
2.0
2.3
DES ALP PER SPOL Dina
SCB GRT
NS GHT PEP
ALP FUR SPOL Seal
(g) CV regret in
(h) Final cost in training
(i) Final success in training
Figure 12: Safety pueth results from runs in 5 seeds: a) Evolution of constraint violation regret during. b) Progression of expected discounted cumulative cost in contexts drawn from the target context distribution. c) Progression of expected success rate in contexts drawn from the target context distribution. d) Constraint violation regret with respect the target context distribution at the final curriculum iteration. e) Progression of expected discounted cumulative cost in contexts sampled during training. f) Progression of expected success in contexts sampled during training. g) Evolution of constraint violation regret with respect to the target context distribution. h) Expected discounted cumulative cost of the final policies in contexts sampled during training. i) Expected success rate of the final policies in contexts sampled during training.
2) Projection-Band -aaeed Constrained Policy Optimization (PCPO), and 3) First Order Constrained Optimization in Policy Space (FOCOPS). As DEFAULT, these approaches do not generate curricula but directly sample contexts from the target context distribution during training. Fig. [ee[[Ievveee that similar to DEFAULT with PPO- LPgganggnnn,nn, other constrained RL algorithms cause high constraint violation regret during training and fail to yield optimal policies multiple training runs.
F.5 CURRICULUM PROGRESSION
2 demonstrate the progression of curricula generated by CURROT, NAIVESAFECURRON CURROT4COST, SPDL, PLR. ALP-GMM, and GOALGAN, re- spectively, safety-maze, safety-goal, safety-passage, and safety-push environments. Figure 25 demonstrates the contexts drawn from the target context distribution during training runs of DEFAULT. NAIVESAFECURROT may have similar curricula to SCG in some settings, as considers reward and cost simultaneously but through penalized reward signal, which takes away the flexibility that SCG provides in prioritizing safe or performant contexts separately and sometimes together. CURROT4CoST takes cost into account, only, but fails to recognize that goals the hazards in safety-maze and safety-goal can lead to high constraint violation regret. ALP-GMM yields higher success than SPDL in safety-passage and safety-pusht due to being able to generate multi-model distribution via Gaussian mixture models, as opposed to unimodal Gaussian distributions of SPDL. PLR fails to prioritize safe or performant contexts.
Figs. 18
22

---

Published as a conference paper at ICLR 2025
S02 NaAnu
ScG
SCG-NAWISS
50S
5002
SOO NeVVIS
Canatzunn: Valaaton
Cacucaatee Cont.
10.0
0.75
1.00
0.50
60
1.5
0.25
0.00
0.5
0.0
1.0
1.5
2.0
0.0
5.5
10
15
2.0
(a) Constraint violation regret
(b) Cost in target contexts
(c) Success in target contexts
SIN 90G-N-1977
500
Nedonn
503 No.Ann
8C0 No05
Comalaaee Cies
10.0
1.00
0.75
0.50
50
0.2
0.0
2.0
1.0
1.5
NoPP
NyPS
NoPPps
Number
X13*
Number
X12
(d) Final CV regret in
(e) Cost in training
(f) Success in training
Construnt Visuaton Bogess
SU-G--NAAa
NoG-
0.4
0.50
20
5.25-
80
0.5
1.0
1.5
2.0
2.5
900
SCT Novann
901 MopS
30G NoPoPS
0.00
806
800 NoAtnn
800 Nore
900 No08
600 NoPPPS
Mumber
enntromment Intecnations
MaPP
(g) CV regret in yp
(h) Final cost training
(i) Final success in training
Eoeech
Tmuch
Ceech
Touch
Booch 40
Sowrch
Fpoch
Stooch
Fpacch
( SCG-NOANN
(k) SCG-NoPP
()) SCG-NoPS
(m) SCG-NoPPPS
Figure 13: Ablation study results in safety-push from runs in seeds: a) Evolution of constraint violation regret during. b) Progression of expected discounted cumulative cost in contexts drawn from the target context distribution. c) Progression of expected success rate contexts drawn from the target context distribution. d) Constraint violation regret with respect the target context distribution at the final curriculum iteration. e) Progression of expected discounted cumulative cost in contexts sampled during training. f) Progression of expected success rate in contexts sampled during training. g) Evolution of constraint violation regret in target contexts. h) Expected discounted cumulative cost of the final policies in contexts sampled during training. Expected success rate of the final policies in contexts sampled during training. j-m) Progression of curricula generated by SCG variations
23

---

Published as a conference paper at ICLR 2025
100
Chm Coss
140.
0.75
50
6.5-
025
0.00
--0775
2-0.5
a=1
e 0.75
6.6
0- 0.875
0
075
0-0.5
6- 0.755
(a) Constraint violation regret
(b) Cost target contexts
(c) Success in target contexts
Constrane: Valatan Magget
9875
Cunabattee Coet
100
0.75
5.0
50
8.9
Number of =mvmmoneett Internctonns X12
Nortbeaaf waweeaa--eaaaaaaa enrmanmee Interationss X12*
Namber of enverammnt iltenctins X12
(d) Constraint violation regret
(e) Cost target contexts
(I) Success in target contexts
e-077 5875
--075 0
100
Cunsbiti Cost
5.0
50
0.25
0.00
2.0
2.5
1.0
1.5
2.0
6.75
0-0.5
0-1
Namberr of anrirnannn interrctiona
Namber of enrervamnnt interactitts
(g) Final regret yp
(h) Cost training
(i) Success in training
Construust Videitio Hogret
mm75
8875
-=1
30
0.6
1.00
90
0.25
0.4
20
0.5
100
155
2.0
2.5
0.0
(i) CV regret in yp
(k) Final cost in training
(I) Final success in training
Figure 14: Ablation study for target GMM weight ratio a in safety-push from runs in 5 seeds: a) Constraint violation regret the final curriculum iteration. b) Expected discounted cumulative cost of the final policies in target contexts, c) Expected success rate the final policies target contexts. d) Evolution of constraint violation regret during. e) Progression of expected discounted cumulative cost contexts drawn from the target context distribution. f) Progression of expected success rate in contexts drawn from the target context distribution. g) Constraint violation regret with respect the target context distribution at the final curriculum iteration. h) Progression of expected discounted cumulative cost in contexts sampled during training. Progression of expected success rate in contexts sampled during training. j) Evolution of constraint violation regret with respect to the target context distribution. k) Expected discounted cumulative cost of the final policies in contexts sampled during training. I) Expected success rate of the final policies in contexts sampled during training.
24

---

Published as a conference paper at ICLR 2025
2..6- aaaannn g
1.0--
OFRigntt
0.75
20
g 0.50
0.a-
0.0-
Kun
Kare
20
(a) Constraint violation regret
(b) Cost in target contexts
(c) Success in target contexts
Conatrunt Valatum Meggus
Kas
10
Kam
Kuu
23
Kaw
20
Gundatio Coat
7.5
100
3.0
0.50
60
0.25
0.00 -4
8.
0.3
2.0
2.5
Namber of contronment Inlectotions x10*
Number of coveraanool Iitoractions
X10*
Namber covenannen interotions x1s*
(d) Constraint violation regret
(c) Cost target contexts
(f) Success in target contexts
Kmn 10
Kaw 5
20
Kip 10
Rue
10.0
Ounubailes Coet
OV Regis
75
0.75
8.0
0.50-
50
2.54
0.25
25
0.00
1.
1.5
2.0
2.3
0.0
1.2
1.5
2.0
Kaw
Naw
Marber of eov RErrot Ittrrmtins
Number envinanret Intermmtine
(g) Final CV regret in yp
(h) Cost training
Success in training
Conitram volaton Segret
Kiy 30
Kiss #5 Kan 20
80
1.5
Cumulattve Cost
10
10
0.25
80
Mumber
x10
(j) CV regret in y
(k) Final cost in training
(I) Final success in training
Figure 15: Ablation study for number of annealing iterations K Kann in safety-push from runs in
5 seeds: a) Constraint violation regret at the final curriculum iteration. b) Expected discounted cumulative cost of the final policies target contexts, c) Expected success rate of the final policies in target contexts. d) Evolution of constraint violation regret during. e) Progression of expected discounted cumulative cost contexts drawn from the target context distribution. f) Progression of expected success rate in contexts drawn from the target context distribution. g) Constraint violation regret with respect to the target context distribution at the final curriculum iteration. h) Progression of expected discounted cumulative cost in contexts sampled during training. i) Progression of expected success rate contexts sampled during training. j) Evolution of constraint violation regret with respect the target context distribution. k) Expected discounted cumulative cost of the final policies in contexts sampled during training. !) Expected success rate of the final policies in contexts sampled during training.
25

---

Published as a conference paper at ICLR 2025
1.5
Comalative Coe
00 Rugge
75
0.25
n.0
0-1
2=2
0-0.5
5-2
6-1
(a) Constraint violation regret
(b) Cost in target contexts
(c) Success in target contexts
Comsirunt Vidagon Mugret
G-1
6505
5-2
0-1 -d-as -0-2
0-1 -0-3 0
150
Combirree Coet
10.0
1.00
0.75
50
0.50
2.50
0.25
0.00
8.3
0.5
1.0
15
2.5
25
0.0
5.5
1G
1.5
2.0
2.3
0.0
5.5
100
1.5
2.0 2.5
(d) Constraint violation regret
(e) Cost in target contexts
(I) Success in target contexts
5-1
0-1 0-03 0-
100
1.00
75
Cunviaatve
7.8
0.75
5.0
0.50
2.5
0.25
25
1.5
2.0
2'5
0.0
1.0
1.5
2.0
0-
0-0.5
6
X10
(g) Final CV regret in
(h) Cost in training
(i) Success in training
Contewaa Volation Hugret
G-1
6w0.5
6-2
80
0.b.
1.00-
10
0.75
20
U.A.
0.25
0.50-
80
1.0
1.5
0.00
Number enveromment Internatiions K104
0-505
--2
d-1
0-005
0-2
(i) CV regret in yp
(k) Final cost training
(l) Final success in training
Figure 16: Ablation study for cost threshold δ safety-push from runs seeds: a) Constraint violation regret at the final curriculum iteration. b) Expected discounted cumulative cost of the final policies in target contexts, c) Expected success rate of the final policies in target contexts. d) Evolution of constraint violation regret during. e) Progression of expected discounted cumulative cost contexts drawn from the target context distribution. f) Progression of expected success rate in contexts drawn from the target context distribution. g) Constraint violation regret with respect the target context distribution at the final curriculum iteration. h) Progression of expected discounted cumulative cost in contexts sampled during training. Progression of expected success rate in contexts sampled during training. j) Evolution of constraint violation regret with respect to the target context distribution. k) Expected discounted cumulative cost of the final policies in contexts sampled during training. I) Expected success rate of the final policies in contexts sampled during training.
26

---

Published as a conference paper at ICLR 2025
10.0
200
7.5:
0.75
5.0
100
2.5-
905
DC2 PPOLag
D2F CPD
PSF POCOPS
ProLag
CPO
PSP
POCCPS
PrOLag
EPG
PCPO
POCOPS
(a) Constraint violation regret
(b) Cost target contexts
(c) Success in target contexts
Conatuust Valation Mageet
500
JEP-2POLag
DISP-POOOORS
#20C
DEF-PCPO
DEP PPCCag
DEP PCPO
Oanabiiber Coet
7.5
100
5.0
00
0.2
Xumber of REVVERIEENE Interaation Tions
x10
Namber of wevernmmenn Intecsctinns K10e
Nimber of aeveeeeeeent internations K1ge
(d) Constraint violation regret
(e) Cost target contexts
(I) Success in target contexts
DEP-OCOPS
500 PoOCag
BESCIO
DEP-EOOOPS
PSCC.g
10.0
Oanstorti Cust
7.8
0.75
3.0
0.50
2.5
0.25
0.00
500G
DCG PhOLag
DEP EVG
JES PC1O
Pisk rocops
0.5
1
1.3
2.5
2.5
0.0
0.5
1.
13
20
Number movecnment
Interactions
Number anveenment
x10
(g) Final CV regret in Vp
(h) Cost training
Success in training
Contrrant Stolatinn Hogret
6D0
CPO
DEF FOCOPS
16
1.00
0.75
80
0.50.
30
0225
80
2.5
Number anvtnstment
X10*
PPOLlg
CPU
PC30
200015
ProLag
POCOPS
(j) CV regret in
(k) Final cost training
(I) Final success in training
Figure 17: Comparison of constrained RL algorithms in safety-push from in 5 seeds: a) Constraint violation regret at the final curriculum iteration. b) Expected discounted cumulative cost of the final policies in target contexts, c) Expected success rate of the final policies target contexts. d) Evolution of constraint violation regret during. e) Progression of expected discounted cumulative cost contexts drawn from the target context distribution. f) Progression of expected success rate in contexts drawn from the target context distribution. g) Constraint violation regret with respect to the target context distribution the final curriculum iteration. h) Progression expected discounted cumulative cost in contexts sampled during training. Progression of expected success rate in contexts sampled during training. j) Evolution of constraint violation regret with respect to the target context distribution. k) Expected discounted cumulative cost of the final policies in contexts sampled during training. !) Expected success rate the final policies in contexts sampled during training.
27

---

Published as a conference paper at ICLR 2025
a3225
5
Beoch 20
0.75
Spoch
0.75
Tiunh 5
Fooch
0.5
Spocs 100
toooch 450
0.25
(a) Safety-maze
(b) Safety-goal
(c) Safety-passage
(d) Safety-push
Figure 18: Curricula generated by CURROT.
Booch x325
Eooch Feech 20
0.75
0.75
Epont:
Epach 50
0.75
Upooh 100
tooch 650
Fooch 100
0.5
0.25
(a) Safcty-maze
(b) Safcty-goal
(c) Safety-passage
(d) Safcty-pushy
Figure 19: Curricula generated by NAIVESAFECURROT.
Eooch
=325
Eooch 5
Eooch 20
0.75
Epach
0.75
Epoct
Epoch 40
Frach 100
Fooch 450
0.5
Fooch 100
455
(a) Safety-maze
(b) Safety-goal
(c) Safety-passage
(d) Safety-push
Figure 20: Curricula generated by CURROT4COST.
Spoch
Gooch
23225
Eoach
Eooch 20
0.75
Truch
Fooch
0.75
Bpach 100
Foech 450
0.5
Fpuch
Faxch 100
05
(a) maze
(b) Safety-goal
(c) Safety-passage
(d) Safcty-push
Figure 21: Curricula generated by SPDL.
28

---

Published as a conference paper at ICLR 2025
a3225
Enach
Bnoeh 20
0.75
Spoch
0.75
Fooch
0.5
Spoca 100
toooh 650
0.5
0.25
(a) Saaety-maze
(b) Safety-goal
(c) Safety-passage
(d) Safety-push
Figure 22: Curricula generated by PLR.
Booch
x325
Enoch Fooch 20
0.75
0.75
Epont:
Epoch 50
0.75
0.5
Upoch 100
booch 650
Foech 100
0.25
(a) Safcty-maze
(b) Safcty-goal
(c) Safety-passage
(d) Safcty-push
Figure 23: Curricula generated by ALP-GMM.
2325
Epoch
Eooch 20
0.75
Epach
Epect
Epoch 40
Procs 100
Foeec 450
0.5
Fooch
-.5
(a) Safety-maze
(b) Safety-goal
(c) Safety-passage
(d) Safety-push
Figure 24: Curricula generated by GOALGAN.
Epoch
Gooch
2322
S
Eooch 20
0.75
0.75
Truch
Fooch
0.75
2.25
Brancs 100
Feeac 450
Fpuch
Faceh 160
05
(a) Safety-maze
(b) Safety-goal
(c) Safety-
(d) Safety-push
Figure 25: Curricula generated by DEFAULT.
29


Reviews:
[{"review_id": "PM3jNccX7w", "reviewer": "Reviewer_Neqb", "summary": "The paper tried to ensure the safety of curriculum learning.\nUnlike previous curriculum learnings that tried to make the agent perform better, this paper prioritized safety but still cared about performance. \nThe paper compared their results with CURROT(Klink 2022) on safety maze, safety-goal, and safety-push, which did not care about safety. \nThe results showed that SCG can achieve a low CV regret and still achieve a high success rate.", "questions": "1 A proper baseline should be using a goal that cares about both safety and performance. The score can weighted sum of both scores. Using different weights will be interesting. \n\n2 Since the paper cares about cumulated regret, have you thought about the possibility of making more mistakes first and learn those mistakes can avoid future mistakes?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The performance seems to be better.\nIt can achieve optimal policy.\nCompared to CURROT,  it is safer.\nCompared to DEFAULT, it is more sample efficiency.", "weaknesses": "1 The paper only compared simple environments. It does not compare to the environments in the previous work. For example, Point Mass and Bipedal Walker Stump Tracks. \n\n2 The method seems trivial to me. It is mostly based on CURROT.\n\n3 There are many other curricular learning methods with different settings. This paper can show that it can be used in the same setting as CURROT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pOve87QOz9", "reviewer": "Reviewer_RChd", "summary": "This paper proposes modifying curriculum generation techniques to incorporate safety constraints. It does so by constraining the curriculum to not produce tasks which tend to result in constraint violations.", "questions": "What is the main limitation to applying this approach to higher-dimensional environments like Brax?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The clear exposition quickly brings the reader up to speed with the pragmatic approaches in the space of curriculum generation. It makes it clear how safety constraints are not incorporated in current techniques, providing a clear gap for this method to fill. The method appears to work approximately as advertised, resulting in similar performance to prior works with less violation of constraints.", "weaknesses": "The biggest weakness of this work is the lack of a demonstration of scalability. The 2-d navigation task with 2-d goals provides a very small space for curriculum design where all tasks can be approximately enumerated. It is unclear how this approach would scale to situations where the spaces of possible tasks is combinatorially large. However, as this is a concern that is shared with much of the prior work, I do not see this as a reason in itself to reject the paper.\n\nThe main limitation keeping me from raising my score is that it appears the impact of this work is currently somewhat limited. This is in-part because of the scale, in part because the gains on both safety and performance are not as dramatic as one would hope, and in part because it appears to be far from the point where it could be tested in concrete applications. My evaluation of this paper would be significantly higher if it were shown to be applicable to a robotics domain, where safety during training is a critical component. I am guessing this is not in the current draft because it is difficult to scale underlying methods like CURROT to those domains, but if that is the case, then it is not clear if this approach would still be applicable with whatever approach does eventually scale. These uncertainties make it less clear for the reader what to take away from this paper for their own future research. \n\nThe one point where the paper can be made significantly more clear is explaining the apparently constraint-violating tasks in Figure 5 c and Figure 3 (right). My sense is that these tasks are not as constraint-violating as they seem since they have a high-tolerance, but it cannot be confirmed from the diagram since there is no correspondence between color and radius of the tolerance. I think this could be resolved by adding a few circles to-scale to ones side of these figures which visually depicts what color would correspond to a given radius. This could then let me check that the tolerance of the radius is sufficient to let the agent avoid a constraint violation.\n\nMinor point: \n- 188 and 193 I think these sentences do not completely describe the objective, since they does not say what happens when \"finding the optimal policy\" and \"minimising the constrained violation regret\" are in tension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xKn1ecODwY", "reviewer": "Reviewer_izzQ", "summary": "This paper presents a safe curriculum generation (SCG) approach aimed at enhancing safety during training while improving sample efficiency. SCG generates task sequences as context distributions, prioritizing tasks with minimal safety violations over high-reward tasks in the early stages to safely converge on the target context distribution. Empirical studies indicate that SCG outperforms state-of-the-art curriculum learning methods and their naively adapted safe versions, achieving optimal performance with the fewest constraint violations during training.", "questions": "- Was the cost annealed in the Naive approach as well?\n- How does the proposed approach compare to other safe-RL algorithms in these domains?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper effectively highlights the gap in current curriculum learning (CL) algorithms and the importance of developing safe CL approaches.\n- Background and related work are thoroughly described, making the contribution and method easy to understand.\n- The paper is well-organized, with clear explanations and logical flow.\n- The algorithm demonstrates state-of-the-art performance across multiple benchmarks.", "weaknesses": "- Limited Novelty: The approach’s novelty is modest, as it primarily adds an additional constraint to the existing CURROT algorithm.\n- Empirical Limitations: \n    - The empirical evaluation lacks comparisons with other safe-RL baselines and is limited to three domains.\n    - Sensitivity to newly introduced parameters (e.g., \\apha_k) is not thoroughly analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nVJtvLB54Q", "reviewer": "Reviewer_SUye", "summary": "This paper introduces a safe curriculum generation (SCG) approach. Using safety constraints, SCG constraints the evolution of the curriculum to stay in the safe zone while annealing towards the maximization of the policy's performance. Based on the CURROT, SCG includes safety constraints to ensure that agent will not venture and make dangerous movements during training. Focusing on Safety-Push, Safety-Maze, and Safety-Goal environments, the authors explores how such design choices makes SCG outperform CURROT and other baselines.", "questions": "* What is the motivation for choosing the safety-maze, safety-goal, and safety-push as benchmarks? Why would you consider that these baselines would demonstrate the pros and cons of SCG better than other benchmarks?\n* Is the 'Default' mentioned in the empirical studies results of running RL without any curriculum generator?\n* As this algorithm is inspired by CURROT, I understand why CURROT was chosen as baselines. But as for the rest, why were the other baselines selected for? What does comparing SCG with the selected baselines demonstrate?\n* In figure 4, we see boxes and bars. Would it be correct to assume that boxes are standard deviations and bars are the maximum and minimum of the data distribution?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper is generally well written and the structures are easy to follow, with the headers clearing marking out the points to be made across. The relative work is quite comprehensive, and extensive experiments conveys the main points well.", "weaknesses": "- The algorithm is heavily inspired by CURROT and it feels like a somewhat of a safe RL implementation of CURROT. While some may consider this to be a incremental improvement of algorithm, I think the paper could be made stronger if the authors could share the design insights that led to SCG. For example, why did the authors decided to design an algorithm inspired by CURROT? The algorithmic features that differentiates SCG from CURROT, what were their motivations? If the authors could summarize their insight in a way that could help other researchers to make improvements to other existing algorithms in RL, the contribution of this paper would be much more significant.\n\n- I think the paper does not convey well some of the underlying assumptions and motivations, especially towards the empirical analysis. I have included the details in the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)