Diffusion-based Curriculum Reinforcement Learning
Authors: Erdi Sayar
Giovanni Iacca
Ozgur S. Oguz
Alois Knoll
Venue: NeurIPS 2024 poster (2024)
Keywords: curriculum reinforcement learning
reinforcement learning
diffusion models
Primary area: reinforcement_learning
TL;DR: A novel diffusion based curriculum reinforcement learning

Abstract: Curriculum Reinforcement Learning (CRL) is an approach to facilitate the learning process of agents by structuring tasks in a sequence of increasing complexity. Despite its potential, many existing CRL methods struggle to efficiently guide agents toward desired outcomes, particularly in the absence of domain knowledge. This paper introduces DiCuRL (Diffusion Curriculum Reinforcement Learning), a novel method that leverages conditional diffusion models to generate curriculum goals. To estimate how close an agent is to achieving its goal, our method uniquely incorporates a $Q$-function and a trainable reward function based on Adversarial Intrinsic Motivation within the diffusion model. Furthermore, it promotes exploration through the inherent noising and denoising mechanism present in the diffusion models and is environment-agnostic. This combination allows for the generation of challenging yet achievable goals, enabling agents to learn effectively without relying on domain knowledge. We demonstrate the effectiveness of DiCuRL in three different maze environments and two robotic manipulation tasks simulated in MuJoCo, where it outperforms or matches nine state-of-the-art CRL algorithms from the literature.

Diffusion-based Curriculum Reinforcement Learning
Erdi Sayar', Giovanni Iacca', Ozgur S. Oguz', Alois Knoll' Technical University of Munich' University of Trento Bilkent University?
erdi www.ssaaaauuaa... de, giovanni. siovanmi.iaccaauuita. wwwcaauniitn.n....... it, ozgurêcs bilkent. bilkent.edu.tee edu knoll@ing tum.de
Abstract
Curriculum Reinforcement Learning (CRL) is an approach to facilitate the learning process of agents by structuring tasks in sequence of increasing complexity. Despite its potential, many existing CRL methods struggle to efficiently guide agents toward desired outcomes, particularly in the absence of domain knowledge. This paper introduces DiCuRL (Diffusion Curriculum Reinforcement Learning), a novel method that leverages conditional diffusion models to generate curriculum goals. To estimate how close an agent to achieving its goal, our method uniquely incorporates Q-function and a trainable reward function based on Adversarial In- trinsic Motivation within the diffusion model. Furthermore, promotes exploration through the inherent noising and denoising mechanism present in the diffusion models and is environment-agnostic.. This combination allows for the generation of challenging yet achievable goals, enabling agents learn effectively without relying on domain knowledge. We demonstrate the effectiveness of DiCuRL three different maze environments and two robotic manipulation tasks simulated in MuloCo, where it outperforms or matches nine state-of-the-art CRL algorithms from the literature.
Introduction
Reinforcement learning (RL) is computational method that allows agent to discover optimal actions through trial and error by receiving rewards and adapting its strategy to maximize cumulative rewards. Deep RL, which integrates deep neural networks (NNs) with RL, is an effective way to solve large-dimensional decision-making problems, such as learning to play video games [ chess 7 g One of the main advantages of deep RI. is that can tackle difficult search problems where the expected behaviors and rewards are often sparsely observed. The drawback, however, is that it typically needs to thoroughly explore the state space, which can be costly especially when the dimensionality of this space grows.
3. Go 14, and robot manipulation tasks 5 6
Some methods, such as reward shaping [9], can mitigate the burden of exploration, but they require domain knowledge and prior task inspection, which limits their applicability. Alternative strategies have been proposed to enhance the exploration efficiency in domain agnostic way, such as priori- domain-agnostic
tizing replay sampling {8] 10 [1] or generating intermediate goals [12 13 14
[5
16
I7 I8 19..
This latter approach, known as Curriculum Reinforcement Learning (CRL., focuses on designing suitable curriculum guide the agent gradually toward the desired goal.
Various approaches have been proposed for the generation of curriculum goals. Some methods focus on interpolation between source task distribution and target task distribution 20 21 22 [7]. However, these methods often rely on assumptions that may not hold in complex RL environments, such as specific parameterization of distributions, hence ignoring the manifold structure in space. Other approaches adopt optimal transport [3] 23 but they are typically applied in less challenging
exploration scenarios. Curriculum generation based on uncertainty awareness has also been explored, but such methods often struggle with identifying uncertain areas as the goal space expands [5] 24 12
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Some research minimizes the distance between generated curriculum and desired outcome distribu- tions using Euclidean distance, although this approach can be problematic in certain environments Other methods incorporate graph-based planning, but require an explicit specification of obstacles 26 27]. Lastly, approaches based on generative Al models have been proposed. For in- stance, 14 uses GANs to generate tasks of intermediate difficulty, but it relies on arbitrary thresholds.
19
25
Alternatively, [28 29 30
apply diffusion models in offline RL settings [4 28 30).
Despite these advancements, existing CRL approaches still struggle with generating suitable inter- mediate goals, particularly in complex environments with significant exploration challenges. To overcome this challenge, in this paper, we propose DICURL (Diffusion Curriculum Reinforcement Learning.. Our method leverages conditional diffusion models dynamically generate curriculum goals, guiding agents towards desired goals while simultaneously considering the Q-function and a trainable reward function based on Adversarial Intrinsic Motivation (AIM) (31]
Contributions
Unlike previous offline RL approaches (2 29 30 that train and use diffusion models
for planning or policy generation relying on pre-existing data, DICURL facilitates online learning, enabling agents to learn effectively without requiring domain-speciff knowledge. This is achieved by three key elements. ! The diffusion model captures the distribution of visited states and facilitates exploration through its inherent noising and denoising mechanism. 2 As the Q-function predicts the cumulative reward starting from a state and given goal while following a policy, we can determine feasible goals by maximizing the Q- D-function, ensuring that the generated goals are challenging yet achievable for the agent. 3 The AIM reward function estimates the agent's 's proximity the desired goal and allows us to progressively shift the curriculum towards the desired goal.
We compare our proposed approach with nine state-of-the-ar CRL baselines in three different maze environments and two robotic manipulation tasks simulated in MuJoCo (32). Our results show that DICURL surpasses or performs on par with the state-of-the-art CRL algorithms.
2
Related Work
Curriculum Reinforcement Learning CRL 33 algorithms generally adjust the sequence of learning experiences to improve the agent's performance or accelerate training. These algorithms focus on formulating intermediate goals that progressively guide the agent toward the desired goal, and have
been successfully applied to various tasks, mainly the field of robot manipulation 34 35 36 37].
Hindsight Experience Replay (HER) 38 tackles the challenge of sparse reward RL tasks by employ- ing hindsight goals, considering the achieved goals pseudo-goals, and substituting them for the desired goal. However, HER struggles to solve tasks when the desired goals are far from the initial position. Hindsight Goal Generation (HGG) [99 addresses the inefficiency issue inherent in HER by generating hindsight goals through maximizing a value function and minimizing the Wasserstein distance between the achieved goal and the desired goal distribution.
CURROT (33 and GRADIENT [3] both employ optimal transport for the generation of inter- mediate goals. CURROT formulates CRL as constrained optimization problem and uses the Wasserstein distance to measure the distance between distributions. Conversely, GRADIENT intro- duces dependentt contextual distance metrics and can manage non- -on-paraettric distributions in both continuous and discrete context settings; moreover, it directly interprets the interpolation as the geodesic from the source to the target distribution.
GOAL-GAN 14 generates intermediate goals using a Generative Adversarial Network (GAN) 39
without considering the target distribution. A goal generator is used to propose goal regions, and a goal discriminator is trained to evaluate if goal is at the right level of difficulty for the current policy. The specification of goal regions is done using an indicator reward function, and policies are conditioned on the goal as well as the state, similarly to a universal value function approximator [40].
PLR [.5 uses selective sampling to prioritize instances with higher estimated learning potential for future revisits during training. Learning potential is estimated using TD-Errors, resulting in the creation of more challenging curriculum. VSD [16] estimates the epistemic uncertainty of the value function and selects goals based on this uncertainty measure. The value function confidently assigns high values to easily achievable goals and low values to overly challenging ones. ACL [8 maximizes the learning progress by considering two main measures: the rate of improvement in prediction accuracy and the rate of increase in network complexity. This signal acts as indicator of

---

the current rate of improvement of the learner. The ALP-GMM [7] fits Gaussian Mixture Models (GMM) using an Absolute Learning Progress (ALP) score, which defined as the absolute difference in rewards between the current episode and the previous episodes. The teacher generates curriculum goals by sampling environments to maximize the student's ALP,, which is modeled by the GMM.
Finally, OUTPACE [12] employs trainable intrinsic reward mechanism, known as Adversarial Intrinsic Motivation (AIM) [31 (the same used our method) which is designed to minimize the Wasserstein distance between the state visitation distribution and the goal distribution. This function increases along the optimal goal-reaching trajectory. For curriculum goal generation, OUTPACE uses Conditional Normalized Maximum Likelihood (CNML), to classify state success labels based on their association with visited states, out-of-distribuite samples, or the desired goal distribution. The method also prioritizes uncertain and temporally distant goals using meta- meta-learning- leat--aaannnbbeeee -bsed uncertainty quantification and Wasserstein dereeei-ddsttnneebbaae based temporal distance approximation.
Diffusion Models for Reinforcement Learning UniPi 42 leverages diffusion models to generate video as a planner, conditioned on an initial image frame and a text description of a current goal. Subsequently, a task-specific policy is employed to infer action sequences from the generated video using an inverse dynamic model. AVDC 43 constructs a video-based robot policy by synthesizing a video that renders the desired task execution and directly regresses actions from the synthesized video without requiring any action labels or inverse dynamic model. takes RGBD observations and textual goal description as inputs, synthesizes a video of the imagined task execution using a diffusion model, and estimates the optical flow between adjacent frames in the video. Then, using the optical flow and depth information, it computes robot commands.
Diffusion Policy 44 uses diffusion model learn policy through conditional denoising diffusion process. BESO 45 adopts an imitation learning approach that learns a goal-specified policy without any rewards from an offline dataset. DBC [46) uses diffusion model learn state- action pairs sampled from an expert demonstration dataset and increases generalization using the joint probability of the state-action pairs. Finally, Diffusion BC 47 uses a diffusion model imitate human behavior and capture the full distribution of observed actions on robot control tasks and 3D gaming environments.
Limitations of current works and distinctive aspects of DICURL The aforementioned studies typically require offiin data for training. Both [42] and 43 employ diffusion models to synthesize videos for rendering the desired task execution, and actions are then inferred from such videos. Studies such as [4 45 46 (7) also focus learning policies from offline datasets. Despite these efforts, reliance on inadequate demonstration data can lead to suboptimal performance [44]. Distinct from these approaches, our method instead does not rely on prior expert data any pre-collected datasets. As an off-policy RL method, DICURL collects in fact data through interaction with the environment.
3 Background
We now introduce the background concepts multi-goal RL, Soft Actor-Critic (SAC), Wasserstein distance, Adversarial Intrinsic Motivation (AIM), and diffusion models.
3.1 Multi-Goal Reinforcement Learning
In the context of multi-goal RI., we can formulate the RL problem goal-oriented Markov decision process (MDP) characterized by continuous state and action spaces. This MDP is defined by the tuple (S, A, T.p, 7r)). Here, S represents the state space, A denotes the action space, and G is the goal space. The transition dynamics, T(s'ls, [[s'[ss a). describes the probability of transitioning to the next state s' given the current state and action a. The joint probability distribution over the initial states and the desired goal distribution is represented by p(Su, 9). and ?r E [00 1] is a discount factor.
We utilize the AIM reward function rip. as outlined in Section 3.3 The objective is to identify the optimal policy æ that maximizes the expected cumulative reward approximated by the value function QT g, a). This function can be expanded to the Universal Value Function (UVF), goal-conditioned value function incorporating the goal into value estimation. The UVF, defined as v (s. 9), where is the current state, g is the goal, and is the policy, estimates the expected return
3

---

from state under policy given goal g. The UVF has been shown to successfully generalize to unseen goals. which makes it viable approach for multi-goal RL
3.2 Soft Actor-Critic
Soft Actor-Critic (SAC) 488 is an off-policy RL algorithm that learns a policy maximizing the sum of the expected discounted cumulative reward and the entropy of the policy, namely: J(=)
Ei=o.E. Ee.wB r + ag H( st, The policy aa s) state to distributions over actions, parameterized by v. A state-action value function is defined as Qa(s, g. :S x G x A I. parameterized by ø. The policy parameters can be learned by directly minimizing the expected KL divergence and the Q-value is trained by minimizing the soft Bellman residual, which are defined as the following loss functions:
+ P(A) defines a map from
Cx E (s.g)) [ENTT (ae log (T (a s, Qo(s.g.a)]]]
(1)
Lg = (s,,,,, 9)8B Q+ (8, (r + B. (V (s))
where Vo(,, g) = Ba- a [Q 9, a) - log To (a 8, g)]. B is the replay buffer and Or the temper- ature parameter that controls the stochasticity of the optimal policy.
3.3 Wasserstein Distance and Adversarial Intrinsic Motivation Reward Function
The Wasserstein distance offers way to quantify the amount of work required transport one distribution to another distribution. The Wasserstein-p distance Wp between two distributions f and on a metric space (X', where X' is a set and d denotes a metric on X, is defined as follows [49
1/p inf Bi [d(X, Y 7EL((U.))
Wp(n, u)
inf yEll(.))
where [(,, u) denotes the of all possible joint distributions 7(x, whose marginals are μ and v. Intuitively, 7xx y) tells what is the least amount of work, as measured by d, that needs to be done to convert the distribution y into the distribution [50]. A timestep quasi-metric d' (8,g) can be used as distance metric. It estimates the work needed to transport one distribution to another, representing the number of transition steps required reach the goal state g E G for the first time when following the policy π
As proposed by Durugkar et al. [11, the AIM reward function be learned by minimizing the Wasserstein distance between the state visitation distribution Pr. and the desired goal distribution G. Through the minimization of the Wasserstein distance Wi (Pr. 9) (for p = Wi is known as the Kantorovich-Rubinste
distance), a reward function can be formulated to estimate the work required to transport the state visitation distribution far the desired goal distribution g and is expressed as follows: W, (Px: 9) = sup ||l| [Eg-g [f (g)] Exsp, [f))] If the state visitation
distribution (s) comprises states that optimally progress towards the goal g. the potential function fs) increases along the trajectory, reaching its maximum value f(g). The reward function, which can be approximated using a neural network, denoted as r increases as the states approach the desired goal E G. rh can be trained using the data collected by the policy æ. Leveraging the estimation of the Wasserstein distance W1 (px, 9), the loss function for training the parameterized reward function r is defined as follows:
C.p = E(s (f)(s)
(g) + (ss.s' max (1fā(() fo -
where the second component of the sum is a penalty term and the coefficient À is necessary to ensure smoothness [3I]. The reward can be calculated as gs) = (s(s) f (g), which is the negative of the Wasserstein distance WW (Pr, G).
3.4 Diffusion Models
Diffusion models [51 express probability distribution p(zo) through latent variables in the form po (mo) = f po (mo:N) ) day,N, where æ #1,..., Ny are latent variables of the same dimensionality as the data xo p (xo). They are characterized by a forward and a reverse diffusion process.

---

The forward diffusion process approximates the posterior q (#1.N (o)) using a Markov chain that perturbs the input data by gradually adding Gaussian noise 2n 9 (xp) N steps with predefined variance schedule ,1....,BNN. This process is defined as:
N
q (ENN an) == II
a (ate xk--1)
(5)
k=1
q (rt xk--1) == N Dki V B,
The reverse diffusion process aims recover the original input data from the noisy (diffused) data. learns to progressively reverse the diffusion process step by step and approximates the joint distribution pe (2o.N). This process defined
N
Po (Io:N) == (xNN II Po (Ix-1 xxk)
(6)
R=1
Po (ax-1 2x.) == NV (x (x 1o (xx,k) k), Eg (xx,
where p p*N) = NV (æN: 0) The optimization of the reverse diffusion process achieved by maximizing the evidence lower bound (ELBO) E,g In N=
52 Once trained, sampling data
o(a1: ma)
from Gaussian noise v ~ p (#xx and running through the reverse diffusion process from k N to k= 0 yields an approximation of the original data distribution.
Methodology
As discussed earlier, in multi-goal RL, the desired goal is sampled from desired goal distribution in each episode, and the agent aims to achieve multiple goals. By integrating a curriculum design into multi-goal RL, we can reformulate a task such a way that it starts with easier goals and progressively increases in difficulty. Our curriculum works follows. Given the presence of two different types of timesteps for the diffusion process and the RL task, we denote the diffusion timesteps using subscripts k { NJ and the RL trajectory timesteps using subscripts E (l..... T). Our curriculum goal generation takes the state as input. It then outputs curriculum goal set G., which is obtained through the reverse diffusion process of a conditional diffusion model as follows:
crffusion-mmdel-baae
goal-gennration method
N
Ge = (go:N = (gN: O, I)
II A=1
pe (gk-1
gk:
(7)
where go is the end sample of the reverse diffusion process used curriculum goal. Commonly, Po (gs-1 gi: 8) a conditional distribution parametrized by 0 and is chosen to model a multivariate Gaussian distribution N (gk-15 M4g (gk, 8, k). Eg (gt, 8, k)). Following [5T], rather than learning
the variances of the forward diffusion process, we assign a fixed covariance matrix Eg (g,, 8, 3,1 and mean defined as:
1 Var
BR V1
Pe (gk, 8, k)
g+
eg (gk: 8, k:)
ās
Initially, we sample a Gaussian noise gN ~ (0, I). We then apply the reverse diffusion process parameterized by 0 starting from the last step N and proceeding backward to step 1:
B Vax (1 āx) 60 (gg, 8, k) + VBL
gk
Org
E N,,,0 I), for l: N....,1
For the case k (last term of the reverse diffusion process), set to O, to enhance the sampling quality by ignoring Vßl and edge effects. In fact, as empirically demonstrated by [51], training the diffusion model yields better results when utilizing simplified loss function that excludes the weighting term. Thus, we adopt the following simplified loss function to train the conditional eg-model, where eo is function approximator designed predict !
Details about deriving the conditional diffusion model equation provided in Supp. Mat.
5

---

La( E- (...). [||e co (Vāggo vI - Cge, 8, k) })]
(10)
where 4 a uniform distribution over the discrete set (1...., N) and B denotes the replay buffer collected by policy и.
To sample feasible curriculum goals for the agent, we integrate the Q-function and the AIM reward function into the loss. The total loss function for training the diffusion model is defined as follows:
L = Ed La(
Qo(*, So, (8,, go)) + &r (B 9a)1
(11)
The rationale is that, while minimizing the loss L allows us to accurately capture the state distribution, we also aim to simultaneously maximize the expected value of Q and the AIM reward function rip- using the weights Ea. Eq. and &r to adjust the relative importance of the three components of the loss function. More in detail, the Q-function predicts the cumulative reward starting from state and following the policy, while the AIM reward function estimates how close an agent is to achieving its goal. By maximizing Q and the AIM reward, we can generate curriculum goals that are neither overly simplistic nor excessively challenging, progressing towards the desired goal.
In Eq. [1), go is obtained by sampling experiences from the replay buffer using Eq. (9) through a reverse diffusion process parameterized by 9. Therefore, taking the gradient of Q and the AIM reward r.g involves mack-propaggttin through the entire diffusion process. After we obtain the set of curriculum goals G. from the diffusion model, we use a bipartite graph G ([V,, VyJ, E)) with edge costs w, composed of vertices V, (i.e., the curriculum goal candidates derived from the diffusion model) and vertices Vy (i.e., the desired goals), where E represents the edge weights, and select the optimal curriculum goal g We employ the Minimum Cost Maximum Flow algorithm in order to solve the bipartite matching problem and identify the edges with the minimal cost w:
(g: g) N
max G
Σ
W
where:
w =
(12)
P E of Eś-
The specifics of the generation of intermediate goals through diffusion models are explained in Algorithm [ while the overall algorithm is outlined in Algorithm 2
In Algorithm| I during the training iterations, we sample Gaussian noise gN NO, I) to denoise the data in the reverse diffusion process. Between lines 5 and 7 we iterate from time step to 1, where we perform the reverse diffusion process using Eq. 9 and subtract the predicted noise ea from gN to denoise the noisy goal iteratively. We then uniformly sample a timestep l from the range between 1 and N (line 8) and sample Gaussian noise ~ (O, I) (line 9 in order to calculate the diffusion loss defined in Eq. [O0). In line 10 we calculate the diffusion loss, the Q-value, and the
AIM reward function using the generated goal go from the reverse diffusion process. Then in line
11 we calculate the total loss defined in Eq. (11) and update the diffusion model parameters @ using gradient descent. Finally, we return the generated goal go-
In Algorithm we begin by defining an off-policy algorithm denoted as A. While any off off-policy algorithm could be employed, we choose Soft Actor-Critic (SAC) to align with the baseline algorithms tested in our experiments. In 5ine we sample the initial state and provide the curriculum goal ge to the policy, along with the current state. The policy generates an action (line 7 and executes it in the environment (line B). Subsequently, the next state and reward are obtained (line Dp. Then, we provide the minibatch b to the curriculum goal generator in line I[ to generate curriculum goal set G.- Then we find the optimal curriculum goal ge using bipartite graph optimization (line 12). Furthermore, the loss functions defined in Eq. (1) and Eq. (2) are calculated, and the networks approximating and Q. as well as the AIM reward function rie defined in Eq. 4 are updated. Between line 15 and [2] we test rollouts and extract the achieved state of the agent using á and calculate the success rate in reaching the desired goal within a threshold value K.
IIn principle, instead of sampling experiences from the replay buffer and providing them the diffusion model to generate a curriculum goal set, only the state of the last timestep, sr. could be provided to the diffusion model to obtain a curriculum goal. The difference is that giving the sampled experiences to the diffusion model generates many curriculum points, while using only the last timestep sr generates only a single curriculum point, eliminating the need to apply any selection strategy. This is investigated in Supp. Mat. B

---

Algorithm Diffusion Curriculum Goal Generator
1: Input: state 8, no, of reverse diffusion timestep N, training step M
2: Obtain states from the minibatch b
3: for I,.... M do
D. Training iterations
4: EN (O,
5:
for k N N..... do
> Reverse diffusion process
6:
(),
7:
9k-1 Vax
(9k VT-ax
colon, _
Bu
VBke
Using Eq. (
8:
~~ Uniform((),..... N))
9:
e~NN(0,1)
10:
(s. Calculate difffusion C0), (s, Bo, x())) r(g), go) Calculate with the generated goal go
11:
Calculate total loss (0) - Qs, Bo, a()) - (.r(go.90 9d)
> Eq
12:
x 0 VoL
D Take gradient descent step and update the diffusion weights
return go
Algorithm 2 RL Training and Evaluation
1: Input: no. of episodes E, timesteps T
2: Select an off- off-policy algorithm
In our case, A is SAC
3: Initialize replay buffer B <- 0, De + {ga) and networks QΦ Tw. Tip
4: for episode 0... do
5: Sample initial state
6: for = do
7:
Gr = = 9c)
8:
Execute ag obtain next state Se+1
9:
Store transition (st, T,,T (e) in B
10:
Sample a minibatch 6 from replay buffer B
11:
Ge
12: Find ge that maximizes w in Eq. (12)
13: Update Q and with b minimize LQ and L, in Eq. ① and in Eq. 22
14:
Update the AIM reward function rip
15: success
Success rate
16:
Sample a desired goal ~ G
for 1... Tetstroloout do 18: at = (Tt,g
Execute de obtain next state 8141 and reward r.
20: if ((((s)))) gal S k then
success success 1/Teestrollout
5
Experiments
To evaluate our proposed method, we conducted experiments across three maze environments simulated in MuJoCq") PointUMaze, PointNMaze, and PointSpiralMaze. In these environments, goal is interpreted the (z, 3) position of the agent achieved in an episode. These environments have been specifically chosen due to their structural characteristics, which are ideal for testing environment-agnostic curriculum generation strategies. Moreover, they present a variety of complex and diverse navigation challenges, requiring an agent to learn effective exploration and exploitation.
We compared our approach DICURL against nine state-of-the-art CRL algorithms, namely ACL (18). GOAL-GAN [14, HGG [19] ALP-GMM [17), VDS [66. PLR (15), CURROT [23], GRADI-
ENT 13 and OUTPACE [2]. each one with 5 different random seeds. The primary conceptual
differences between ours and baseline algorithms are summarized in Table Details on the baseline
algorithms, parametrization, training setup, and maze environments are given in Supp. Mat. C Our
codebase is available at:
Details available htthtt:///obbbtt farama. wttp:///w//////////ottttoom

---

The results, shown in Fig. ii and detailed in Table demonstrate the effectiveness of the proposed DICURL method. Notably, DICURL outperforms or matches all the baseline methods. For the PointNMaze and PointSpiralMaze, the success rate of all methods, except for ours and OUTPACE (and HGG for PointNMaaee,, is close to zero (detailed results are reported in Supp. Mat. C ACI., GoalGAN, ALP-GMM, VDS, and PLR, which lack awareness of the target distribution, underper- form compared to target-aware methods such as our proposed approach, OUTPACE, GRADIENT, CURROT, and HGG. HGG encounters difficulties because of infeasible curriculum proposals. This a result of its reliance on the Euclidean distance metric.
Both CURROT and GRADIENT generate curriculum goals using the optimal transport method and the Wasserstein distance metric, relying on the geometry of the environment. This dependence might be the reason for their inconsistent and poor performance across the different environments.
OUTPACE, utilizes instead the Wasserstein distance and uncertainty classifier for uncertainty- aware CRL, exhibiting similar performance to our approach but with slower convergence and higher variance in success rate. This is likely due the fact that OUTPACE's curriculum heavily relies on the visited state distributions, which necessitate an initial exploration by the agent. While also our approach depends on visited states, incorporating the Qand AIM reward functions into curriculum generation facilitates exploration beyond them, potentially explaining the performance differences between DICURL and OUTPACE.
Lastly, we recall that our approach generates curriculum goals based on the reverse diffusion process, which allows the generated curriculum goals to gradually shift from the initial state distribution the desired goal distribution. Fig. 2 shows an example of a curriculum set generated by DICURL. I., for the case of the PointUMaze environment, at each iteration of the reverse diffusion process. Fig. B instead, shows the differences between the curriculum goals generated by DICURL, GRADIENT and HGG in the PointSpiralMaze environment: can be seen how DICURL manages to generate curriculum goals that explore the whole environment more effectively than the baseline algorithms. Supp. Mat. shows the curriculum goals generated in the other two environments and illustrates the dynamics of the diffusion process during training for the case of PointUMaze. To demonstrate the applicability of DICURL robot manipulation tasks, we evaluated it on the FetchPush and FetchPickAndPlace tasks using sparse reward setting, Further details can be found in Supp. Mat.
Table 1: Comparison of DICURL with previous CRL methods from the literature (sorted by year).
Curriculum method
Target dist. curriculum
Geometry- agnostic
Off- policy
External reward
Algorithm
Venue, year
ACL 18
LSTM GAN
X
X
X
PMLR, 2017
GoalGAN
X
X
X
PMLR, 2018
HGG 19
Q. 13, W22
G
X
V
X
NeurIPS, 2019
ALP-GMM
GMM
X
V
X
PMIR, 2020
VDS [6
Q B
X
V
X
NeurIPS, 2020
PLR 15
TD-Error
X
X
X
PMLR, 2021
CURROT 23
W2
g+_ U
X
PMLR, 2022
GRADIENT 13
W2
G
X
NeurIPS, 2022
OUTPACE 12
CNML.
g+
ICIRR, 2023
DICURL (Ours)
Diffusion
g+
Ablation Study We conducted ablation study investigate the impact of the AIM reward function To and Qo function in generating curriculum goals with our method (DICURL), For that, we omitted, separately, the reward function Tip and the Qs function from Eq. IT and plotted the success rate (with three different seeds) in Fig. 4a for the most challenging maze environment, PointSpiralMaze. The results indicate that the agent performs worse without the AIM reward function r'p and fails to achieve the task without the Qo function. The generated curriculum goals without the T or Q& function are shown in Fig. 4b and 4c. respectively. Fig. 4d instead, illustrates the AIM reward value across different training episodes in a clockwise direction. Specifically, the first row and first column
'A simplified version of the same figure shown Supp. Mat.
'We omit baselines unable to reach a success rate of 1.0 within the allotted timesteps on any of the environments.

---

0.8
0.8
0.8
Suczee Rolc
Succe Rulco-RRlcc
0.4
0.4
0.4
0.0
0.0
0.0
Timesstops
Timestops
x10
Timnsstas
x10"
(a) PointUMaze
(b) PointNMaze
(c) PointSpiralMazee
Figure 1: Test success rate for the algorithms under comparison on the three maze tasks.
Table 2: No. of timesteps (rounded) to reach success rate 1.0 (average + std. dev, across 5 runs). NaN indicates that success rate 1.0 not reached within the maximum budget of 1e6 timesteps.
Algorithm
PointUMaze
PointNMaze
PointSpiralMaze
DICURL (Ours)
28333 3036
108428 34718
305833
43225
OUTPACE
29800 + 4166
113333 + 24267
396875 + 111451
HGG
48750 + 24314
NaN
NaN
GRADIENT
263431 + 114795
NaN
NaN
Figure 2: The curriculum goal set Ge generated by DICURL during the reverse diffusion process (lines 7 in Algorithm[[] for the PointUMaze environment. The color indicates the goals generated specific iteration step during the reverse diffusion process of the diffusion model. Then these goals are selected in Eq. based on the given cost function.
in Fig. 4d represent the reward values at the very beginning of training. As training progresses, the reward values shift towards the left corner of the maze environment (Ist row, 2nd column). In the middle of training, the reward values are concentrated around the left corner of the maze environment (2nd row, 2nd column), and, by the end of training, the reward values converge the desired goal area (2nd row, 1st column). This progression explains why the generated curriculum goals are not guiding the agent effectively but are instead distributed in the corner points shown Fig. 44c We have also demonstrated the behavior of the AIM reward function across different training episodes in our Supp. Mat. for the PointUMaze environment. Additionally, SAC with a fixed initial state [0, 0] and SAC with random initial state. To do that, we removed the curriculum goal generation mechanism and assigned the desired goal, and then trained the agent using either SAC with a fixed initial state j0, 0] SAC with a random initial state. For the random initial state, we sampled goals randomly in the environment. To avoid starting the agent inside the maze walls, we performed an infeasibility check, resampling the initial state until it was feasible. We compared our approach using three different seeds with both the fixed initial state + SAC and the random initial state + SAC across all maze environments, and the success rates shown in Fig.
examined the impact of

---

3+322
E
22430
12
(a) DICURL
OUTPACE
(c) GRADIENT
Figure 3: Curriculum goals generated by DICURL, GRADIENT, and HGG in the PointSpiralMan environment. The colors ranging from red to purple indicate the curriculum goals across different episodes of the training, and the orange dot and red dot are the agent and desired goal, respectively.
MA
(a)
(b)
(d)
Figure 4: (a) Test success rate the ablation study DiCuRL on the PointSpiralMaze environment. (b) Curriculum goals when using only Qs. (c) Curriculum goals when using only the AIM reward rip-
(d) AIM reward ri across different training episodes (around 60k, 100k, 250k and 600k timesteps), displayed clockwise from the top-left to the bottom-left quadrant.
10
Stappss Rac
Ruee
06-
0.-
SCC
80
0.6
0.2
04
03
2.4
0.5
Timesdops
K10
Timestape
Timosiope
(a) PoineUMaze
(b) PointNMaze
(c) PointSpiralMaaze
comparison between DiCuRL, SAC with random initial position, and SAC with fixed initial position. Note that the success rate for the PointUMaze environment in Fig. 5a is shown up timestep 105, whereas the others shown up 105.
Figure 5: Test
6 Conclusion and Limitations
In this work, we introduced DICURL, novel approach that utilizes diffusion models generate curriculum goals for an RL agent. The diffusion model is trained to minimize its loss function while simultaneously maximizing the expected value of Q and the AIM reward function. The minimization of the diffusion loss helps capture the visited state distribution, while the maximization of Q and AIM reward function helps generate goals an appropriate difficulty level and the same time guide the generated curriculum goals closer to the desired goal. Furthermore, the generated goals promote exploration due the inherent noising and denoising mechanism of the diffusion model. Our proposed approach has two main limitations. First, while diffusion models excel at handling high- dimensional data, such as images [53], incorporating the AIM reward into a combined loss function can hinder the curriculum goal generation in such settings the AIM reward may underperform higher dimensionalities. Secondly, we employ the Minimum Cost Maximum Flow algorithm to select the optimal curriculum goals from the set generated by the diffusion model. However, alternative selection strategies could potentially be more effective. Future work will aim address these limitations and extend our approach to more complex environments.
10

---

7 Acknowledgments
This study was supported by the ChhoChaggi project, which is funded by the Bundesministerimm für Wirtschaft und Klimaschutz (BMWK) of Germany under funding number 01M220002C. Additionally, this work was supported by TUBITAK under the 2232 program, project number 121C148 ("LiRA")). We would like to express our gratitude Arda Sarp Yenicesu for his assistance in writing this paper.
References
[1] Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Graves, Alex and Antonoglou, Ioannis and Wierstra, Daan and Riedmiller, Martin. Playing Atari with deep reinforcement learning, 2013. arXiv: 112.56....
[2] Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Rusu, Andrei and Veness, Joel and Bellemare, Mare G and Graves, Alex and Riedmiller, Martin and Fidjeland, Andreas K and Ostrovski, Georg and others. Human-level control through deep reinforcement learning. Nature, 518:529-555 533, 2015.
[3] Silver, David and Hubert, Thomas and Schrittwieser, Julian and Antonoglou, Ioannis and Lai, Matthew and Guez, Arthur and Lanctot, Marc and Sifre, Laurent and Kumaran, Dharshan and Graepel, Thore and others. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419): 111:44199::1141111 1100-114444444 2018.
[4] Silver, David and Huang, Aja and Maddison, Chris J and Guez. Arthur and Sifre, Laurent and Van Den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershel- vam, Veda and Lanctot, Marc and others. Mastering the game of Go with deep neural networks and tree search. Nature, 529:484 489, 2016.
[5] Rajeswaran, Aravind and Lowrey, Kendall and Todorov, Emanuel V and Kakade, Sham M. Towards generalization and simplicity in continuous control. In Advances in Neural Information Processing Systems, volume 30, San Diego, CA, USA, 2017. Neural Information Processing Systems Foundation.
[6] Ng, Andrew Y and Coates, Adam and Diel, Mark and Ganapathi, Varun and Schulte, Jamie and Tse, Ben and Berger, Eric and Liang, Eric. Autonomous inverted helicopter flight via reinforcement learning. International Symposium on Experimental Robotics, pages 363-372, Berlin Heidelberg, Germany, 2006. Springer.
[7] Timothy Lillicrap and Jonathan Hunt and Alexander Pritzel and Nicolas Heess and Tom Erez and Yuval Tassa and David Silver and Daan Wierstra. Continuous control with deep reinforcement learning, 2019. arXiv: XXXi:1509.00271..
[8] Sayar, Erdi and Bing, Zhenshan and D' D'Eramo, Carlo and Oguz, Ozgur S and Knoll, Alois. Contact Energy Based Hindsight Experience Prioritization, 2023. arXiv:2312.02677.
[9] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transfor- mations: Theory and application to reward shaping. In International Conference Machine Learning, volume 99, pages 278-287. Citeseer, 1999.
[10] Rui Zhao and Volker Tresp. Eaergy-based hindsight experience prioritization. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto, editors, Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 113-122. PMLR, 29-31 Oct 2018.
[11] Rui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy-regus regroay-regularrized multi-goal rein- forcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7553-7562. PMLR, 09-15 Jun 2019.
[12] Daesol Cho, Seungjae Lee, and H. Jin Kim. Outcome-directed reinforcement learning by uncertainty temporal distance-aware curriculum goal generation. International Conference on Learning Representations. OpenReview.net. 2023.

---

[13] Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and DING ZHAO. Curriculum reinforcement learning using optimal transport via gradual domain adaptation. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 10656-10670. Curran Associates, Inc., 2022.
[14] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning, pages 1515-1528. PMLR, 2018.
[15] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Marina Meila and Tong Zhang, editors, International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4940-4950. PMLR, 18-24 Jul 2021.
[16] Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7648-7659, Curran Associates, Inc., 2020.
[17] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yves Oudeyer. Teacher algorithms for curriculum learning of deep in continuously parameterized environments. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 835-853. PMLR, 30 Oct-01 Nov 2020.
[18] Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. Doina Precup and Yee Whye Teh, editors, International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1311-1320. PMLR, 06-11 Aug 2017.
[19] Ren, Zhizhou and Dong, Kefan and Zhou, Yuan and Liu, Qiang and Peng. Jian. Exploration via Hindsight Goal Generation. arXiv:1906.044: 1906.0479.
[20] Pascal Klink, Hany Abdulsamad, Boris Belousov, Carlo D) D'Eramo, Jan Peters, and Joni Pajarinen. probabilistic interpretation of self-paced learning with applications to reinforcement learning. Journal of Machine Learning Research, 22(182):1-3 2021.
[21] Pascal Klink, Carlo D' Eramo, Jan R Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9216-9223 Curran Associates, Inc., 2020.
[22] Pascal Klink, Hany Abdulsamad, Boris Belousov, and Jan Peters. Self-paced contextual reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 513-529. PMLR, 30 Oct-01 Nov 2020.
[23] Pascal Klink, Haoyi Yang. Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum rein- forcement learning via constrained optimal transport. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 11341-11358. PMLR, 17-23 Jul 2022.
[24] Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mural: Meta-learning uncertainty-awart aware rewards for outcome- driven reinforcement learning. In Marina Meila and Tong Zhang, editors, International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 13346-6356 PMIR, 18-24 Jul 2021.
[25] Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guided hindsight experience replay. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
12

---

[26] Zhenshan Bing, Matthias Brucker, Fabrice O. Morin, Rui Li, Xiaojie Su, Kai Huang, and Alois Knoll. Complex robotic manipulation via graph-based hindsight goal generation. IEEE Transactions on Neural Networks and Learning Systems, 33(12):7863-777, 2022.
[27] Zhenshan Bing, Hongkuan Zhou, Rui Li, Xiaojie Su, Fabrice O. Morin, Kai Huang, and Alois Knoll. Solving robotic manipulation with sparse reward reinforcement learning via graph-based diversity and proximity, IEEE Transactions on Industrial Electronics, 70(3):2759-22 2769,, 2023.
[28] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepes- vari, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 9902 9902-9915, PMLR, 17-23 Jul 2022.
[29] Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchi- cal planning with diffusion, 2024, arXiv:2401.024 0.26444.
[30] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies an expressive policy class for offline reinforcement learning, 2022. utXiv:2208.6111
[31] Ishan Durugkar, Mauricio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and
J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8622-8636. Curran Associates, Inc., 2021.
[32] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In IEEE/RSJ International Conference on Intellgent Robots and Systems, pages 5026-5033. IEEE, 2012.
[33] Sanmit Narvekar, Bei Peng. Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181): 2020.
[34] Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International Conference Machine Learning, pages 4344 4353. PMLR, 2018
[35] Tim Hertweck, Martin Riedmiller, Michael Bloesch, Jost Tobias Springenberg, Noah Siegel, Markus Wulfmeier, Roland Hafner, and Nicolas Heess. Simple sensor intentions for exploration, 2020. arXiv:2005.07541.
[36] Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine Visual reinforcement learning with imagined goals. Advances in Neural Informatin Processing Systems, 31, 2018.
[37] Dmytro Korenkevych, A Rupam Mahmood, Gautham Vasan, and James Bergstra. Autoregres- sive policies for continuous control deep reinforcement learning, 2019. 1903v:1903.1133 15224.
[38] Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017.
[39] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z.. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[40] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In Francis Bach and David Blei, editors, International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1312-1320, Lille, France, 07-09 Jul 2015. PMLR.
13

---

[41] Chelsea Finn, Pieter Abbeel, and Sergey Levine. bodel-aggostic meta-learning for fast adapta- tion of deep networks. Doina Precup and Yee Whye Teh, editors, International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1 1126-11355 PMLR, 06-11 Aug 2017.
[42] Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
[43] Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning to act from actionless videos through dense correspondences. 2023, arXiv:2310.8857.
[44] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjja Xu, Eric Cousineau, Benjamin Burchtiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion, 2023. arXiv:2303.04137.
[45] Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies, 2023. arXiv:2304.02532.
[46] Hsiang-Chum Wang. Shang-Fu Chen, Ming-Hao Hsu, Chun-Mao Lai, and Shao-Hua Sun. Diffusion model-augmented behavioral cloning, 2023. MtXiv:2302.13335
[47] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models, 2023. arXiv:2301.10677
[48] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870. PMLR, 10-15 Jul 2018.
[49] Villani, Cédric and others. Optimal transport: old and new, volume 338. Springer, Berlin Heidelberg, Germany, 2009.
[50] Arjovsky, Martin and Chintala, Soumith and Bottou, Léon. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pages 214-223, Sydney, Australia, 2017. PMLR.
[51] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1840-6851. Curran Associates, Inc., 2020.
[52] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: review for statisticians. Journal of the American Statistical Association, 1 112 877, April 2017.
[53] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusson models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[54] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 0256-2265 Lille, France, 07-09 Jul 2015. PMLR.
[55]
Calvin Luo. Understanding diffusion models: A unified perspective, 2022. arXiv:2208.1197
[56] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017, arXiv: 1e.:110070074777
14

---

Supplemental Material
A
Conditional Diffusion Model
Diffusion models 51 54 are usually defined as Markov chains and trained using variational inference.
The forward diffusion process gradually adds Gaussian noise the data, while the reverse diffusion process subtracts the learned noise from the data.
Given the goal data So sampled from the real goal distribution go G. we can define the forward diffusion process using Markov chains and add small amount of Gaussian noise each timestep k according to variance schedule (.3x (0, ==)==1 and generate sequence of noisy goal data samples g1:.... g.N:
N II
(gi:
go) =
(gk gk-1)
(13)
k=1
q (gs gk = NV
Bki
V
BI. 1,,
As the index of the timestep k increases, the goal sample go gradually loses its features and becomes an isotropic Gaussian noise when N + 20.
Instead of sampling data recursively for given timestep k, we can sample in a closed form using the reparameterization trick
gk N gk: V 1,
(14)
Let Ork - Bg and 0g
II Ug.. We can write:
gk = Va -
CxCk-1
(15)
Vagak- 1gk VI CGEK-1 + Var(1- ax-1)Ek-2
(16)
We then merge two Gaussian distributions with different variance (0, (f) and N oΣ) into new Gaussian distribution (000 of + o²). Specifically, we can merge the two equations above, where
VI O% + +1 Ok VT 04.0--1::
gk aa
aAak-1Čk-2
(17)
N
N
8k
Orggo
1
ag
(18)
1=1
where ~ N (0,
Now, we can write:
gk = aggo + vI - q (ge go) N (gx: Vorgo. (1 T)
(19)
(20)
Using the equation above, we can sample data in closed form at any arbitrary timestep k.
As mentioned in the main text, the reverse diffusion process aims recover the original input data from the noisy (diffused) data. It learns to progressively reverse the diffusion process step by step, and approximates the joint distribution pe (Fo.r). This process is defined as:
Pa (go.N) = p (gN) II pe (ge gx)
(21)
po (gx gx) ./ (gk-1: M4 (gi, k)) Eo (gt, k))
'The reparameterization trick works by separating the deterministic and the stochastic parts of the sampling operation. Instead of directly sampling from the distribution ~ N' (H, a). we can sample from the normal distribution (0, 1). multiply it by standard deviation a, and add mean μi: μ
15

---

The reverse conditional probability is tractable when conditioned by gº.
By Bayes rule, we have:
9 (gk
gk
8o
(gc-1
go)
q (g-- Bk go)
(22)
(gk
g0)
As assume that the diffusion model is Markov chain, the future state depends only the present state, namely:
q gg
1k-1; 1: go) q (gk gk- = N (gk: vaRgk-1, ax)
(23)
(gk-1
go) = N (gx- V82-18o (1 (1)
(24)
9 [gk]
go) = V (gx: vāggo, (1 ax) I)
(25)
N (gki v dagk- as H)N (gK-15 Vāl- (1 ax-
9 (gk-1
Bk: go)
(26)
N (gai /0:80 (1-āz)
where:
1
( 1
I
N (gei vargt-i. (1 (1-)))))))))))
exp
(- (gk
(aagk-1) 1)
(gi vasgat-1)
(11 (2x)"/² n/2
okk 111/2
(1- a1.) (1-a))
(27)
/ 00 1) (gh
x exp
(28)
2(1 0/k I
1
N (84-11 )z-
(2T) 1(1 0zg 1) 1 exp
(
[Bc
Vou 180)
(1 - 0g. 1)
[gi
Vōx 180)
(29)
(Bk-1 v vak-1gu)
2s exp
(30)
2(1 ax-
I
N (Bri Vāngo. (1 (1- (1-x))1)
(2x)) n// 111 0x. )
exp
[gk
vāngo)) T
(1 (g)
(gk vango)
(31)
2d exp (-(B)
(Bk
v /0
(32)
2(1 (xa
We then can write:
gg / varga-
1) +
(gk (0k-1gn)
(gk
04
q (gk-1
gk: Bo)
exp
21
(1k)
1 ak-1
(33)
(gi V (arge-1)
2 (gk -8g)
)
(gk v O go)
exp
21
(1 a4k)
1 a
(34)
1
(s) Vagga88 aggg
(R) 22/
11B- 18t-180
180
.gg 18
(Ec
v
exp
21
(1-ag) at)
1 06--
(35)
1-2VaggiBr + aggi
(g)
a2. 2
1-4--11-h 1
exp
+ C (gk+ go)
21
-w)))
Ōk-1
(36)
gk
2 a
gg 180
R exp
+
21
(1 0.k)
0kk
1 σ4
1 aK-1
(37)
16

---

1
2
valgg
V VOk-180
exp
21 11
8k-1
gk--1
1 - Cx-1
1-ak akk
1
(38)
(1x(1 (x-11) + 1 - 0t
Vaagge
V Va
exp
gk-1 -
8k-1
21
(1 ax) -ak-1)
Oc
(39)
exp
1
O/ 04- āΦ + 1 - 04k
8k-1 -2
2
(
VauiBl
+
180
gk-1
21
(1 (1-aa)(( (1 -
1 akk
ak-1
(40)
1 āx
vakgk
Crp. g0
exp
8k -
gk-1
21
(1- a1) [1 az-1)
1- 0kk
(41)
G g
exp
1
gk- -
T-06
1-0
gk
21
ax) ãx-1)
1-0p
[1-aa]( 0k
(42)
axx (1 ak-1)
- σx
exp
21
(1--a))1 (1 (ix-1)
8h-1
1 Gg
gk-1
(43)
0k
1)
VOk
)oo
exp
21
1- (1-ax)(1-ae-1)
āk
gk
(44)
var(1
02 1/
gk / vak-11
(1-ax)(1 cx- I
N
gg-1;
O%k
1 - Ö
(()
xT(k)
(45)
where C (gi,go) Bo) is constant term with respect g and is calculated using only gk: go, and oxk The Eq. 45 shows that at every step gk-1 ~ q (gx-1 ge go) follows a normal distribution with mean plgx. go) and variance (k)))
Given this, the goal is learn denoising model po (gt-1 go) by approximating the ground-truth denoising transition (gk-1 gk: go). This can be done by minimizing the KL divergence between pe (g--1 gx) and (gk-1 gk gu).
The KL divergence between two Gaussian distributions. N N(x;; May Ex) and ( My: Eny) is defined
Σg d + tr(2 E.)) + (My - " (My - Mw)
DKL (N Has Ex) N My: nn)) = 2
log Σn
(46)
The KI. divergence between the denoising model po (gk-1 gx) and the ground-truth denoising transition y (gt-1 gt, Bo) is:
arg min DKL (g (gk-1 gi: go) pa (gx-1 [gx))
(47)
The learning of the denoising model can be written as:
arg min DKL (N(gk- (( go), Σ) [[ Σ))
(48)
8
1
Σ
arg min log
+ tr( + (Mg p(gn, go)) (us - u(gu.
(49)
0 2
Σ
17

---

1
arg min a 2
[log 1 - + d + (1o (1gk: g0))²'= " ((gt.
(50)
1
arg
[(MB plg 2o (mo ((Br:go))]
(51)
0 2
1
arg min
[ ul go)) (I) (uo u(gr.go))
(52)
0 2
|llue nlge, 80)1 2
1
arg min
(53)
We can rearrange Eq. 19 follows:
gk vI d.e
(54)
Vax
and rewrite u(gk, Bo) in Eq. (45) as follows:
Mg Bu) vax(1 1)gh + vāk-i(1 axs)
RR- RR-y Von
(55)
1 āk
where ā% = 11 ag
Then:
Var((1 ((k-))g + Vu102 ... 0ak-1(1 01k)
Sk - 11-122 &gt
(B
Vaiaz...ar
(56)
1 0kk
varl (1
σk-1 gg
(1 - (x) Bk
- (1-az) VI Cre
(57)
1 āk
(1 ā+) yark
(1 04)) V
voue( (1
1x-1)g1
(1 - ax) gk
(1 (1- ax) v1 CAE
(58)
Oek
(1 ax) Vou
(1 (xx) Vark
v aak (1 ax-11)
(1 - an)
- 0%k) vI akk
+
gk
(59)
āx
(1 āx) yan
(1 ax)) ) VOt
0 (1 - k)-1)
- 0k
(1 - 04k)
gk
(60)
(x)) / /ax
(1 āx) Vol
āx yakk
ak āy + akk
1-0k
gk
(61)
ax) vot
VI ākya 0
1- -xk
(62)
0gg
ork /ak
We can model the denoising transition mean as follows:
1- Ork
Me (gk, k)
gA
cn (B k)
(63)
Vog
VI a
Given this, the minimization of the KL divergence can be written as:
arg min DKI. (q (gt-1 1 gk: go) Pa (gi-1 |gx)) 8
(64)
arg min DEL (N(g: H(gk go). Σ) || |[ E))
(65)
1 ak
1- akk
arg min
Bk
Eo(ge, k))
Bk
(66)
VI
akvak
Var
VI - āx akk
1 Ork
1 Oak
arg min
cD (g k)
(67)
0
2
az /ok
VI - ak
2.
1 (1 a1)
arg min
[||e - co(ggr =1)1
(68)
2 (1 āg )ak
18

---

Thus far, we have only modeled the goal distribution (g). To transform this into a conditional diffusion model, we can incorporate the state information each diffusion timestep [55] in Eq. (T
T
pe (go:N) == p (gN) II
Pe (gk-1 g&s)
(69)
k=1
Pa (gk-1 gk) ;= N (gx-1i (gk. 8, k) Eg (gk, k,)
To conclude, model the denoising transition mean by rewriting Eq. (70) based on the state as follows:
1 1 ark
μg (gk, k,k
gh
s,
(70)
v Vaz
v1 02
B ST Strategy
In contrast to selecting optimal curriculum goals via bipartite graph optimization, shown in Eq. 12 in the main tex we performed an additional experiment where we considered only the state from the final timestep T. under the assumption that is closer to achieving the desired goal. The state sT at the last timestep of each episode is then input into the diffusion model, which generates curriculum goals to be achieved in the subsequent episode. The resulting variant detailed Algorithm with the RL training process described in Algorithm 4
As demonstrated by this additional experiment, the assumption that the final state is closer to the desired goal does not always hold true. As a matter of fact, the state at the last timestep in some episodes might be even further away from the desired goal compared the states in previous timesteps. Consequently, the curriculum goals generated by the diffusion model may not gradually shift from the initial position toward the desired goal. If the state the last timestep not progressively moving towards the desired goal, the curriculum goals generated by the diffusion model may jump to different areas within the maze environment. This can lead to a decrease in sample efficiency and slower success rate. This issue is illustrated in Fig. 6 where it can be seen, that particularly in the case of PointSpiralMaze, the variant of our algorithm that makes use of this strategy (that we call "sr strategy") converges later than the main described in the main text.
1.0
1.0
1.0-
0.8
Suavon Rale
Suare Rate
0.6-
0.6
0.6
0.4
0.4
0.4
0.0
0.0
DCCuR by
0.0
bchut by
0.4
0.6
0.2
0.4
0.0
0.2
0.8
1.0
Timestops
x10
Timestops
x10
Timestops
x10"
(a) PointUMaze
(b) PointNMaze
(c) PointSpirallMaze
Figure 6: Test success rate for the original DICURL algorithm described in the main text compared variant that makes of the 8r strategy.
"As a reminder, the proposed algorithm sample mini-batch (containing many states from different timesteps) from the replay buffer B and provide it to the curriculum goal generator (i.c., the diffusion model) to generate curriculum goal distribution G., and then select the optimal curriculum goal using bipartite graph optimization by maximizing the cost function given in Eq. 12
19

---

Algorithm 3 Diffusion Curriculum Goal Generator with the sT Strategy
1: Input: no. of reverse diffusion timestep N, training step M
2: for I,.... .M do
Training iterations
3: gN ~ (0, I)
4:
for m= N. ...... do
Reverse diffusion process
5:
<~ (0,
6:
9k-1 You k~ Uniform((11..... N])
Be Ok aT
colon, (T, + BBEE
Di using Eq.
7:
8:
I)
9:
Calculate diffuusin C.( go,
go, r(s)) Calculate with the generated goal go
10:
Calculate total loss L = (aCa(O) (s, go, (s)) (gr
Eq
11:
3--6 0 VoL
b Take gradient descent step and update the diffusion weights
return go
Algorithm 4 RL Training with the Strategy
1: Input: no. of episodes E, timesteps '
2: Select an off-policy algorithm
In our case, A is SAC
3: Initialize replay buffer B t- 0. + 9a and networks Qş. Tu, r.
4: for episode- -..... do
5: Sample initial state
6: for 0.. do
7:
0r (c), Execute dz obtain next state 8c+1
8:
9:
Store transition (st, Tr (e) in B
10: Sample a minibatch from replay buffer B
11:
ge f Difffusinnuurpp cul ummeneratoor($7)
12: Update Q and π with to minimize LQ and C in Eq.
and in Eq. 2
13: Update the AIM reward function N'p
14: success +-
D Success Rate
15: Sample desired goal Od
16: for 1... Mestrrllont do
17:
at IT( 9a)
18:
Execute Or obtain next state 8141 and reward rz
if' ([((0)+)) gal S k then
20:
success = success + I/Mtsttollout
C Experimental Details
For implementing DICURL, we utilized the original implementation of OUTPACE 12 and aug- mented this codebase with our diffusion model for curriculum goal generation. We conducted our experiments cluster computer using an NVIDIA RTX A5000 GPU, 64GB of RAM, and 4-core CPU, For the PointSpiralMaze environment, the total compute time of the main method and the sy strategy is approximately 23 hours and 5 hours, respectively. For the remaining environments, the main method and the ST strategy require approximately 11.5 hours and 2.3 hours, respectively.
Baselines.
The baseline CRL algorithms are trained as follows:
HGG [9]: We utilized the default settings from the original implementation, which can be found at ptttps://githube camtub..ommf5 www/Stilweeeeeeo..omoo.t... Git/Hindsight htt::iiiniiggggo-olaoo-ooooa Geneeaain
CURROT 23
We adhered the default settings of the original implementation, available https:://iithb. cow http///sckknnk/c/c/ currot
ALP-GMM 17, VDS 16,
15 ACL [8]. GoalGAN 14 We followed the default settings
from the implementation available at https // h/gggttbbb...t//////////e wwtp//sccc.i///curuooo.ooooo
20

---

GRADIENT 13:
We used the default settings from the implementation available at https
//github. wwwm http//eeeeeeuunggrriiii....
We implemented the default settings from the implementation available at https: thub. www.ittu...o.o.eeeeeeeeeeeeeeeee cot.//jayLEEO3011/otttpeeae oftpacc_ooficciall
OUTPACE [2
Training details. All baseline models, except for the GRADIENT method, were trained using the Soft Actor-Critic (SAC) [48]. Although the original implementations of these algorithms primarily use on-policy methods, adaptations were made in the CURROT repository to utilize the off-policy SAC algorithm. This modification, available in the CURROT repository, allows for
direct comparison of sample efficiency across all models. The GRADIENT method was trained using both the SAC and Proximal Policy Optimization (PPO) algorithms (56). However, we present only the results obtained with the PPO algorithm, as it outperforms SAC in the three considered maze environments, All our algorithms' hyperparameters used in the experiments are reported in Table 3
Table 3: Hyperparameters for DICURL.
Parameter
Value
Parameter
Value
Critic hidden dim
512
Discount factor "r
0.99
Critic hidden depth
3
r'e Update frequency
1000
Critic target 7
0.01
No. of gradient steps for rip update
10
Critic target update frequency
No. ensemble networks for Tip
5
Actor hidden dim
512
learning rate for Tip
le-4
Actor hidden depth
RL optimizer
Adam
Actor update frequency
2
Diffusion network updates per episode
25
RL batch size
512
Diffusion loss coefficient &a
I
Init. temperature Qinis of SAC
0.3
Q-function coefficient E,
10
Replay buffer size
1e6
AIM reward function coefficient &
1
Diffusion training iterations
300
Reverse diffusion timesteps
10
Diffusion learning rate Diffusion update frequency
3e-4
Diffusion loss type
2500
No. of training timesteps
le6
Environment details. In each task, the agent's state is represented vector, comprising its position, velocity along the x and axes, orientation angle, and angular velocity around the 2-axis. The agent's 's actions are determined by its velocity and angular velocity. The agent starts each episode from an initial state of [0, 0j. The desired goal distribution is derived by introducing uniform noise the desired goal position. The desired goal positions are set [0, 8). [8, 16). and [8, -8), while the dimensions of the map are 12 x 12. 12 x 20, 20 x 20, respectively for the PointUMaze, the PointNMaze, and the PointSpiralMaze task. A task is considered successful when the agent reaches the sampled desired goal within a threshold distance of 0.5.
Detailed results. We report in Table the detailed results in terms of no. timesteps needed to reach success rate of 0.3 for all the algorithms under comparison. It can be seen that most of the algorithms do not reach this rate within the given budget.
D Generated Curriculum Goals
Fig. and Fig. g show the differences between the curriculum goals generated by DICURL, OUTPACE, GRADIENT, and HGG. respectively in the PointUMaze and PointNMaze environment. As shown in the case of PointSprrllMaee reported in the main text, also these two environments DICURL manages generate curriculum goals that explore the whole environment more effectively than GRADIENT and HGG.
D.1 Dynamics of the Diffusion Process during Training
Considering the case of the PointUMaze environment, the curriculum goal sets, denoted as g9, ....... ... go in Fig. I0 generated during the initial phase of training (~~2000 timesteps),
21

---

Table 4: No. of timesteps (rounded) reach success rate 0.3 (average + std. dev. across runs). NaN indicates that success rate of 0.3 is not reached within the maximum budget of 1eó timesteps.
Algorithm
MaintUMaze
PointNMaze
PointSpiralMaze
DICURL (Ours)
25000 + 3958
99000 + 32031
305 + 43225
OUTPACE
27600 = 4176
106666 25603
396875 = 111451
HGG
34750 + 14956
68000 = 15000.0
NaN
GRADIENT
166986 + 86132
755573 65967
NaN
CURROT
700000 + 234520
NaN
NaN
ALP-GMM
660000 + 205912
NaN
NaN
GoalGAN
960000 + 290516
NaN
NaN
PLR
833333 + 253859
NaN
NaN
VDS
838461 + 294927
NaN
NaN
ACL
1000000 + 316227
NaN
NaN
3X10
3X10
SMPE
13
Time
2
2x
DICURL
OUTPACE
GRADIENT
(d) HGG
Figure 7: Curriculum goals generated by DICURL, OUTPACE, GRADIENT, and HGG in the PointUMaze environment. The colors ranging from red to purple indicate the curriculum goals across different episodes of the training, and the orange dot and red dot are the agent and desired goal, respectively.
obtained through the reverse diffusion process. This process starts from Gaussian noise go and culminates in the final curriculum goal set go. expressed in lines lines 5nd in Algorithm It can be observed that the final curriculum goal set go is generated diagonally. This pattern arises due to the combined loss function of the AIM reward and the Q-value, along with the diffusion loss, as expressed in Eq. ITP.
The AIM reward function, relative to the desired goal ga. and the Q-value, derived from a sampled mini-batch from the replay buffer B at timestep ~ 2000, are depicted Fig. [1a and Fig. [Tb respectively, using 10-level contour plot. It can be observed that the reward values increase diagonally, while the Q-value not yet shown discernible pattern. goal set generated by the diffusion model exhibits pattern similar to the reward value.
result, the curriculum
22

---

18
8x 10
6X
Time Steps
109
Time Steps
2x 102
(a) DICURL
(b) OUTPACE
8x100
Time Stops
Steps
GRADIENT
(d) HGG
Figure 8: Curriculum goals generated by DICURL, OUTPACE, GRADIENT, and HGG in the PointNMaze environment. The colors ranging from red to purple indicate the curriculum goals across different episodes of the training, and the orange dot and red dot are the agent and desired goal, respectively.
In the mid- mid-training stage (~15000 timesteps), the final curriculum goal set reflects the explored region of the environment, extending up the top right corner, shown in Fig. 12 This behavior aligns with the agent's exploration progress.
The AIM reward function in Fig. exhhibiss higher and better-converged values compared to Fig. Iaa particularly in the top right corner. Similarly, the Q-value in Fig. [13b] demonstratsss its highest values concentrated from the middle right side the top right corner. These observations suggest that the final curriculum goal set go has indeed been formed, encompassing the explored area up to the top right corner.
In the ncar-optimal policy learning phase (~ 30000 timesteps), the final curriculum goal set go encompasses the entire environment, extending from the initial position to the desired goal, as depicted Fig. 14 This behavior aligns with the agent's progress towards achicving the optimal
policy.
The AIM reward function Fig. 15a appears to be converging to the desired goal more slowly. The Q-value in Fig. demonstrtss its highest values concentrated around the desired goal area. These observations suggest that final curriculum goal set go has indeed been formed, encompassing the entire environment. Furthermore, it appears that the Q-value converges to the desired goal more rapidly than the ATM reward function. This observation aligns with our choice of higher hyperparameter coefficient& &, for the Q-value function compared hyperparameter coefficient &
23

---

8X
6.10
00100
2X
(a) DICURL
(b) OUTPACE
82.30
6x
Time: Stpp
2x30
(c) GRADIENT
(d) HGG
Figure 9: Curriculum goals generated by DICURL, GRADIENT, and HGG in the PointSpiralMaze environment. The colors ranging from red to purple indicate the curriculum goals across different episodes of the training, and the orange dot and red dot are the agent and desired goal. respectively.
96
9i
91
Figure 10:
The curriculum goal set G. generated by DICURL during the reverse diffusion process in the early stage of the training (~2000 timesteps) for the PointUMaze environment.
for the reward function in the total loss function shown in Eq. (T). This prioritizes the Q-value's influence shaping the curriculum goals, promoting faster convergence towards the desired behavior.

---

0.5
000
-15
-015
(a) AIM reward
(b) Q-Value
Figure 11: Visualization of the AIM reward function and Q-function in the early stage of the training (~2000 timesteps) for the PointUMaze environment.
9
94
9:
Figure 12: The curriculum goal set Ge generated by DICURL during the reverse diffusion process in the middle stage of the training (~15000 timesteps) for the PointUMaze environment.
10
-06
(a) AIM reward
(b) Q-Value
Figure 13: Visualization of the AIM reward function and Q-function in the middle stage of the training (~~5000 timesteps) for the PointUMaze environment.
25

---

9s
9+
9,
91
Figure 14: The curriculum goal set 9. generated by DICURL during the reverse diffusion process in the near-optimal stage of the training (~30000 timesteps) for the PointUMaze environment.
-0
-05
24
-15
(a) AIM reward
(b) Q-Value
Figure 15: Visualization of the AIM reward function and Q Q-function in the near-optimal stage of the training (~330000 timesteps) for the PointUMaze environment.
E Robot Manipulation Tasks
To demonstrate the applicability of our method to different tasks, particularly in robot manipulation tasks, implemented approach using the original HGG algorithm [19] repository, which be found at //tttps://githu: com hiissreel-iitt/inniiihhh.. the HGG code from TensorFlow PyTorrch integrate it with diffusion model, which is based on Py'Trrch. We selected two robot manipulation tasks, FetchPush and FetchPickAndPlanr increased the environment difficulty by expanding the desired goal area. This is shown in Fig. [6c and [6d] where the yellow area indicates the object sampling region and the blue area indicates the desired goal sampling region. The action space four- Nour-dimensional, where three dimensions represent the Cartesian displacement of the end effector, and the last dimension controls the opening and closing of the gripper. The state space is 25-dimensional. including the end-effector position, position, and rotation of the object, the linear and angular velocity of the object, and the left and right gripper velocity. More detailed information regarding the action space and observation space of these robotic tasks can be found in the Gymnasium library documentation. For these additional experiments, we compared our method with HGG and HER using the DDPG algorithm, to ensure alignment with the baselines, using five different seeds. The results are shown in Fig. [6a and Fig. 16b] respectively for FetchPush and FetchPickAndPPan Note that in this setting, all RI. algorithms (including ours) binary reward (i.e., sparse reward). However, since our curriculum goal
We converted
and
26

---

generation algorithm is based on the AIM reward, we implemented the AIM reward function solely generate curriculum goals while still using the sparse reward setting to train the DDPG algorithm using the diffusion model to generate curriculum goals.
(a) FetchPush
(b) FetchPickAndPlace
(c) FetchPickAndPlace
(d) FetchPush
Figure 16: (a, b) Test success rate with solid lines indicating mean success rates and shaded areas variability over five seeds. (c, d) Overview of the two robot manipulation tasks. Blue and yellow regions denote the sampling areas for objects and goals, respectively.
Code available at: https///ghhaa..ooomcccchhllllllt manipulation
F
Test Success Rate in Maze Tasks (Simplified Visualization)
In Fig. in the main text, we presented the success rate of ten different algorithms (including ours) across the three maze tasks. To facilitate visual comparison of the algorithms performance, we provide in Fig. [7 simplified visualization that reduces the line overlap by showing only the four top-performing algorithms (including ours).
1.0
0.8
0.8
Succin Buie
Suprio Rdde
0.6
0.4
0.4
GIADENT
0.0
0.0
0.0
355
Timnestops
Tinnestops
X10
Timestops
(a) PointUMaze
(b) PointNMaze
(c) PointSpirallMaze
Figure 17: Test success rate for the algorithms under comparison on the three maze tasks.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section o
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All the theoretical results are detailed in Supp. Mat.
A
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all the experimental details along with our codebase.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: See Section 5 and Supp. Mat. C and
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Supp. Mat.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide all results terms of mean and standard deviation.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: See Supp. Mat.
9. Code Of Ethics
28

---

Question: Does the research conducted in the paper conform, in every respect. with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our research conforms with the NeurlPS Code of Ethics.
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We summarize the importance and impact of this research in the Introduction.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
29

---

Answer: [Yes]
Justification: The creators or original owners of assets (e.g., code, data, models) used in the paper are properly credited through appropriate citations or website URLs.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide code GitHHub (see Section 5 and Supp. Mat. E) well as all the details about training and computational setup (see Supp. Mat.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
30

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "UCnEZHqo5b", "reviewer": "Reviewer_c6He", "summary": "This work introduces DiCuRL, a novel approach that uses diffusion models to generate curriculum goals for reinforcement learning agents. The method trains a model to capture the distribution of visited states, focusing on those with higher Q-values and intrinsic motivation rewards (i.e., AIM rewards). This approach aims to generate goals at an appropriate difficulty level while guiding the curriculum closer to the desired final goal. DiCuRL employs the Minimum Cost Maximum Flow algorithm to solve a bipartite matching problem to select curriculum goals.", "questions": "- In Fig. 3, what do the colours represent? Please be more elaborate. It is not clear at all at the moment\n- In the appendix, it's mentioned that \"The agent starts each episode from an initial state of [0, 0].\" In RL environments, environmental resets can implicitly help exploration [1]. How would DiCuRL + fixed start state fare against SAC only + random start states?\n- How does SAC only perform in the comparisons in Fig. 1?\n- How important is the AIM reward? It is a bit weird to sum the Q value and one-step intrinsic motivation reward. This results in different scales/magnitudes of values, which is why the authors needed to tune the coefficients.\n- To ask the previous question differently, can the AIM reward be substituted with simpler intrinsic motivation rewards like RND [2] or TD-error?\n- It seems SAC + HER would be a lot simpler to use computationally and algorithmically. How does DiCuRL compare against SAC + HER?\n\n\n*References*\n1. Vasan, G., Wang, Y., Shahriar, F., Bergstra, J., Jagersand, M., & Mahmood, A. R. (2024). Revisiting Constant Negative Rewards for Goal-Reaching Tasks in Robot Learning. arXiv preprint arXiv:2407.00324.\n2. Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018). Exploration by random network distillation. arXiv preprint arXiv:1810.12894.", "limitations": "Please suggest how this work can be extended to challenging environments with larger state-action spaces.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Strong empirical evaluation against competitors (Fig. 1)\n- The paper is information-dense but reasonably well-written. It helps with the comprehension of the proposed ideas", "weaknesses": "- The approach is quite complicated and possibly unnecessarily so. I'd like to emphasize that I did not find any faults with the proposed method. It's just that I do not see how it will scale to more challenging, realistic environments.\n- They missed citing a rich literature on exploration and curriculum RL. For example, see papers [1-5].\n- The reward function for the Maze envs is not provided. Is this dense or sparse reward env? Note that, dense reward would not be a justifiable choice in this case.\n\n\n*References*\n1. Riedmiller, M., Hafner, R., Lampe, T., Neunert, M., Degrave, J., Wiele, T., Mnih, V., Heess, N., and Springenberg, J. T. (2018). Learning by playing solving sparse reward tasks from scratch. In International conference on machine learning, pages 4344–4353. PMLR.\n2. Hertweck, T., Riedmiller, M., Bloesch, M., Springenberg, J. T., Siegel, N., Wulfmeier, M., Hafner, R., and Heess, N. (2020). Simple sensor intentions for exploration. arXiv preprint arXiv:2005.07541.\n3. Nair, A. V., Pong, V., Dalal, M., Bahl, S., Lin, S., and Levine, S. (2018). Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31.\n4. Korenkevych, D., Mahmood, A. R., Vasan, G., and Bergstra, J. (2019). Autoregressive policies for continuous control deep reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2754–2762.\n5. Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., & Stone, P. (2020). Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181), 1-50.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: The approach is quite complicated [...]\n\nWe carried out additional experiments on two robot manipulation tasks, please see the **General comment** and its **attached PDF** (Fig. 16) for more details.\n\n**W2**:They missed citing a rich literature [...]\n\nWe will carefully review these papers and integrate them appropriately into the related work section to ensure a comprehensive discussion of existing research.\n\n**W3**:The reward function for the Maze envs is not provided [...]\n\nWe are using a trainable reward function both for training the RL algorithm and for generating curriculum goals. The trainable reward function, see Eq. 4, is learned by minimizing the Wasserstein distance between state visitation and the desired goal distribution. Given that we are working with maze environments, using Euclidean distance would not be appropriate for a reward function. Following the same reward approach as OUTPACE, we use the AIM reward function trained simultaneously with the RL algorithms. As demonstrated in the supp. mat. and Fig. 13b in the **attached PDF**, the reward function converges to the desired goal as training progresses. Therefore, we integrated the reward function into our curriculum goal generation mechanism, as it is useful in Eq. 11 and has the potential to guide the agent toward the desired goal. In summary, we do not use a sparse or pre-defined dense reward equation. Instead, we utilize a trainable reward function which is randomly initialized and specifically trained to minimize the Wasserstein distance between state-visitation and the desired goal distribution.\n\n**Q1**: In Fig. 3, what do the colours represent?\n\nWe will add a color bar to Fig. 3, which illustrates the corresponding timesteps for each different color of the curriculum goals. You can refer to our new plot in Fig. 15 in the **attached PDF** to have a preview of the modified figure.\n\n**Q2**: [...] How would DiCuRL + fixed start state fare against SAC only + random start states?\n\n**Q3**: How does SAC only perform in the comparisons in Fig. 1?\n\nTo address jointly Q2 and Q3, we removed the curriculum goal generation part from the code and provided only the final desired goal to agents in the maze tasks, which corresponds to training the agent solely with SAC. We obtained the first set of results by starting the agent in a fixed initial state at $(0,0)$. For the second set of results, we uniformly sampled the initial position of the agent randomly in the environment. To avoid starting the agent inside the walls of the maze, we performed an infeasibility check, such that if the initial state sampled was inside the walls, we continued sampling until a feasible initial state was found. We compared our DICURL approach with both SAC+fixed initial state and SAC + random initial state across all maze environments. The success rate is shown in Fig. 14 of the **attached PDF**.\nWe observe that including curriculum goals significantly improves the success rate in simpler tasks such as PointUMaze, helping the agent achieve the task successfully across different training seeds. Without the curriculum (SAC + fixed initial position), the agent struggles to achieve complex tasks consistently, resulting in high variance in performance. With SAC + random initial state, the agent often reaches the desired goal, at least in some scenarios. This success can be attributed to trial and error (without requiring curriculum goals) because the random starting positions help the agent avoid becoming stuck in maze walls, thus enhancing its ability to navigate the environment effectively.\n\n**Q4**: How important is the AIM reward? [...]\n\nYou are absolutely right; this is why we need to use different coefficients. To illustrate the effects of the Q and AIM reward functions, we conducted an ablation study. Please see our **General comment** and its **attached PDF** (Fig. 13a) for more details.\n\n**Q5**: [...] can the AIM reward be substituted with simpler intrinsic motivation rewards [...]\n\nYes, we believe that both RND and TD-error can potentially be substituted with the AIM reward. Indeed, in the AIM reward paper [a], the authors compare this approach with RND and show that RND performs worse compared to the AIM reward in certain tasks. Based on this evidence, we preferred to use the AIM reward for our method.\n\n**Q6**: [...] How does DiCuRL compare against SAC + HER?\n\nGiven the timeframe we had, we couldn’t test SAC+HER on the maze tasks. Instead, to demonstrate that our approach can be extended to robotic manipulation and sparse reward tasks, and compare it to HER, we used the official repository of [b], where the authors compare their approach with HER in a sparse reward setting. As detailed in the **General comment**, we implemented DiCuRL using HER settings in sparse reward robotic manipulation tasks. Please note that we used DDPG instead of SAC, which is another off-policy RL algorithm. All methods, including ours, use sparse rewards for training DDPG. However, since our method is based on the AIM reward, we only use the AIM reward for generating curriculum goals. You can find the comparison in Fig. 16 of the **attached PDF**.\n\nLimitations: Please suggest how this work can be extended to challenging environments with larger state-action spaces.\n\nTo demonstrate that our proposed method can be extended to more complex environments such as robotic manipulation tasks, we compared our approach with HGG and HER. The success rates are shown in Fig. 16 of the **attached PDF**. We will be happy to include these additional experiments -along with a brief discussion of these new results- in the final version of our paper.\n\n[a] Durugkar, I., et al. (2021), Adversarial intrinsic motivation for reinforcement learning\n\n[b] Ren, Z., et al. (2019), Exploration via hindsight goal generation", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b6CqQLktWj", "reviewer": "Reviewer_FHis", "summary": "This work presents a novel diffusion model-based curriculum learning approach, called DiCURL, for multi-goal reinforcement learning, namely goal-conditioned RL. The proposed conditional diffusion model leverages a Q-function and a learned reward function based on the Adversarial Intrinsic Motivation principle to incentivize goals that are reachable yet challenging to an RL agent. The paper evaluates DiCURL against state-of-the-art curriculum learning approaches in maze environments with differing maps. In PointUMaze and PointNMaze, DiCURL matches or slightly outperforms OUTPACE, which seems to be the best-performing method in these maze environments. In the most challenging map, PointSpiralMaze, DiCURL outperforms OUTPACE, while the rest of the methods fail to yield an optimal policy at the end of the training.", "questions": "- How do Q and AIM rewards differ in a goal-conditioned environment that provides a (sparse) reward for reaching the goal? Could you please give me an illustrative example to highlight how including both in the loss function of the diffusion model is better?\n\n- What is g_d that initializes g_c in Algorithm 2?\n\n- What do colors in figures illustrating curricula stand for specifically?", "limitations": "I don't see any explicit limitations regarding the proposed approach and the problem setting of interest other than those discussed by the authors in the conclusion section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The related work section is extensive in terms of content and covers most of the recent advances in automatic curriculum learning for RL. The background and methodology sections are also detailed, and the problem setting and the proposed approach are explained clearly.\n\n- The proposed curriculum learning approach is novel as it employs a conditional diffusion model. The idea of leveraging a Q-function and a learned intrinsic reward function to select achievable but challenging goals is intuitive, as well.\n\n- Table 1 highlights the advantages of DiCURL, and the introduction section also supports this table.\n\n- The curricula generated by DiCURL in Figures 2 and 3 (as well as the ones in the appendix) illustrate how DiCuRL yields optimal policies and outperforms some existing methods in evaluated environments.", "weaknesses": "- The introduction section should be improved in terms of writing. Content-wise, it is informative but also too dense. Some of the paragraphs are either too long or too short. Restructuring this section and making it more to the point would improve the readers' experience immensely. \n\n- OUTPACE is the second best-forming automatic curriculum learning method in the evaluated environments. However, the paper does not demonstrate the curricula generated by OUTPACE, unlike the curricula of GRADIENT and HGG in Figure 3, which do not perform as well.\n\n- All environments (point maze domain in MuJoCo with different maps) in the empirical validation section have the same dynamics, low-dimensional state, and action spaces. Although DiCuRL's advantages seem apparent as the map gets more complex, the empirical validation is insufficient to conclude that DiCuRL can outperform state-of-the-art methods in various goal-conditioned domains.\n\n- The roles of loss components related to the Q-function and AIM reward function sound intuitive, yet they are explained briefly. I suggest the authors run an ablation study to highlight their separate contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: The introduction section should be improved in terms of writing [...]\n\nWe agree with the reviewer's suggestions: we will restructure the introduction to ensure it is more concise and better organized.\n\n**W2**: [...], the paper does not demonstrate the curricula generated by OUTPACE [...]\n\nIn the **attached PDF**, we show the generated curriculum goals of OUTPACE for all maze environments (Fig. 15). Additionally, we have added a color bar to indicate which colors of the curriculum goals correspond to which timesteps. These figures will be included in the final version of the paper. Furthermore, we will update Fig. 3 in the paper to include the color bars.\n\n**W3**: [...] the empirical validation is insufficient to conclude that DiCuRL can outperform state-of-the-art [...]\n\nWe have additionally tested our approach on two robotic manipulation tasks. Please refer to our **General comment** and the **attached PDF** (Fig. 16) for more details.\n\n**W4**: The roles of loss components related to the Q-function and AIM reward [... ]\n\nWe have conducted an ablation study to explore the individual contributions of the Q function and the AIM reward when integrated with the diffusion loss $L_d$, as outlined in Eq. 11 in the paper. The success rate results from this study are presented in Fig. 13a in the **attached PDF**.\n\nConcerning the roles of the Q function and AIM reward, here’s our intuitive explanation:\n\n* Loss $L_d$: Minimizing this component helps us accurately capture the state distribution. It ensures that our generated curriculum goals are representative of the state-visitation distribution.\n\n* Q-function and AIM reward: The Q-function predicts the cumulative reward starting from a state, following the policy, while the AIM reward estimates how close an agent is to achieving its goal. We integrate both terms in the loss function by inverting their signs because our objective is to maximize Q and the AIM reward. By doing this, the diffusion model focuses on generating goals that not only minimize $L_d$ but also maximize the expected Q and AIM rewards.\n\nThis approach ensures that our generated curriculum goals are neither overly simplistic nor excessively challenging and progress towards the desired goal.\n\nWe have provided a detailed analysis with various visualizations for the PointUMaze environment in Sec. D in our supp. mat. If further details are required, we can provide a similar analysis for the PointSpiralMaze environment as well.\n\n**Q1**: How do Q and AIM rewards differ in a goal-conditioned environment [...]\n\nTo demonstrate the effects of using either the AIM reward or the $Q_\\phi$ function in conjunction with the $L_d$ component in Eq. 11 in the paper, we have provided illustrative examples in the **attached PDF**. In particular, Fig. 13c and 13d display the generated curriculum goals using $L_d$ with only $Q_\\phi$ and $L_d$ only with the AIM reward, respectively. The generated curriculum goals reveal that omitting the AIM reward results in suboptimal performance, whereas the absence of the $Q_\\phi$ function leads to the agent's inability to accomplish the task.\n\nAdditionally, we have implemented our method within a sparse reward, goal-conditioned RL framework across two different robotic manipulation tasks. We compared our method with HGG [a] and HER [b], as detailed in Fig. 16 in the **attached PDF**.\n\nIn this setting, the curriculum goals are integrated into our policy and the Q network (see line 7 in Algorithm 2 for the policy and line 10 in Algorithm 1 for the Q function). We utilize the AIM reward for both training RL algorithms and generating curriculum goals. This reward function, also a feature of OUTPACE, is a trainable neural network parametrized by $\\varphi$ that is initialized randomly and trained simultaneously with the RL algorithms. It is specifically trained to minimize the Wasserstein distance between state visitation and the desired goal distribution, as detailed in Section 3.3.\n\nThe distinction between training RL algorithms using the sparse (binary) reward and the AIM reward is significant. For instance, consider an agent at position (x,y) with a goal at (m,n). In a sparse reward setting, such as that used in the HER methodology, the reward is calculated based on the Euclidean distance between the agent's position and the goal. If this distance is greater than a predefined threshold (set to 0.05m in Gymnasium-Robotics), the agent receives a reward of 0; otherwise, it receives a reward of 1. In other words, If the agent reaches a given goal within a given threshold distance, the agent receives a positive reward, otherwise a non-positive one. In contrast, both our method and OUTPACE utilize a neural network-based reward—the AIM reward—which provides a continuous reward value. To illustrate this concept, we have included a 2D color map of the AIM reward function for the PointUMaze environment, showing how the reward evolves during different training episodes. These visualizations can be found in Fig. 8a, 10a, and 12a of our supp. mat. Additionally, the changes in the AIM reward for the PointSpiralmaze environment are illustrated in Fig. 13b in the **attached PDF**. \n\n**Q2**: What is g_d that initializes g_c in Algorithm 2?\n\n$g_d$ is the desired goal, and we initialize $g_c$ with $g_d$. At the beginning of training, our diffusion algorithm has not yet been run, so we initially provide the agent with $g_d$. Then, our curriculum goal generation algorithm generates curriculum goals for the agent.\n\n**Q3**: What do colors in figures illustrating curricula stand for specifically?\n\nWe will add a color bar to Fig. 3 to illustrate the corresponding timesteps for each different color of the curriculum goals. You can refer to the new plot in Fig. 15 of the **attached PDF**, which shows the intermediate goals for OUTPACE.\n\n[a] Andrychowicz, M., et al. (2017), Hindsight experience replay\n\n[b] Ren, Z., et al. (2019), Exploration via hindsight goal generation", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YVjBfqjLo7", "reviewer": "Reviewer_CrFc", "summary": "This paper studies curriculum reinforcement learning (RL) in the context of multi-goal RL, which aims to generate a series of goals with increasing difficulty to facilitate guiding learning policies. To this end, the paper proposes a framework that employs a conditional diffusion model that learns to generate a goal conditioned on the current state. The experiments in three maze navigation tasks show that the proposed method can reliably solve the tasks and perform them comparably to existing methods. This work studies a meaningful problem and proposes a reasonable framework. Yet, I am concerned with the limited domain (navigation) and tasks (maze) used for evaluation, the significance of the results, and the limited applicability beyond multi-goal RL, etc. Therefore, I am slightly leaning toward rejecting this paper, but I am willing to adjust my score if the rebuttal addresses my concern.", "questions": "See above", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Motivation and intuition**\n- The motivation for studying curriculum learning for multi-goal RL is convincing.\n- Leveraging diffusion models to generate goals is reasonable.\n\n**Clarity**\n- The overall writing is clear. The authors utilize figures well to illustrate the ideas. \n\n**Related work**\n- The authors provide comprehensive descriptions of existing works in curriculum RL.\n\n**Experimental results**\n- The experimental results show that the proposed method performs comparably to existing methods.\n\n**Reproducibility**\n- The code is provided, which helps understand the details of the proposed framework.", "weaknesses": "**Clarity**\n- The first paragraph of the introduction is unnecessarily long, making it very difficult to follow.\n- While the related work section describes several existing works in detail, it fails to differentiate these works from the proposed method exactly.\n\n**Limited to goal-conditioned RL**\n- The proposed method is limited to multi-goal RL, which requires a given goal. However, in many real-world applications, specifying a goal could be difficult or even impossible, making using the proposed method undoable. I feel it is entirely possible to extend the proposed method to the general RL setup, where only the current state is given. This will greatly increase the applicability of the proposed method.\n\n**Evaluation is limited to the Maze navigation**\n- The proposed method was only compared to existing methods in the Maze navigation tasks, where goals are represented as coordinates. It would be a lot more convincing if the evaluation was also conducted in other domains, such as robot arm manipulation, locomotion, and games. Additionally, evaluating in grid-world navigation tasks can add value to the paper by exploring discrete state and action spaces. \n\n**Significance of the results**\n- According to Figure 1, I am not entirely convinced that the proposed method performs significantly better than the baselines. Also, the plotting scheme makes it difficult to interpret when many curves overlap.\n\n**Related work**\n- The related work section focuses on existing works in curriculum RL yet fails to discuss many works that use diffusion models for RL or imitation learning, including but not limited to\n\t- \"Learning Universal Policies via Text-Guided Video Generation\"\n\t- \"Diffusion Policy: Visuomotor Policy Learning via Action Diffusion\"\n\t- \"Learning to Act from Actionless Video through Dense Correspondences\"\n\t- \"Goal-conditioned imitation learning using score-based diffusion policies\"\n\t- \"Diffusion model-augmented behavioral cloning\"\n\t- \"Imitating human behaviour with diffusion models\"\n\n**Algorithm 2**\n- While Algorithm 2 is titled RL Training, Lines 15-21 are for evaluation/testing, which is a bit confusing.\n\n**Minor errors**\n- L282: It seems that a non-break newline is used here, which gives no space between this paragraph and the next paragraph starting from Line 283.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: The first paragraph of the introduction is unnecessarily long [...]\n\nWe will restructure the introduction to ensure it is more concise and better organized, which we believe will significantly improve the clarity and readability of the paper.\n\n**W2**: While the related work section describes several existing works in detail, it fails to differentiate these works from the proposed method exactly.\n\nWe address this comment under **W6**.\n\n**W3**: [...] I feel it is entirely possible to extend the proposed method to the general RL setup, where only the current state is given. This will greatly increase the applicability of the proposed method.\n\nWe recognize that the necessity for giving a curriculum goal can restrict the applicability of our approach in real-world scenarios where such goals are not readily available or definable.\n\nIndeed, extending our method to a general RL setup, where the system operates solely based on the current state without explicit goal definitions, is a valuable direction for future research. Inspired by your suggestion and valuable references, we are considering an approach similar to that described in [a]: in this approach, a desired goal is defined using text encoding, and then a planner generates a series of future frames that illustrate the actions. Control actions are then derived from this generated video, enabling the agent to navigate without the need for specifically predefined goals. Additionally, we could adapt our policy (possibly our Q function using diffusion models), as demonstrated in [b]. Alternatively, we might explore learning directly from the environment using techniques such as RGB video, as proposed in [c], or through point clouds.\n\n**W4**: [...] It would be a lot more convincing if the evaluation was also conducted in other domains, such as robot arm manipulation, locomotion, and games [...]\n\nGiven the time constraints, we have carried out additional experiments on two robot manipulation tasks, although it would be entirely possible to apply to other domains such as indeed locomotion and games as well as discrete state and action spaces. Please see our **General comment** and its **attached PDF** (Fig. 16). These experiments hopefully demonstrate the applicability of our method to tasks that are quite different from maze navigation. \n\n**W5**: According to Fig. 1, I am not entirely convinced that the proposed method performs significantly better than the baselines. Also, the plotting scheme makes it difficult to interpret when many curves overlap.\n\nWe will provide clearer plots in our supp. mat. by creating different subplots and plotting together groups of 2 to 3 baseline methods. We hope that comparing only 2 or 3 baseline methods in each subplot will make the comparison clearer and easier to understand.\n\nConcerning the comparison between OUTPACE and DiCuRL, we have conducted a statistical analysis using the Wilcoxon rank-sum test to compare the no. of timesteps needed by the two methods to achieve a success rate greater than 0.99 across five different seeds for training. Here are the detailed test results for three specific environments:\n- PointSpiralMaze: p=0.04\n- PointNMaze: p=0.44\n- PointUMaze: p=0.04\n\nFor PointSpiralMaze and PointUMaze, there is statistically significant evidence to reject the null hypothesis (p<0.05), suggesting that DiCuRL statistically outperforms OUTPACE in these environments. Conversely, for PointNMaze, this is not the case. We note however that with 5 samples this analysis may be limited.\n\n**W6**: The related work section focuses on existing works in curriculum RL yet fails to discuss many works that use diffusion models for RL or imitation learning, including but not limited to [...]\n\nWe acknowledge our oversight and appreciate your detailed suggestions. We will thoroughly review these studies and will update the related work section to discuss them extensively, highlighting how they relate to our research.\n\n**W7**: While Algorithm 2 is titled RL Training, Lines 15-21 are for evaluation/testing, which is a bit confusing.\n\nThank you for pointing this out. We will update our algorithm title to \"Algorithm 2: RL Training and Testing”.\n\n**W8**: L282: It seems that a non-break newline is used here, which gives no space between this paragraph and the next paragraph starting from Line 283.\n\nThank you for noticing this layout issue. We will fix it in the revised version of the paper.\n\n[a] Du, Y., et al. (2024), Learning universal policies via text-guided video generation\n\n[b] Chi, C., et al. (2023), Diffusion policy: Visuomotor policy learning via action diffusion\n\n[c] Ko, P-C, et al. (2023), Learning to act from actionless videos through dense correspondences", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gi31VZl0c5", "reviewer": "Reviewer_7Mgp", "summary": "The paper presents an intuitive way to apply curriculum learning using diffusion based models to learn a goal distribution that can interpolate between the state-visitation distribution to states with high-value and high-intrinsic reward. As a result, the curriculum generates goals that lie at the edge of the states with non-zero occupancy, and higher value/ closeness to the target-goal.\n\nThe technical details are mostly complete and seem sound upon initial reading, I did not delve into the proofs/ derivations in the appendix. But the exposition of how we go from diffusion-models, to AIM, to visitation-count modelling, and to the newly proposed DiCuRL method, is mostly clear. \n\nAlthough there are multiple points of improvements, I think many practitioners will appreciate the author's work.", "questions": "1) Could the authors revise the current version and improve upon (most) of my critiques, then I'd be willing to raise my score.\n2) Will the authors share code?", "limitations": "The authors shortly discuss the limitations of their method, which I mostly agree with.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The technical details are quite clear upon initial reading, even if one is not familiar with AIM or diffusion models. I.e., the new method should be clear enough to reproduce from reading the paper.", "weaknesses": "### Major comments\n - The introduction dumps too much related work together to find the actual point and criticism that the authors want to make on the current state of the field.\n - Before reading the background/ technical details, motivation DiCuRL 1) is unclear how noising/ denoising is unique to helping exploration? Why can't another method (like a VAE, or GAN) do this through modelling the state-visitation distribution. Aren't we just choosing another, perhaps more powerful, generative method? **After reading the full paper:** I disagree that this is a sound motivation, any stochastic method over the state-visitation distribution could achieve this. I agree that modelling the state-visitation distribution is useful as it allows learning of goals that the agent has seen and can reach.  \n - 4.0 Line 222, it is not clear from the text what problem the authors are trying to solve through the graph construction and the optimization of the curiculum goal (Eq. 12). How is the 'optimal' curriculum goal even defined? Eq 12 of course shows the objective, but why do we need this? How is the graph even constructed (meaning the edges), is this fully-connected? Initial reading of this paragraph gives the impression of severe over-engineering of the goal-sampler.  \n - Figure 1 overlaps with table 1 and contains too many overlapping lines to draw a conclusion. This must be improved for presentation. Reduce the number of unnecesary baselines, show these in the appendix.\n - The results section spends most of its time speculating why the baselines perform in a certain way but does not focus on the authors' method. Line 281, states that there is a difference between OUTPACE and DiCuRL, however, neither method statistically significantly outperforms the other. Too much of the experimental setup is moved to the appendix.\n - It is unclear from figure 3 at what point during training this plot was made. Now the baseline methods look arbitrarily bad compared to the authors' method. It is color-coded, but maybe add a colorbar to figure 3 indicating the training episodes.\n\n### Technical comments\n - 3.3 Slight confusion on the reward $r^\\pi_\\phi$, it's good to mention that you're actually learning $f(s)$ and using this to compute $r$.\n - 4.0 Explanation on the mixing parameter $\\bar{\\alpha}_k$ is omitted. Shortly state it in the main text.\n - 4.0 The definition of $g_d$ is too hidden. I infer from Alg.2 that this is supposed to represent the *true* goal distribution. \n - Results, figure 1, table 2. Why plot the standard-deviations? Why not a non-parametric tolerance interval to get a sense of spread, or plot a confidence interval for the expected success-rate?\n\n\n### Minor comments\n - Intro paragraph 1 should be split into separate paragraphs making distinct points. Not a lumpsum of information.\n - Intro paragraph 1, maybe make a distinction between hierarchical RL + curriculum RL for goal-generation. Even if HRL can implicitly generate curriculums, the motivation is often slightly different.\n - Direct reference to papers should be done with the author: 'Person et al., (year) showed ...', not '[1, 2] showed ...'. Or you could write, 'Other studies [1, 2, 3], investigated ...' or something similar.\n- Intro paragraph 2 is not a paragraph but 1 sentence.\n- Figure 3, since DiCuRL is mostly on par with OUTPACE this should be compared in the plot for comparing curriculum goals", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: The introduction dumps too much related work together [...]\n\nWe will restructure the introduction to separate and clarify the related work.\n\n**W2**: [...] Why can't another method (like a VAE, or GAN) do this through modelling the state-visitation distribution? [...]\n\nWe acknowledge that other methods could effectively model the state distribution. However, we are cautious not to speculate on which method would yield the most successful output without further comparative analysis, which may be an interesting line of future research.\n\nWe believe that the noising/denoising mechanism of the diffusion model is particularly beneficial for the following reasons:\n* *Denoising* (lines 5-7 in Alg. 1): Gaussian noise is incrementally reduced by the neural network according to a specific variance schedule. This process is inherently imperfect: due to the neural network's limitations in precisely matching the sampled noise, a small degree of randomness remains. We believe that this residual noise introduces a subtle variability in the curriculum goals, which aids exploration by generating slightly varied goals.\n* *Noising* (lines 8-9 in Alg. 1): Original data sampled from a distribution are intentionally corrupted with Gaussian noise based on the sampled timestep k. The “noised” data are then processed by the neural network, and the loss is calculated (see Eq. 10 of the paper).\n\n**W3**: [...] it is not clear from the text what problem the authors are trying to solve through the graph construction and the optimization of the curriculum goal [...]\n\nWe employ two different strategies for generating curriculum goals:\n\n* **Bipartite Graph Optimization Strategy**: In this strategy, as outlined in the paper and also utilized in baseline methods like HGG and OUTPACE, we sample a mini-batch $b$ from the replay buffer $B$. This batch contains many states from different timesteps, which we provide to the curriculum goal generator, i.e., the diffusion model. The diffusion model generates a distribution of curriculum goal candidates $G_c$ from which we select the optimal curriculum goal $g_c$ using bipartite graph optimization. This graph maximizes the cost function given in Eq. 12.\n  - *Graph Construction*: The vertices of the bipartite graph can be divided into two disjoint sets $V_x$ and $V_y$, where $V_x$ consists of the generated curriculum goals and $V_y$ includes the desired goals sampled from the desired goal distribution. The graph is not fully connected as every pair of distinct vertices in our graph is not connected by a unique edge, but each vertex from $V_x$ is only connected to vertices in $V_y$.\n  - *Purpose of Eq. 12*: The objective here is to select K curriculum goals that are most diverse and beneficial for training, ensuring that the goals selected cover a broad spectrum of possible scenarios. We then randomly sample a single $g_c$ from these K goals for each episode.\n\n* **Single State Strategy** (see Sec. B of the supp. mat.): This method involves feeding only the state from the last timestep into the diffusion model, which then generates a single curriculum goal. The rationale behind using the last timestep is the assumption that the final state is closer to the desired goal. However, this does not always hold true as agents might not always progress linearly towards the goal, sometimes moving backwards or sideways. In other words, if the last state does not accurately represent progress towards the desired goal, the generated curriculum goals might not optimally guide the agent, potentially leading to decreased sample efficiency and slower learning rates. This effect is demonstrated in Fig. 4 in Section B of our supp. mat.\n\n**W4**: Fig. 1 overlaps with table 1 and contains too many overlapping lines to draw a conclusion [...]\n\nWe will revise Fig. 1 by reducing the number of baseline methods shown and displaying only the baselines that match our proposed method. Additionally, we will relocate some of the comparisons to the supp. mat.\n\n**W5**: [...] states that there is a difference between OUTPACE and DiCuRL, however, neither method statistically significantly outperforms the other [...]\n\nWe acknowledge that our analysis might have focused excessively on the baselines’ performances. We will revise the discussion accordingly and move part of the discussion from the Appendix to the main text (given the possibility of including an extra page in case of acceptance).\n\nConcerning the comparison between OUTPACE and DiCuRL, we have conducted a statistical analysis using the Wilcoxon rank-sum test to compare the no. of timesteps needed by the two methods to achieve a success rate greater than 0.99 across five different seeds for training. Here are the detailed test results for three specific environments:\n\n* PointSpiralMaze: p=0.04\n* PointNMaze: p=0.44\n* PointUMaze: p=0.04\n\nFor PointSpiralMaze and PointUMaze, there is statistically significant evidence to reject the null hypothesis (p<0.05), suggesting that DiCuRL statistically outperforms OUTPACE in these environments. Conversely, for PointNMaze, this is not the case. We note however that with 5 samples this analysis may be limited.\n\n**W6**: It is unclear from Fig. 3 at what point during training this plot was made [...]\n\nWe will add a color bar to Fig. 3 to indicate the timestep corresponding to each different color of the curriculum goals. You can refer to the new plot layout in Fig. 15 of the **attached PDF**, which shows the intermediate goals for OUTPACE.\n\n**Technical and minor comments**\n\nDue to space limitations, we cannot provide detailed replies. However, we acknowledge these comments and we'll do our best to handle them in the final version of the paper.\n\n**Q1**: Could the authors revise the current version and improve upon (most) of my critiques, then I'd be willing to raise my score.\n\nWe did (and will do) our best to address all the comments.\n\n**Q2**: Will the authors share code?\n\nOf course! Please see the link reported at L264.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **General Comment**

We sincerely thank all reviewers for the time and effort devoted to reviewing our manuscript. To address the key points raised, we have provided detailed responses to each reviewer. All responses are organized into questions and weaknesses. For example, **Q1** refers to the first question, while **W1** to the first weakness. Where needed, we also included a reply concerning the limitations.

For some responses, we have included additional results, which are available in the **attached PDF** (Fig. 13-16) included in this General comment.

We have been diligently working on improving the paper on several fronts, addressing all comments to the best of our capacity. We hope that the reviewers and chairs will appreciate our efforts and we wish to engage in a fruitful discussion during the rebuttal period.

Below, we summarize the main changes made:

* We conducted an **ablation study** to investigate the impact of the AIM reward function and Q function in generating curriculum goals with our method. For that, we omitted, separately, the reward function and the Q function from Eq. 11 in the paper, and plotted the success rate (with three different seeds) in Fig. 13a for the most challenging maze environment, PointSpiralMaze. The results indicate that the agent performs worse without the AIM reward function and fails to achieve the task without the Q function. The generated curriculum goals without the reward or Q function are shown in Fig. 13c and 13d, respectively. Fig. 13b, instead, illustrates the AIM reward value across different training episodes in a clockwise direction. Specifically, the first row and first column in Fig. 13b represent the reward values at the very beginning of training. As training progresses, the reward values shift towards the left corner of the maze environment (1st row, 2nd column). In the middle of training, the reward values are concentrated around the left corner of the maze environment (2nd row, 2nd column), and, by the end of training, the reward values converge to the desired goal area (2nd row, 1st column). This progression explains why the generated curriculum goals are not guiding the agent effectively but are instead distributed in the corner points shown in Fig. 13d. We have also demonstrated the behavior of the AIM reward function across different training episodes in our supp. mat. for the PointUMaze environment.
* Additionally, we **examined the impact of SAC with a fixed initial state [0,0] and SAC with a random initial state**. To do that, we removed the curriculum goal generation mechanism and assigned the desired goal, and then trained the agent using either SAC with a fixed initial state [0,0] or SAC with a random initial state. For the random initial state, we sampled goals randomly in the environment. To avoid starting the agent inside the maze walls, we performed an infeasibility check, resampling the initial state until it was feasible. We compared our approach using three different seeds with both the fixed initial state + SAC and the random initial state + SAC across all maze environments. The success rates are shown in Fig. 14 in the **attached PDF**. Note that the success rate for the PointUMaze environment in Fig. 14a is shown up to timestep $10^5$, whereas it was shown up to $10^6$ in the paper.
* We also **displayed the generated curriculum goals by the baseline method OUTPACE** in Fig. 15, with a color bar indicating the corresponding timesteps.
* To demonstrate the applicability of our method to different tasks, particularly in robot manipulation tasks, we implemented our approach using the official Hindsight Goal Generation (HGG) repository. We converted the HGG code from TensorFlow to PyTorch to integrate it with our diffusion model, which is based on PyTorch. We selected **two robot manipulation tasks, FetchPush and FetchPickAndPlace**, and increased the environment difficulty by expanding the desired goal area. This is shown in Fig. 16c and 16d, where the yellow area indicates the object sampling region and the blue area indicates the desired goal sampling region. The action space is four-dimensional, where three dimensions represent the Cartesian displacement of the end effector, and the last dimension controls the opening and closing of the gripper. The state space is 25-dimensional, including the end-effector position, position, and rotation of the object, the linear and angular velocity of the object, and the left and right gripper velocity. More detailed information regarding the action space and observation space of these robotic tasks can be found in the Gymnasium library documentation. For these additional experiments, we compared our method with HGG and HER using the DDPG algorithm, to ensure alignment with the baselines, using five different seeds. The results are shown in Fig. 16a and Fig. 16b, respectively for FetchPush and FetchPickAndPlace. Note that in this setting, all RL algorithms (including ours) use a binary reward (i.e., a sparse reward). However, since our curriculum goal generation algorithm is based on the AIM reward, we implemented the AIM reward function solely to generate curriculum goals while still using the sparse reward setting to train the DDPG algorithm using the diffusion model to generate curriculum goals.

**NOTE**: Reviewer `7Mgp` asked about GAN. We did compare our method with GOAL-GAN, where a goal generator proposes goal regions, and a goal discriminator is trained to evaluate if a goal is at the right level of difficulty for the policy. While GOAL-GAN does not consider the target distribution as our approach, we still consider it a pertinent baseline.

### **Reproducibility**

We have created a new (anonymous) repository containing the code for the additional experiments:

https://anonymous.4open.science/r/HER_diffusion-EB4F

The original (anonymous) codebase is available at (see L264 in the paper):

https://anonymous.4open.science/r/diffusioncurriculum/.

Meta-review: This paper has received unanimously positive reviews, with some Reviewers raising their scores during the discussion period. The engagement of Authors and Reviewers has been quite good, which plays in favor of the quality of the assessment of this work. In particular, Reviewers have appreciated the novelty of the proposed method, the clarity of the presentation, and the empirical evaluation.

I encourage the Authors to incorporate Reviewers' feedback in the final version.
Decision: Accept (poster)