Title: Diffusion-based Curriculum Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: curriculum reinforcement learning

reinforcement learning

diffusion models
Abstract: Curriculum Reinforcement Learning (CRL) is an approach to facilitate the learning process of agents by structuring tasks in a sequence of increasing complexity. Despite its potential, many existing CRL methods struggle to efficiently guide agents toward desired outcomes, particularly in the absence of domain knowledge. This paper introduces DiCuRL (Diffusion Curriculum Reinforcement Learning), a novel method that leverages conditional diffusion models to generate curriculum goals. To estimate how close an agent is to achieving its goal, our method uniquely incorporates a $Q$-function and a trainable reward function based on Adversarial Intrinsic Motivation within the diffusion model. Furthermore, it promotes exploration through the inherent noising and denoising mechanism present in the diffusion models and is environment-agnostic. This combination allows for the generation of challenging yet achievable goals, enabling agents to learn effectively without relying on domain knowledge. We demonstrate the effectiveness of DiCuRL in three different maze environments and two robotic manipulation tasks simulated in MuJoCo, where it outperforms or matches nine state-of-the-art CRL algorithms from the literature.

Diffusion-based Curriculum Reinforcement Learning

Abstract

Curriculum Reinforcement Learning (CRL) is an approach to facilitate the learning process of agents by structuring tasks in a sequence of increasing complexity. Despite its potential, many existing CRL methods struggle to efficiently guide agents toward desired outcomes, particularly in the absence of domain knowledge. This paper introduces DiCuRL (Diffusion Curriculum Reinforcement Learning), a novel method that leverages conditional diffusion models to generate curriculum goals. To estimate how close an agent is to achieving its goal, our method uniquely incorporates a Q -function and a trainable reward function based on Adversarial Intrinsic Motivation within the diffusion model. Furthermore, it promotes exploration through the inherent noising and denoising mechanism present in the diffusion models and is environment-agnostic. This combination allows for the generation of challenging yet achievable goals, enabling agents to learn effectively without relying on domain knowledge. We demonstrate the effectiveness of DiCuRL in three different maze environments and two robotic manipulation tasks simulated in MuJoCo, where it outperforms or matches nine state-of-the-art CRL algorithms from the literature.

1 Introduction

Reinforcement learning (RL) is a computational method that allows an agent to discover optimal actions through trial and error by receiving rewards and adapting its strategy to maximize cumulative rewards. Deep RL, which integrates deep neural networks (NNs) with RL, is an effective way to solve large-dimensional decision-making problems, such as learning to play video games [1, 2], chess [3], Go [4], and robot manipulation tasks [5, 6, 7, 8]. One of the main advantages of deep RL is that it can tackle difficult search problems where the expected behaviors and rewards are often sparsely observed. The drawback, however, is that it typically needs to thoroughly explore the state space, which can be costly especially when the dimensionality of this space grows.

Some methods, such as reward shaping [9], can mitigate the burden of exploration, but they require domain knowledge and prior task inspection, which limits their applicability. Alternative strategies have been proposed to enhance the exploration efficiency in a domain-agnostic way, such as prioritizing replay sampling [8, 10, 11], or generating intermediate goals [12, 13, 14, 15, 16, 17, 18, 19]. This latter approach, known as Curriculum Reinforcement Learning (CRL), focuses on designing a suitable curriculum to guide the agent gradually toward the desired goal.

Various approaches have been proposed for the generation of curriculum goals. Some methods focus on interpolation between a source task distribution and a target task distribution [20, 21, 22, 17]. However, these methods often rely on assumptions that may not hold in complex RL environments, such as specific parameterization of distributions, hence ignoring the manifold structure in space. Other approaches adopt optimal transport [13, 23], but they are typically applied in less challenging exploration scenarios. Curriculum generation based on uncertainty awareness has also been explored, but such methods often struggle with identifying uncertain areas as the goal space expands [15, 24, 12].

Some research minimizes the distance between generated curriculum and desired outcome distributions using Euclidean distance, although this approach can be problematic in certain environments [19, 25]. Other methods incorporate graph-based planning, but require an explicit specification of obstacles [26, 27]. Lastly, approaches based on generative AI models have been proposed. For instance, [14] uses GANs to generate tasks of intermediate difficulty, but it relies on arbitrary thresholds. Alternatively, [28, 29, 30] apply diffusion models in offline RL settings [14, 28, 30].

Despite these advancements, existing CRL approaches still struggle with generating suitable intermediate goals, particularly in complex environments with significant exploration challenges. To overcome this challenge, in this paper, we propose DICURL (Diffusion Curriculum Reinforcement Learning). Our method leverages conditional diffusion models to dynamically generate curriculum goals, guiding agents towards desired goals while simultaneously considering the Q -function and a trainable reward function based on Adversarial Intrinsic Motivation (AIM) [31].

Contributions Unlike previous offline RL approaches [28, 29, 30] that train and use diffusion models for planning or policy generation relying on pre-existing data, DICURL facilitates online learning , enabling agents to learn effectively without requiring domain-specific knowledge. This is achieved by three key elements. 1 The diffusion model captures the distribution of visited states and facilitates exploration through its inherent noising and denoising mechanism. 2 As the Q -function predicts the cumulative reward starting from a state and a given goal while following a policy, we can determine feasible goals by maximizing the Q -function, ensuring that the generated goals are challenging yet achievable for the agent. 3 The AIM reward function estimates the agent's proximity to the desired goal and allows us to progressively shift the curriculum towards the desired goal.

We compare our proposed approach with nine state-of-the-art CRL baselines in three different maze environments and two robotic manipulation tasks simulated in MuJoCo [32]. Our results show that DICURL surpasses or performs on par with the state-of-the-art CRL algorithms.

2 Related Work

Curriculum Reinforcement Learning CRL[33] algorithms generally adjust the sequence of learning experiences to improve the agent's performance or accelerate training. These algorithms focus on formulating intermediate goals that progressively guide the agent toward the desired goal, and have been successfully applied to various tasks, mainly in the field of robot manipulation [34, 35, 36, 37].

Hindsight Experience Replay (HER) [38] tackles the challenge of sparse reward RL tasks by employing hindsight goals, considering the achieved goals as pseudo-goals, and substituting them for the desired goal. However, HER struggles to solve tasks when the desired goals are far from the initial position. Hindsight Goal Generation (HGG) [19] addresses the inefficiency issue inherent in HER by generating hindsight goals through maximizing a value function and minimizing the Wasserstein distance between the achieved goal and the desired goal distribution.

CURROT [23] and GRADIENT [13] both employ optimal transport for the generation of intermediate goals. CURROT formulates CRL as a constrained optimization problem and uses the Wasserstein distance to measure the distance between distributions. Conversely, GRADIENT introduces task-dependent contextual distance metrics and can manage non-parametric distributions in both continuous and discrete context settings; moreover, it directly interprets the interpolation as the geodesic from the source to the target distribution.

GOAL-GAN [14] generates intermediate goals using a Generative Adversarial Network (GAN) [39], without considering the target distribution. A goal generator is used to propose goal regions, and a goal discriminator is trained to evaluate if a goal is at the right level of difficulty for the current policy. The specification of goal regions is done using an indicator reward function, and policies are conditioned on the goal as well as the state, similarly to a universal value function approximator [40].

PLR [15] uses selective sampling to prioritize instances with higher estimated learning potential for future revisits during training. Learning potential is estimated using TD-Errors, resulting in the creation of a more challenging curriculum. VSD [16] estimates the epistemic uncertainty of the value function and selects goals based on this uncertainty measure. The value function confidently assigns high values to easily achievable goals and low values to overly challenging ones. ACL [18] maximizes the learning progress by considering two main measures: the rate of improvement in prediction accuracy and the rate of increase in network complexity. This signal acts as an indicator of

the current rate of improvement of the learner. The ALP-GMM [17] fits Gaussian Mixture Models (GMM) using an Absolute Learning Progress (ALP) score, which is defined as the absolute difference in rewards between the current episode and the previous episodes. The teacher generates curriculum goals by sampling environments to maximize the student's ALP, which is modeled by the GMM.

Finally, OUTPACE [12] employs a trainable intrinsic reward mechanism, known as Adversarial Intrinsic Motivation (AIM) [31] (the same used in our method) which is designed to minimize the Wasserstein distance between the state visitation distribution and the goal distribution. This function increases along the optimal goal-reaching trajectory. For curriculum goal generation, OUTPACE uses Conditional Normalized Maximum Likelihood (CNML), to classify state success labels based on their association with visited states, out-of-distribution samples, or the desired goal distribution. The method also prioritizes uncertain and temporally distant goals using meta-learning-based uncertainty quantification [41] and Wasserstein-distance-based temporal distance approximation.

Diffusion Models for Reinforcement Learning UniPi [42] leverages diffusion models to generate a video as a planner, conditioned on an initial image frame and a text description of a current goal. Subsequently, a task-specific policy is employed to infer action sequences from the generated video using an inverse dynamic model. AVDC [43] constructs a video-based robot policy by synthesizing a video that renders the desired task execution and directly regresses actions from the synthesized video without requiring any action labels or inverse dynamic model. It takes RGBD observations and a textual goal description as inputs, synthesizes a video of the imagined task execution using a diffusion model, and estimates the optical flow between adjacent frames in the video. Then, using the optical flow and depth information, it computes robot commands.

Diffusion Policy [44] uses a diffusion model to learn a policy through a conditional denoising diffusion process. BESO [45] adopts an imitation learning approach that learns a goal-specified policy without any rewards from an offline dataset. DBC [46] uses a diffusion model to learn stateaction pairs sampled from an expert demonstration dataset and increases generalization using the joint probability of the state-action pairs. Finally, Diffusion BC [47] uses a diffusion model to imitate human behavior and capture the full distribution of observed actions on robot control tasks and 3D gaming environments.

Limitations of current works and distinctive aspects of DICURL The aforementioned studies typically require offline data for training. Both [42] and [43] employ diffusion models to synthesize videos for rendering the desired task execution, and actions are then inferred from such videos. Studies such as [44, 45, 46, 47] also focus on learning policies from offline datasets. Despite these efforts, reliance on inadequate demonstration data can lead to suboptimal performance [44]. Distinct from these approaches, our method instead does not rely on prior expert data or any pre-collected datasets. As an off-policy RL method, DICURL collects in fact data through interaction with the environment.

3 Background

We now introduce the background concepts on multi-goal RL, Soft Actor-Critic (SAC), Wasserstein distance, Adversarial Intrinsic Motivation (AIM), and diffusion models.

3.1 Multi-Goal Reinforcement Learning

In the context of multi-goal RL, we can formulate the RL problem as a goal-oriented Markov decision process (MDP) characterized by continuous state and action spaces. This MDP is defined by the tuple ⟨S , A , G , T , p, γ r ⟩ . Here, S represents the state space, A denotes the action space, and G is the goal space. The transition dynamics, T ( s ′ | s, a ) , describes the probability of transitioning to the next state s ′ given the current state s and action a . The joint probability distribution over the initial states and the desired goal distribution is represented by p ( s 0 , g ) , and γ r ∈ [0 , 1] is a discount factor.

We utilize the AIM reward function r φ , as outlined in Section 3.3. The objective is to identify the optimal policy π that maximizes the expected cumulative reward approximated by the value function Q π ( s, g, a ) . This function can be expanded to the Universal Value Function (UVF), a goal-conditioned value function incorporating the goal into value estimation. The UVF, defined as V π ( s, g ) , where s is the current state, g is the goal, and π is the policy, estimates the expected return

from state s under policy π , given goal g . The UVF has been shown to successfully generalize to unseen goals, which makes it a viable approach for multi-goal RL [40].

3.2 Soft Actor-Critic

Soft Actor-Critic (SAC) [48] is an off-policy RL algorithm that learns a policy maximizing the sum of the expected discounted cumulative reward and the entropy of the policy, namely: J ( π ) = ∑ T t =0 E s t ∼B r + α e H ( π ( · | s t , g )) . The policy π ψ ( a | s ) : S ↦→ P ( A ) defines a map from state to distributions over actions, parameterized by ψ . A state-action value function is defined as Q ϕ ( s, g, a ) : S × G × A → R , parameterized by ϕ . The policy parameters can be learned by directly minimizing the expected KL divergence and the Q -value is trained by minimizing the soft Bellman residual, which are defined as the following loss functions:

where V ϕ ( s, g ) = E a ∼ π [ Q ϕ ( s, g, a ) -log π ψ ( a | s, g )] , B is the replay buffer and α e is the temperature parameter that controls the stochasticity of the optimal policy.

3.3 Wasserstein Distance and Adversarial Intrinsic Motivation Reward Function

The Wasserstein distance offers a way to quantify the amount of work required to transport one distribution to another distribution. The Wasserstein-p distance W p between two distributions µ and ν on a metric space ( X , d ) , where X is a set and d denotes a metric on X , is defined as follows [49]:

where Π( µ, ν ) denotes the set of all possible joint distributions γ ( x, y ) whose marginals are µ and ν . Intuitively, γ ( x, y ) tells what is the least amount of work, as measured by d , that needs to be done to convert the distribution µ into the distribution ν [50]. A timestep quasi-metric d π ( s, g ) can be used as a distance metric. It estimates the work needed to transport one distribution to another, representing the number of transition steps required to reach the goal state g ∈ G for the first time when following the policy π .

As proposed by Durugkar et al. [31], the AIM reward function can be learned by minimizing the Wasserstein distance between the state visitation distribution ρ π and the desired goal distribution G . Through the minimization of the Wasserstein distance W 1 ( ρ π , G ) (for p = 1 , W 1 is known as the Kantorovich-Rubinstein distance), a reward function can be formulated to estimate the work required to transport the state visitation distribution ρ π to the desired goal distribution G and is expressed as follows: W 1 ( ρ π , G ) = sup ∥ f ∥ L ≤ 1 [ E g ∼G [ f ( g )] -E s ∼ ρ π [ f ( s )]] . If the state visitation distribution ρ π ( s ) comprises states that optimally progress towards the goal g , the potential function f ( s ) increases along the trajectory, reaching its maximum value at f ( g ) . The reward function, which can be approximated using a neural network, denoted as r π φ , increases as the states approach the desired goal g ∈ G . r π φ can be trained using the data collected by the policy π . Leveraging the estimation of the Wasserstein distance W 1 ( ρ π , G ) , the loss function for training the parameterized reward function r π φ is defined as follows:

where the second component of the sum is a penalty term and the coefficient λ is necessary to ensure smoothness [31]. The reward can be calculated as r φ ( s, g ) = f π φ ( s ) -f π φ ( g ) , which is the negative of the Wasserstein distance -W 1 ( ρ π , G ) .

3.4 Diffusion Models

Diffusion models [51] express a probability distribution p ( x 0 ) through latent variables in the form p θ ( x 0 ) := ∫ p θ ( x 0: N ) d x 1: N , where x 1 , . . . , x N are latent variables of the same dimensionality as the data x 0 ∼ p ( x 0 ) . They are characterized by a forward and a reverse diffusion process.

The forward diffusion process approximates the posterior q ( x 1: N | x 0 ) using a Markov chain that perturbs the input data by gradually adding Gaussian noise x 0 ∼ q ( x 0 ) in N steps with a predefined variance schedule β 1 , . . . , β N . This process is defined as:

The reverse diffusion process aims to recover the original input data from the noisy (diffused) data. It learns to progressively reverse the diffusion process step by step and approximates the joint distribution p θ ( x 0: N ) . This process is defined as:

where p ( x N ) = N ( x N ; 0 , I ) . The optimization of the reverse diffusion process is achieved by maximizing the evidence lower bound (ELBO) E q [ ln p θ ( x 0: N ) q ( x 1: N | x 0 ) ] [52]. Once trained, sampling data from Gaussian noise x N ∼ p ( x N ) and running through the reverse diffusion process from k = N to k = 0 yields an approximation of the original data distribution.

4 Methodology

As discussed earlier, in multi-goal RL, the desired goal is sampled from a desired goal distribution in each episode, and the agent aims to achieve multiple goals. By integrating a curriculum design into multi-goal RL, we can reformulate a task in such a way that it starts with easier goals and progressively increases in difficulty. Our curriculum diffusion-model-based goal-generation method works as follows. Given the presence of two different types of timesteps for the diffusion process and the RL task, we denote the diffusion timesteps using subscripts k ∈ { 1 , . . . , N } and the RL trajectory timesteps using subscripts t ∈ { 1 , . . . , T } . Our curriculum goal generation takes the state s as an input. It then outputs a curriculum goal set G c , which is obtained through the reverse diffusion process of a conditional diffusion model as follows:

where g 0 is the end sample of the reverse diffusion process used as a curriculum goal. Commonly, p θ ( g k -1 | g k , s ) is a conditional distribution parametrized by θ and is chosen to model a multivariate Gaussian distribution N ( g k -1 ; µ θ ( g k , s , k ) , Σ θ ( g k , s , k )) . Following [51], rather than learning the variances of the forward diffusion process, we assign a fixed covariance matrix Σ θ ( g i , s , i ) = β i I and a mean defined as:

Initially, we sample a Gaussian noise g N ∼ N ( 0 , I ) . We then apply the reverse diffusion process parameterized by θ starting from the last step N and proceeding backward to step 1 :

For the case k = 1 (last term of the reverse diffusion process), we set ϵ to 0 , to enhance the sampling quality by ignoring √ β k and edge effects. In fact, as empirically demonstrated by [51], training the diffusion model yields better results when utilizing a simplified loss function that excludes the weighting term. Thus, we adopt the following simplified loss function to train the conditional ϵ θ -model, where ϵ θ is a function approximator designed to predict ϵ 1 .

where U is a uniform distribution over the discrete set { 1 , . . . , N } and B denotes the replay buffer collected by policy π .

To sample feasible curriculum goals for the agent, we integrate the Q -function and the AIM reward function into the loss. The total loss function for training the diffusion model is defined as follows:

The rationale is that, while minimizing the loss L d allows us to accurately capture the state distribution, we also aim to simultaneously maximize the expected value of Q and the AIM reward function r φ , using the weights ξ d , ξ q , and ξ r to adjust the relative importance of the three components of the loss function. More in detail, the Q -function predicts the cumulative reward starting from a state and following the policy, while the AIM reward function estimates how close an agent is to achieving its goal. By maximizing Q and the AIM reward, we can generate curriculum goals that are neither overly simplistic nor excessively challenging, progressing towards the desired goal.

In Eq. (11), g 0 is obtained by sampling experiences from the replay buffer using Eq. (9) through a reverse diffusion process parameterized by θ . Therefore, taking the gradient of Q and the AIM reward r φ involves back-propagating through the entire diffusion process. After we obtain the set of curriculum goals G c from the diffusion model, we use a bipartite graph G ( { V x , V y } , E ) with edge costs w , composed of vertices V x (i.e., the curriculum goal candidates derived from the diffusion model) and vertices V y (i.e., the desired goals), where E represents the edge weights, and select the optimal curriculum goal g c 2 . We employ the Minimum Cost Maximum Flow algorithm in order to solve the bipartite matching problem and identify the edges with the minimal cost w :

The specifics of the generation of intermediate goals through diffusion models are explained in Algorithm 1, while the overall algorithm is outlined in Algorithm 2.

In Algorithm 1, during the training iterations, we sample Gaussian noise g N ∼ N ( 0 , I ) to denoise the data in the reverse diffusion process. Between lines 5 and 7, we iterate from time step N to 1 , where we perform the reverse diffusion process using Eq. (9) and subtract the predicted noise ϵ θ from g N to denoise the noisy goal iteratively. We then uniformly sample a timestep k from the range between 1 and N (line 8) and sample a Gaussian noise ϵ ∼ N ( 0 , I ) (line 9) in order to calculate the diffusion loss defined in Eq. (10). In line 10, we calculate the diffusion loss, the Q -value, and the AIM reward function using the generated goal g 0 from the reverse diffusion process. Then in line 11, we calculate the total loss defined in Eq. (11) and update the diffusion model parameters θ using gradient descent. Finally, we return the generated goal g 0 .

In Algorithm 2 we begin by defining an off-policy algorithm denoted as A . While any off-policy algorithm could be employed, we choose Soft Actor-Critic (SAC) to align with the baseline algorithms tested in our experiments. In line 5, we sample the initial state and provide the curriculum goal g c to the policy, along with the current state. The policy generates an action (line 7), and executes it in the environment (line 8). Subsequently, the next state and reward are obtained (line 9). Then, we provide the minibatch b to the curriculum goal generator in line 11 to generate a curriculum goal set G c . Then we find the optimal curriculum goal g c using bipartite graph optimization (line 12). Furthermore, the loss functions defined in Eq. (1) and Eq. (2) are calculated, and the networks approximating π and Q , as well as the AIM reward function r φ defined in Eq. (4), are updated. Between line 15 and 21, we run n test rollouts and extract the achieved state of the agent using ϕ and calculate the success rate in reaching the desired goal within a threshold value κ .

Algorithm 1 Diffusion Curriculum Goal Generator

Algorithm 2 RL Training and Evaluation

5 Experiments

To evaluate our proposed method, we conducted experiments across three maze environments simulated in MuJoCo 3 : PointUMaze , PointNMaze , and PointSpiralMaze . In these environments, a goal is interpreted as the ( x, y ) position of the agent achieved in an episode. These environments have been specifically chosen due to their structural characteristics, which are ideal for testing environment-agnostic curriculum generation strategies. Moreover, they present a variety of complex and diverse navigation challenges, requiring an agent to learn effective exploration and exploitation.

We compared our approach DICURL against nine state-of-the-art CRL algorithms, namely ACL [18], GOAL-GAN [14], HGG [19], ALP-GMM [17], VDS [16], PLR [15], CURROT [23], GRADIENT [13], and OUTPACE [12], each one run with 5 different random seeds. The primary conceptual differences between ours and baseline algorithms are summarized in Table 1. Details on the baseline algorithms, parametrization, training setup, and maze environments are given in Supp. Mat. C. Our codebase is available at: https://github.com/erdiphd/DiCuRL/.

The results, shown in Fig. 1 4 and detailed in Table 2 5 , demonstrate the effectiveness of the proposed DICURL method. Notably, DICURL outperforms or matches all the baseline methods. For the PointNMaze and PointSpiralMaze, the success rate of all methods, except for ours and OUTPACE (and HGG for PointNMaze), is close to zero (detailed results are reported in Supp. Mat. C). ACL, GoalGAN, ALP-GMM, VDS, and PLR, which lack awareness of the target distribution, underperform compared to target-aware methods such as our proposed approach, OUTPACE, GRADIENT, CURROT, and HGG. HGG encounters difficulties because of infeasible curriculum proposals. This is a result of its reliance on the Euclidean distance metric.

Both CURROT and GRADIENT generate curriculum goals using the optimal transport method and the Wasserstein distance metric, relying on the geometry of the environment. This dependence might be the reason for their inconsistent and poor performance across the different environments.

OUTPACE, utilizes instead the Wasserstein distance and an uncertainty classifier for uncertaintyaware CRL, exhibiting similar performance to our approach but with slower convergence and higher variance in success rate. This is likely due to the fact that OUTPACE's curriculum heavily relies on the visited state distributions, which necessitate an initial exploration by the agent. While also our approach depends on visited states, incorporating the Qand AIM reward functions into curriculum generation facilitates exploration beyond them, potentially explaining the performance differences between DICURL and OUTPACE.

Lastly, we recall that our approach generates curriculum goals based on the reverse diffusion process, which allows the generated curriculum goals to gradually shift from the initial state distribution to the desired goal distribution. Fig. 2 shows an example of a curriculum set generated by DICURL, for the case of the PointUMaze environment, at each iteration of the reverse diffusion process. Fig. 3, instead, shows the differences between the curriculum goals generated by DICURL, GRADIENT, and HGG in the PointSpiralMaze environment: it can be seen how DICURL manages to generate curriculum goals that explore the whole environment more effectively than the baseline algorithms. Supp. Mat. D shows the curriculum goals generated in the other two environments and illustrates the dynamics of the diffusion process during training for the case of PointUMaze. To demonstrate the applicability of DICURL to robot manipulation tasks, we evaluated it on the FetchPush and FetchPickAndPlace tasks using a sparse reward setting. Further details can be found in Supp. Mat. E.

Ablation Study We conducted an ablation study to investigate the impact of the AIM reward function r φ and Q ϕ function in generating curriculum goals with our method (DICURL). For that, we omitted, separately, the reward function r φ and the Q ϕ function from Eq. 11, and plotted the success rate (with three different seeds) in Fig. 4a for the most challenging maze environment, PointSpiralMaze. The results indicate that the agent performs worse without the AIM reward function r φ and fails to achieve the task without the Q ϕ function. The generated curriculum goals without the r φ or Q ϕ function are shown in Fig. 4b and 4c, respectively. Fig. 4d, instead, illustrates the AIM reward value across different training episodes in a clockwise direction. Specifically, the first row and first column

PointUMaze

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

(a) PointUMaze

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

(b) PointNMaze

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

DiCuRL

(ours)

OUTPACE

HGG

GRADIENT

CURROT

ACL

ALP-GMM

GoalGAN

PLR

VDS

(c) PointSpiralMaze

PointNMaze

PointSpiralMaze

Algorithm

PointUMaze

PointNMaze

PointSpiralMaze

DICURL (Ours)

28333

±

3036

108428

±

34718

305833

±

43225

OUTPACE

29800

±

4166

113333

±

24267

396875

±

111451

HGG

48750

±

24314

NaN

NaN

GRADIENT

263431

±

114795

NaN

NaN

g8

g7

96

95

94

g3

g2

91

go

in Fig. 4d represent the reward values at the very beginning of training. As training progresses, the reward values shift towards the left corner of the maze environment (1st row, 2nd column). In the middle of training, the reward values are concentrated around the left corner of the maze environment (2nd row, 2nd column), and, by the end of training, the reward values converge to the desired goal area (2nd row, 1st column). This progression explains why the generated curriculum goals are not guiding the agent effectively but are instead distributed in the corner points shown in Fig. 4c. We have also demonstrated the behavior of the AIM reward function across different training episodes in our Supp. Mat. D for the PointUMaze environment. Additionally, we examined the impact of SAC with a fixed initial state [0 , 0] and SAC with a random initial state. To do that, we removed the curriculum goal generation mechanism and assigned the desired goal, and then trained the agent using either SAC with a fixed initial state [0 , 0] or SAC with a random initial state. For the random initial state, we sampled goals randomly in the environment. To avoid starting the agent inside the maze walls, we performed an infeasibility check, resampling the initial state until it was feasible. We compared our approach using three different seeds with both the fixed initial state + SAC and the random initial state + SAC across all maze environments, and the success rates are shown in Fig. 5.

(a) DICURL

(b) OUTPACE

(c) GRADIENT

8×105

8 ×105

8×105

6 × 105

6×105

Time Steps

Time Steps

6×105

4×105

4×105

2×105

2×105

4×105

2 × 105

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate



d

,

Qϕ

,

rφ



d

,

Qϕ



d

,

rφ

(a)

(b)

(c)

(d)

8×105

8×105

6 × 105

6 × 105

4×105

4 ×105

2 ×105

2 ×105

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

×10 5

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

DiCuRL

SAC + random

SAC + fixed

(a) PointUMaze

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

(b) PointNMaze

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

(c) PointSpiralMaze

6 Conclusion and Limitations

In this work, we introduced DICURL, a novel approach that utilizes diffusion models to generate curriculum goals for an RL agent. The diffusion model is trained to minimize its loss function while simultaneously maximizing the expected value of Q and the AIM reward function. The minimization of the diffusion loss helps capture the visited state distribution, while the maximization of Q and AIM reward function helps generate goals at an appropriate difficulty level and at the same time guide the generated curriculum goals closer to the desired goal. Furthermore, the generated goals promote exploration due to the inherent noising and denoising mechanism of the diffusion model. Our proposed approach has two main limitations. First, while diffusion models excel at handling highdimensional data, such as images [53], incorporating the AIM reward into a combined loss function can hinder the curriculum goal generation in such settings as the AIM reward may underperform in higher dimensionalities. Secondly, we employ the Minimum Cost Maximum Flow algorithm to select the optimal curriculum goals from the set generated by the diffusion model. However, alternative selection strategies could potentially be more effective. Future work will aim to address these limitations and extend our approach to more complex environments.

7 Acknowledgments

This study was supported by the CiLoCharging project, which is funded by the Bundesministerium für Wirtschaft und Klimaschutz (BMWK) of Germany under funding number 01ME20002C. Additionally, this work was supported by TUBITAK under the 2232 program, project number 121C148 ('LiRA'). We would like to express our gratitude to Arda Sarp Yenicesu for his assistance in writing this paper.

References

Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Graves, Alex and Antonoglou, Ioannis and Wierstra, Daan and Riedmiller, Martin. Playing Atari with deep reinforcement learning, 2013. arXiv:1312.5602.

Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Rusu, Andrei A and Veness, Joel and Bellemare, Marc G and Graves, Alex and Riedmiller, Martin and Fidjeland, Andreas K and Ostrovski, Georg and others. Human-level control through deep reinforcement learning. Nature , 518:529-533, 2015.

Silver, David and Hubert, Thomas and Schrittwieser, Julian and Antonoglou, Ioannis and Lai, Matthew and Guez, Arthur and Lanctot, Marc and Sifre, Laurent and Kumaran, Dharshan and Graepel, Thore and others. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science , 362(6419):1140-1144, 2018.

Silver, David and Huang, Aja and Maddison, Chris J and Guez, Arthur and Sifre, Laurent and Van Den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershelvam, Veda and Lanctot, Marc and others. Mastering the game of Go with deep neural networks and tree search. Nature , 529:484-489, 2016.

Rajeswaran, Aravind and Lowrey, Kendall and Todorov, Emanuel V and Kakade, Sham M. Towards generalization and simplicity in continuous control. In Advances in Neural Information Processing Systems , volume 30, San Diego, CA, USA, 2017. Neural Information Processing Systems Foundation.

Ng, Andrew Y and Coates, Adam and Diel, Mark and Ganapathi, Varun and Schulte, Jamie and Tse, Ben and Berger, Eric and Liang, Eric. Autonomous inverted helicopter flight via reinforcement learning. In International Symposium on Experimental Robotics , pages 363-372, Berlin Heidelberg, Germany, 2006. Springer.

Timothy P. Lillicrap and Jonathan J. Hunt and Alexander Pritzel and Nicolas Heess and Tom Erez and Yuval Tassa and David Silver and Daan Wierstra. Continuous control with deep reinforcement learning, 2019. arXiv:1509.02971.

Sayar, Erdi and Bing, Zhenshan and D'Eramo, Carlo and Oguz, Ozgur S and Knoll, Alois. Contact Energy Based Hindsight Experience Prioritization, 2023. arXiv:2312.02677.

Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning , volume 99, pages 278-287. Citeseer, 1999.

Rui Zhao and Volker Tresp. Energy-based hindsight experience prioritization. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto, editors, Conference on Robot Learning , volume 87 of Proceedings of Machine Learning Research , pages 113-122. PMLR, 29-31 Oct 2018.

Rui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy-regularized multi-goal reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 7553-7562. PMLR, 09-15 Jun 2019.

Daesol Cho, Seungjae Lee, and H. Jin Kim. Outcome-directed reinforcement learning by uncertainty & temporal distance-aware curriculum goal generation. In International Conference on Learning Representations . OpenReview.net, 2023.

Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and DING ZHAO. Curriculum reinforcement learning using optimal transport via gradual domain adaptation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 10656-10670. Curran Associates, Inc., 2022.

Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning , pages 1515-1528. PMLR, 2018.

Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Marina Meila and Tong Zhang, editors, International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 4940-4950. PMLR, 18-24 Jul 2021.

Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 7648-7659. Curran Associates, Inc., 2020.

Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Conference on Robot Learning , volume 100 of Proceedings of Machine Learning Research , pages 835-853. PMLR, 30 Oct-01 Nov 2020.

Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In Doina Precup and Yee Whye Teh, editors, International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1311-1320. PMLR, 06-11 Aug 2017.

Ren, Zhizhou and Dong, Kefan and Zhou, Yuan and Liu, Qiang and Peng, Jian. Exploration via Hindsight Goal Generation. arXiv:1906.04279.

Pascal Klink, Hany Abdulsamad, Boris Belousov, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. A probabilistic interpretation of self-paced learning with applications to reinforcement learning. Journal of Machine Learning Research , 22(182):1-52, 2021.

Pascal Klink, Carlo D' Eramo, Jan R Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 9216-9227. Curran Associates, Inc., 2020.

Pascal Klink, Hany Abdulsamad, Boris Belousov, and Jan Peters. Self-paced contextual reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Conference on Robot Learning , volume 100 of Proceedings of Machine Learning Research , pages 513-529. PMLR, 30 Oct-01 Nov 2020.

Pascal Klink, Haoyi Yang, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforcement learning via constrained optimal transport. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 11341-11358. PMLR, 17-23 Jul 2022.

Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mural: Meta-learning uncertainty-aware rewards for outcome-driven reinforcement learning. In Marina Meila and Tong Zhang, editors, International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6346-6356. PMLR, 18-24 Jul 2021.

Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guided hindsight experience replay. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Zhenshan Bing, Matthias Brucker, Fabrice O. Morin, Rui Li, Xiaojie Su, Kai Huang, and Alois Knoll. Complex robotic manipulation via graph-based hindsight goal generation. IEEE Transactions on Neural Networks and Learning Systems , 33(12):7863-7876, 2022.

Zhenshan Bing, Hongkuan Zhou, Rui Li, Xiaojie Su, Fabrice O. Morin, Kai Huang, and Alois Knoll. Solving robotic manipulation with sparse reward reinforcement learning via graph-based diversity and proximity. IEEE Transactions on Industrial Electronics , 70(3):2759-2769, 2023.

Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 9902-9915. PMLR, 17-23 Jul 2022.

Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion, 2024. arXiv:2401.02644.

Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning, 2022. arXiv:2208.06193.

Ishan Durugkar, Mauricio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 8622-8636. Curran Associates, Inc., 2021.

Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033. IEEE, 2012.

Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research , 21(181):1-50, 2020.

Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International Conference on Machine Learning , pages 4344-4353. PMLR, 2018.

Tim Hertweck, Martin Riedmiller, Michael Bloesch, Jost Tobias Springenberg, Noah Siegel, Markus Wulfmeier, Roland Hafner, and Nicolas Heess. Simple sensor intentions for exploration, 2020. arXiv:2005.07541.

Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems , 31, 2018.

Dmytro Korenkevych, A Rupam Mahmood, Gautham Vasan, and James Bergstra. Autoregressive policies for continuous control deep reinforcement learning, 2019. arXiv:1903.11524.

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Francis Bach and David Blei, editors, International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 1312-1320, Lille, France, 07-09 Jul 2015. PMLR.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh, editors, International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1126-1135. PMLR, 06-11 Aug 2017.

Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems , 36, 2024.

Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning to act from actionless videos through dense correspondences, 2023. arXiv:2310.08576.

Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion, 2023. arXiv:2303.04137.

Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies, 2023. arXiv:2304.02532.

Hsiang-Chun Wang, Shang-Fu Chen, Ming-Hao Hsu, Chun-Mao Lai, and Shao-Hua Sun. Diffusion model-augmented behavioral cloning, 2023. arXiv:2302.13335.

Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models, 2023. arXiv:2301.10677.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1861-1870. PMLR, 10-15 Jul 2018.

Villani, Cédric and others. Optimal transport: old and new , volume 338. Springer, Berlin Heidelberg, Germany, 2009.

Arjovsky, Martin and Chintala, Soumith and Bottou, Léon. Wasserstein generative adversarial networks. In International Conference on Machine Learning , pages 214-223, Sydney, Australia, 2017. PMLR.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020.

David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association , 112(518):859-877, April 2017.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR.

Calvin Luo. Understanding diffusion models: A unified perspective, 2022. arXiv:2208.11970.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. arXiv:1707.06347.

Supplemental Material

A Conditional Diffusion Model

Diffusion models [51, 54] are usually defined as Markov chains and trained using variational inference. The forward diffusion process gradually adds Gaussian noise to the data, while the reverse diffusion process subtracts the learned noise from the data.

Given the goal data g 0 sampled from the real goal distribution g 0 ∼ G , we can define the forward diffusion process using Markov chains and add a small amount of Gaussian noise in each timestep k according to a variance schedule { β k ∈ (0 , 1) } N k =1 , and generate a sequence of noisy goal data samples g 1 , . . . , g N :

As the index of the timestep k increases, the goal sample g 0 gradually loses its features and becomes an isotropic Gaussian noise when N →∞ .

Instead of sampling data recursively for a given timestep k , we can sample g k in a closed form using the reparameterization trick 6 :

Let α k = 1 -β k and ¯ α k = ∏ N i =1 α i . We can write:

We then merge two Gaussian distributions with different variance N (0 , σ 2 1 ) and N (0 , σ 2 2 ) into a new Gaussian distribution N (0 , σ 2 1 + σ 2 2 ) . Specifically, we can merge the two equations above, where √ 1 -α k + α k (1 -α k -1 ) = √ 1 -α k α k -1 :

Using the equation above, we can sample data in a closed form at any arbitrary timestep k .

As mentioned in the main text, the reverse diffusion process aims to recover the original input data from the noisy (diffused) data. It learns to progressively reverse the diffusion process step by step, and approximates the joint distribution p θ ( x 0: T ) . This process is defined as:

where ϵ k ∼ N (0 , 1) .

Now, we can write:

The reverse conditional probability is tractable when conditioned by g 0 . By Bayes rule, we have:

As we assume that the diffusion model is a Markov chain, the future state depends only on the present state, namely:

where:

We then can write:

where C ( g k , g 0 ) is a constant term with respect to g k -1 and is calculated using only g k , g 0 , and α k . The Eq. (45) shows that at every step g k -1 ∼ q ( g k -1 | g k , g 0 ) follows a normal distribution with mean µ ( g k , g 0 ) and variance Σ( k ) .

Given this, the goal is to learn a denoising model p θ ( g k -1 | g k ) by approximating the ground-truth denoising transition q ( g k -1 | g k , g 0 ) . This can be done by minimizing the KL divergence between p θ ( g k -1 | g k ) and q ( g k -1 | g k , g 0 ) .

The KL divergence between two Gaussian distributions N ( x ; µ x , Σ x ) and N ( y ; µ y , Σ y ) is defined as:

The KL divergence between the denoising model p θ ( g k -1 | g k ) and the ground-truth denoising transition q ( g k -1 | g k , g 0 ) is:

The learning of the denoising model can be written as:

We can rearrange Eq. (19) as follows:

and rewrite µ ( g k , g 0 ) in Eq. (45) as follows:

where ¯ α k = ∏ N i =1 α i .

Then:

We can model the denoising transition mean as follows:

Given this, the minimization of the KL divergence can be written as:

Thus far, we have only modeled the goal distribution p θ ( g ) . To transform this into a conditional diffusion model, we can incorporate the state s information at each diffusion timestep k [55] in Eq. (21):

To conclude, we can model the denoising transition mean by rewriting Eq. (70) based on the state s as follows:

B s T Strategy

In contrast to selecting optimal curriculum goals via bipartite graph optimization, as shown in Eq. 12 in the main text 7 we performed an additional experiment where we considered only the state from the final timestep T , under the assumption that it is closer to achieving the desired goal. The state s T at the last timestep of each episode is then input into the diffusion model, which generates curriculum goals to be achieved in the subsequent episode. The resulting variant is detailed in Algorithm 3, with the RL training process described in Algorithm 4.

As demonstrated by this additional experiment, the assumption that the final state is closer to the desired goal does not always hold true. As a matter of fact, the state at the last timestep in some episodes might be even further away from the desired goal compared to the states in previous timesteps. Consequently, the curriculum goals generated by the diffusion model may not gradually shift from the initial position toward the desired goal. If the state at the last timestep is not progressively moving towards the desired goal, the curriculum goals generated by the diffusion model may jump to different areas within the maze environment. This can lead to a decrease in sample efficiency and a slower success rate. This issue is illustrated in Fig. 6, where it can be seen, that particularly in the case of PointSpiralMaze, the variant of our algorithm that makes use of this strategy (that we call ' s T strategy') converges later than the main described in the main text.

PointUMaze

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

DiCuRL

DiCuRL

sT

(a) PointUMaze

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

DiCuRL

DiCuRL

sT

(b) PointNMaze

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

DiCuRL

DiCuRL

sT

(c) PointSpiralMaze

PointNMaze

PointSpiralMaze

Algorithm 3 Diffusion Curriculum Goal Generator with the s T Strategy

Algorithm 4 RL Training with the s T Strategy

C Experimental Details

For implementing DICURL, we utilized the original implementation of OUTPACE [12] and augmented this codebase with our diffusion model for curriculum goal generation. We conducted our experiments on a cluster computer using an NVIDIA RTX A5000 GPU, 64GB of RAM, and a 4-core CPU. For the PointSpiralMaze environment, the total compute time of the main method and the s T strategy is approximately 23 hours and 5 hours, respectively. For the remaining environments, the main method and the s T strategy require approximately 11 . 5 hours and 2 . 5 hours, respectively.

Baselines. The baseline CRL algorithms are trained as follows:

HGG [19]: We utilized the default settings from the original implementation, which can be found at https://github.com/Stilwell-Git/Hindsight-Goal-Generation .

CURROT [23]: We adhered to the default settings of the original implementation, available at https://github.com/psclklnk/currot .

ALP-GMM [17], VDS [16], PLR [15], ACL [18], GoalGAN [14]: We followed the default settings from the implementation available at https://github.com/psclklnk/currot .

GRADIENT [13]: We used the default settings from the implementation available at https: //github.com/PeideHuang/gradient .

OUTPACE [12]: We implemented the default settings from the implementation available at https://github.com/jayLEE0301/outpace_official .

Training details. All baseline models, except for the GRADIENT method, were trained using the Soft Actor-Critic (SAC) [48]. Although the original implementations of these algorithms primarily use on-policy methods, adaptations were made in the CURROT repository to utilize the off-policy SAC algorithm. This modification, available in the CURROT repository, allows for a more direct comparison of sample efficiency across all models. The GRADIENT method was trained using both the SAC and Proximal Policy Optimization (PPO) algorithms [56]. However, we present only the results obtained with the PPO algorithm, as it outperforms SAC in the three considered maze environments. All our algorithms' hyperparameters used in the experiments are reported in Table 3.

Environment details. In each task, the agent's state s is represented as a vector, comprising its position, velocity along the x and y axes, orientation angle, and angular velocity around the z -axis. The agent's actions are determined by its velocity and angular velocity. The agent starts each episode from an initial state of [0 , 0] . The desired goal distribution is derived by introducing uniform noise to the desired goal position. The desired goal positions are set to [0 , 8] , [8 , 16] , and [8 , -8] , while the dimensions of the map are 12 × 12 , 12 × 20 , 20 × 20 , respectively for the PointUMaze, the PointNMaze, and the PointSpiralMaze task. A task is considered successful when the agent reaches the sampled desired goal within a threshold distance of 0 . 5 .

Detailed results. We report in Table 4 the detailed results in terms of no. of timesteps needed to reach a success rate of 0 . 3 for all the algorithms under comparison. It can be seen that most of the algorithms do not reach this rate within the given budget.

D Generated Curriculum Goals

Fig. 7 and Fig. 8 show the differences between the curriculum goals generated by DICURL, OUTPACE, GRADIENT, and HGG, respectively in the PointUMaze and PointNMaze environment. As shown in the case of PointSpiralMaze reported in the main text, also in these two environments DICURL manages to generate curriculum goals that explore the whole environment more effectively than GRADIENT and HGG.

D.1 Dynamics of the Diffusion Process during Training

Considering the case of the PointUMaze environment, the curriculum goal sets, denoted as g 9 , g 8 , . . . , g 0 in Fig. 10, generated during the initial phase of training ( ∼ 2000 timesteps), are

(a) DICURL

(b) OUTPACE

(c) GRADIENT

(d) HGG

5×105

5×105

4×105

4×105

Time Steps

Time Steps

3×105

3×105

2×105

2×105

1×105

1×105

8×105

8×105

Steps

6×105

6×105

Time

4 ×105

4×105

2×105

2 ×105

obtained through the reverse diffusion process. This process starts from Gaussian noise g 9 and culminates in the final curriculum goal set g 0 , as expressed in lines 5 and 7 in Algorithm 1. It can be observed that the final curriculum goal set g 0 is generated diagonally. This pattern arises due to the combined loss function of the AIM reward and the Q -value, along with the diffusion loss, as expressed in Eq. (11).

The AIM reward function, relative to the desired goal g d , and the Q -value, derived from a sampled mini-batch b from the replay buffer B at timestep ∼ 2000, are depicted in Fig. 11a and Fig. 11b, respectively, using a 10-level contour plot. It can be observed that the reward values increase diagonally, while the Q -value has not yet shown a discernible pattern. As a result, the curriculum goal set generated by the diffusion model exhibits a pattern similar to the reward value.

(a) DICURL

(b) OUTPACE

(c) GRADIENT

(d) HGG

8×105

8×105

6×105

Time Steps

6 ×105

TimeSteps

4×105

4×105

2×105

2×105

8 ×105

8×105

6×105

Time Steps

Time Steps

6×105

4×105

4×105

2×105

2×105

In the mid-training stage ( ∼ 15000 timesteps), the final curriculum goal set g 0 reflects the explored region of the environment, extending up to the top right corner, as shown in Fig. 12. This behavior aligns with the agent's exploration progress.

The AIM reward function in Fig. 13a exhibits higher and better-converged values compared to Fig. 11a, particularly in the top right corner. Similarly, the Q -value in Fig. 13b demonstrates its highest values concentrated from the middle right side to the top right corner. These observations suggest that the final curriculum goal set g 0 has indeed been formed, encompassing the explored area up to the top right corner.

In the near-optimal policy learning phase ( ∼ 30000 timesteps), the final curriculum goal set g 0 encompasses the entire environment, extending from the initial position to the desired goal, as depicted in Fig. 14. This behavior aligns with the agent's progress towards achieving the optimal policy.

The AIM reward function in Fig. 15a appears to be converging to the desired goal more slowly. The Q -value in Fig. 15b demonstrates its highest values concentrated around the desired goal area. These observations suggest that the final curriculum goal set g 0 has indeed been formed, encompassing the entire environment. Furthermore, it appears that the Q -value converges to the desired goal more rapidly than the AIM reward function. This observation aligns with our choice of a higher hyperparameter coefficient ξ q for the Q -value function compared to hyperparameter coefficient ξ r

(a) DICURL

(b) OUTPACE

(c) GRADIENT

(d) HGG

8×105

8×105

6×105

Time Steps

6×105

Time Steps

4×105

4×105

2×105

2×105

8×105

8×105

Time Steps

6×105

TimeSteps

6×105

4×105

4×105

2×105

2 ×105

g8

g7

96

g5

g4

g3

g2

g1

go

for the reward function in the total loss function shown in Eq. (11). This prioritizes the Q -value's influence on shaping the curriculum goals, promoting faster convergence towards the desired behavior.

(a) AIM reward

(b)

Q

-Value

0.10

-0.9

0.05

1.2

0.00

-1.5

-0.05

-1.8

0.10

2.1

0.15

2.4

g8

g7

96

95

g4

g3

g2

g1

06

(a) AIM reward

(b)

Q

-Value

0.6

0

0.3

-10

-20

0.0

-30

0.3

-40

0.6

-50

g8

g7

96

95

g4

g3

g2

g1

06

(a) AIM reward

(b)

Q

-Value

1.0

0.5

8

0.0

-16

0.5

-24

1.0

32

1.5

-40

E Robot Manipulation Tasks

To demonstrate the applicability of our method to different tasks, particularly in robot manipulation tasks, we implemented our approach using the original HGG algorithm [19] repository, which can be found at https://github.com/Stilwell-Git/Hindsight-Goal-Generation . We converted the HGG code from TensorFlow to PyTorch to integrate it with our diffusion model, which is based on PyTorch. We selected two robot manipulation tasks, FetchPush and FetchPickAndPlace, and increased the environment difficulty by expanding the desired goal area. This is shown in Fig. 16c and 16d, where the yellow area indicates the object sampling region and the blue area indicates the desired goal sampling region. The action space is four-dimensional, where three dimensions represent the Cartesian displacement of the end effector, and the last dimension controls the opening and closing of the gripper. The state space is 25 -dimensional, including the end-effector position, position, and rotation of the object, the linear and angular velocity of the object, and the left and right gripper velocity. More detailed information regarding the action space and observation space of these robotic tasks can be found in the Gymnasium library documentation. For these additional experiments, we compared our method with HGG and HER using the DDPG algorithm, to ensure alignment with the baselines, using five different seeds. The results are shown in Fig. 16a and Fig. 16b, respectively for FetchPush and FetchPickAndPlace. Note that in this setting, all RL algorithms (including ours) use a binary reward (i.e., a sparse reward). However, since our curriculum goal

generation algorithm is based on the AIM reward, we implemented the AIM reward function solely to generate curriculum goals while still using the sparse reward setting to train the DDPG algorithm using the diffusion model to generate curriculum goals.

0

5000

10000

15000

20000

Timesteps

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

DiCuRL

(ours)

HER

HGG

(a) FetchPush

0

5000

10000

15000

20000

Timesteps

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

DiCuRL

(ours)

HER

HGG

(b) FetchPickAndPlace

(c) FetchPickAndPlace

Code available at: https://github.com/erdiphd/DiCuRL/tree/robot_manipulation

F Test Success Rate in Maze Tasks (Simplified Visualization)

In Fig. 1 in the main text, we presented the success rate of ten different algorithms (including ours) across the three maze tasks. To facilitate the visual comparison of the algorithms' performance, we provide in Fig. 17 a simplified visualization that reduces the line overlap by showing only the four top-performing algorithms (including ours).

PointUMaze

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

×10 5

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

(a) PointUMaze

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Timesteps

×10 5

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

(b) PointNMaze

0

1

2

3

4

5

Timesteps

×10 5

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

DiCuRL

(ours)

OUTPACE

HGG

GRADIENT

(c) PointSpiralMaze

PointNMaze

PointSpiralMaze

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 6.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the theoretical results are detailed in Supp. Mat. A.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclose all the experimental details along with our codebase.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See Section 5, and Supp. Mat. C and E.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Supp. Mat. C.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide all results in terms of mean and standard deviation.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Supp. Mat. C.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms with the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We summarize the importance and impact of this research in the Introduction.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets (e.g., code, data, models) used in the paper are properly credited through appropriate citations or website URLs.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide our code on GitHub (see Section 5 and Supp. Mat. E), as well as all the details about training and computational setup (see Supp. Mat. C).

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an intuitive way to apply curriculum learning using diffusion based models to learn a goal distribution that can interpolate between the state-visitation distribution to states with high-value and high-intrinsic reward. As a result, the curriculum generates goals that lie at the edge of the states with non-zero occupancy, and higher value/ closeness to the target-goal.\n\nThe technical details are mostly complete and seem sound upon initial reading, I did not delve into the proofs/ derivations in the appendix. But the exposition of how we go from diffusion-models, to AIM, to visitation-count modelling, and to the newly proposed DiCuRL method, is mostly clear. \n\nAlthough there are multiple points of improvements, I think many practitioners will appreciate the author's work.\n\nSTRENGTHS:\nThe technical details are quite clear upon initial reading, even if one is not familiar with AIM or diffusion models. I.e., the new method should be clear enough to reproduce from reading the paper.\n\nWEAKNESSES:\n### Major comments\n - The introduction dumps too much related work together to find the actual point and criticism that the authors want to make on the current state of the field.\n - Before reading the background/ technical details, motivation DiCuRL 1) is unclear how noising/ denoising is unique to helping exploration? Why can't another method (like a VAE, or GAN) do this through modelling the state-visitation distribution. Aren't we just choosing another, perhaps more powerful, generative method? **After reading the full paper:** I disagree that this is a sound motivation, any stochastic method over the state-visitation distribution could achieve this. I agree that modelling the state-visitation distribution is useful as it allows learning of goals that the agent has seen and can reach.  \n - 4.0 Line 222, it is not clear from the text what problem the authors are trying to solve through the graph construction and the optimization of the curiculum goal (Eq. 12). How is the 'optimal' curriculum goal even defined? Eq 12 of course shows the objective, but why do we need this? How is the graph even constructed (meaning the edges), is this fully-connected? Initial reading of this paragraph gives the impression of severe over-engineering of the goal-sampler.  \n - Figure 1 overlaps with table 1 and contains too many overlapping lines to draw a conclusion. This must be improved for presentation. Reduce the number of unnecesary baselines, show these in the appendix.\n - The results section spends most of its time speculating why the baselines perform in a certain way but does not focus on the authors' method. Line 281, states that there is a difference between OUTPACE and DiCuRL, however, neither method statistically significantly outperforms the other. Too much of the experimental setup is moved to the appendix.\n - It is unclear from figure 3 at what point during training this plot was made. Now the baseline methods look arbitrarily bad compared to the authors' method. It is color-coded, but maybe add a colorbar to figure 3 indicating the training episodes.\n\n### Technical comments\n - 3.3 Slight confusion on the reward $r^\\pi_\\phi$, it's good to mention that you're actually learning $f(s)$ and using this to compute $r$.\n - 4.0 Explanation on the mixing parameter $\\bar{\\alpha}_k$ is omitted. Shortly state it in the main text.\n - 4.0 The definition of $g_d$ is too hidden. I infer from Alg.2 that this is supposed to represent the *true* goal distribution. \n - Results, figure 1, table 2. Why plot the standard-deviations? Why not a non-parametric tolerance interval to get a sense of spread, or plot a confidence interval for the expected success-rate?\n\n\n### Minor comments\n - Intro paragraph 1 should be split into separate paragraphs making distinct points. Not a lumpsum of information.\n - Intro paragraph 1, maybe make a distinction between hierarchical RL + curriculum RL for goal-generation. Even if HRL can implicitly generate curriculums, the motivation is often slightly different.\n - Direct reference to papers should be done with the author: 'Person et al., (year) showed ...', not '[1, 2] showed ...'. Or you could write, 'Other studies [1, 2, 3], investigated ...' or something similar.\n- Intro paragraph 2 is not a paragraph but 1 sentence.\n- Figure 3, since DiCuRL is mostly on par with OUTPACE this should be compared in the plot for comparing curriculum goals\n\nQUESTIONS:\n1) Could the authors revise the current version and improve upon (most) of my critiques, then I'd be willing to raise my score.\n2) Will the authors share code?\n\nLIMITATIONS:\nThe authors shortly discuss the limitations of their method, which I mostly agree with.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies curriculum reinforcement learning (RL) in the context of multi-goal RL, which aims to generate a series of goals with increasing difficulty to facilitate guiding learning policies. To this end, the paper proposes a framework that employs a conditional diffusion model that learns to generate a goal conditioned on the current state. The experiments in three maze navigation tasks show that the proposed method can reliably solve the tasks and perform them comparably to existing methods. This work studies a meaningful problem and proposes a reasonable framework. Yet, I am concerned with the limited domain (navigation) and tasks (maze) used for evaluation, the significance of the results, and the limited applicability beyond multi-goal RL, etc. Therefore, I am slightly leaning toward rejecting this paper, but I am willing to adjust my score if the rebuttal addresses my concern.\n\nSTRENGTHS:\n**Motivation and intuition**\n- The motivation for studying curriculum learning for multi-goal RL is convincing.\n- Leveraging diffusion models to generate goals is reasonable.\n\n**Clarity**\n- The overall writing is clear. The authors utilize figures well to illustrate the ideas. \n\n**Related work**\n- The authors provide comprehensive descriptions of existing works in curriculum RL.\n\n**Experimental results**\n- The experimental results show that the proposed method performs comparably to existing methods.\n\n**Reproducibility**\n- The code is provided, which helps understand the details of the proposed framework.\n\nWEAKNESSES:\n**Clarity**\n- The first paragraph of the introduction is unnecessarily long, making it very difficult to follow.\n- While the related work section describes several existing works in detail, it fails to differentiate these works from the proposed method exactly.\n\n**Limited to goal-conditioned RL**\n- The proposed method is limited to multi-goal RL, which requires a given goal. However, in many real-world applications, specifying a goal could be difficult or even impossible, making using the proposed method undoable. I feel it is entirely possible to extend the proposed method to the general RL setup, where only the current state is given. This will greatly increase the applicability of the proposed method.\n\n**Evaluation is limited to the Maze navigation**\n- The proposed method was only compared to existing methods in the Maze navigation tasks, where goals are represented as coordinates. It would be a lot more convincing if the evaluation was also conducted in other domains, such as robot arm manipulation, locomotion, and games. Additionally, evaluating in grid-world navigation tasks can add value to the paper by exploring discrete state and action spaces. \n\n**Significance of the results**\n- According to Figure 1, I am not entirely convinced that the proposed method performs significantly better than the baselines. Also, the plotting scheme makes it difficult to interpret when many curves overlap.\n\n**Related work**\n- The related work section focuses on existing works in curriculum RL yet fails to discuss many works that use diffusion models for RL or imitation learning, including but not limited to\n\t- \"Learning Universal Policies via Text-Guided Video Generation\"\n\t- \"Diffusion Policy: Visuomotor Policy Learning via Action Diffusion\"\n\t- \"Learning to Act from Actionless Video through Dense Correspondences\"\n\t- \"Goal-conditioned imitation learning using score-based diffusion policies\"\n\t- \"Diffusion model-augmented behavioral cloning\"\n\t- \"Imitating human behaviour with diffusion models\"\n\n**Algorithm 2**\n- While Algorithm 2 is titled RL Training, Lines 15-21 are for evaluation/testing, which is a bit confusing.\n\n**Minor errors**\n- L282: It seems that a non-break newline is used here, which gives no space between this paragraph and the next paragraph starting from Line 283.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a novel diffusion model-based curriculum learning approach, called DiCURL, for multi-goal reinforcement learning, namely goal-conditioned RL. The proposed conditional diffusion model leverages a Q-function and a learned reward function based on the Adversarial Intrinsic Motivation principle to incentivize goals that are reachable yet challenging to an RL agent. The paper evaluates DiCURL against state-of-the-art curriculum learning approaches in maze environments with differing maps. In PointUMaze and PointNMaze, DiCURL matches or slightly outperforms OUTPACE, which seems to be the best-performing method in these maze environments. In the most challenging map, PointSpiralMaze, DiCURL outperforms OUTPACE, while the rest of the methods fail to yield an optimal policy at the end of the training.\n\nSTRENGTHS:\n- The related work section is extensive in terms of content and covers most of the recent advances in automatic curriculum learning for RL. The background and methodology sections are also detailed, and the problem setting and the proposed approach are explained clearly.\n\n- The proposed curriculum learning approach is novel as it employs a conditional diffusion model. The idea of leveraging a Q-function and a learned intrinsic reward function to select achievable but challenging goals is intuitive, as well.\n\n- Table 1 highlights the advantages of DiCURL, and the introduction section also supports this table.\n\n- The curricula generated by DiCURL in Figures 2 and 3 (as well as the ones in the appendix) illustrate how DiCuRL yields optimal policies and outperforms some existing methods in evaluated environments.\n\nWEAKNESSES:\n- The introduction section should be improved in terms of writing. Content-wise, it is informative but also too dense. Some of the paragraphs are either too long or too short. Restructuring this section and making it more to the point would improve the readers' experience immensely. \n\n- OUTPACE is the second best-forming automatic curriculum learning method in the evaluated environments. However, the paper does not demonstrate the curricula generated by OUTPACE, unlike the curricula of GRADIENT and HGG in Figure 3, which do not perform as well.\n\n- All environments (point maze domain in MuJoCo with different maps) in the empirical validation section have the same dynamics, low-dimensional state, and action spaces. Although DiCuRL's advantages seem apparent as the map gets more complex, the empirical validation is insufficient to conclude that DiCuRL can outperform state-of-the-art methods in various goal-conditioned domains.\n\n- The roles of loss components related to the Q-function and AIM reward function sound intuitive, yet they are explained briefly. I suggest the authors run an ablation study to highlight their separate contributions.\n\nQUESTIONS:\n- How do Q and AIM rewards differ in a goal-conditioned environment that provides a (sparse) reward for reaching the goal? Could you please give me an illustrative example to highlight how including both in the loss function of the diffusion model is better?\n\n- What is g_d that initializes g_c in Algorithm 2?\n\n- What do colors in figures illustrating curricula stand for specifically?\n\nLIMITATIONS:\nI don't see any explicit limitations regarding the proposed approach and the problem setting of interest other than those discussed by the authors in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces DiCuRL, a novel approach that uses diffusion models to generate curriculum goals for reinforcement learning agents. The method trains a model to capture the distribution of visited states, focusing on those with higher Q-values and intrinsic motivation rewards (i.e., AIM rewards). This approach aims to generate goals at an appropriate difficulty level while guiding the curriculum closer to the desired final goal. DiCuRL employs the Minimum Cost Maximum Flow algorithm to solve a bipartite matching problem to select curriculum goals.\n\nSTRENGTHS:\n- Strong empirical evaluation against competitors (Fig. 1)\n- The paper is information-dense but reasonably well-written. It helps with the comprehension of the proposed ideas\n\nWEAKNESSES:\n- The approach is quite complicated and possibly unnecessarily so. I'd like to emphasize that I did not find any faults with the proposed method. It's just that I do not see how it will scale to more challenging, realistic environments.\n- They missed citing a rich literature on exploration and curriculum RL. For example, see papers [1-5].\n- The reward function for the Maze envs is not provided. Is this dense or sparse reward env? Note that, dense reward would not be a justifiable choice in this case.\n\n\n*References*\n1. Riedmiller, M., Hafner, R., Lampe, T., Neunert, M., Degrave, J., Wiele, T., Mnih, V., Heess, N., and Springenberg, J. T. (2018). Learning by playing solving sparse reward tasks from scratch. In International conference on machine learning, pages 4344–4353. PMLR.\n2. Hertweck, T., Riedmiller, M., Bloesch, M., Springenberg, J. T., Siegel, N., Wulfmeier, M., Hafner, R., and Heess, N. (2020). Simple sensor intentions for exploration. arXiv preprint arXiv:2005.07541.\n3. Nair, A. V., Pong, V., Dalal, M., Bahl, S., Lin, S., and Levine, S. (2018). Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31.\n4. Korenkevych, D., Mahmood, A. R., Vasan, G., and Bergstra, J. (2019). Autoregressive policies for continuous control deep reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2754–2762.\n5. Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., & Stone, P. (2020). Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181), 1-50.\n\nQUESTIONS:\n- In Fig. 3, what do the colours represent? Please be more elaborate. It is not clear at all at the moment\n- In the appendix, it's mentioned that \"The agent starts each episode from an initial state of [0, 0].\" In RL environments, environmental resets can implicitly help exploration [1]. How would DiCuRL + fixed start state fare against SAC only + random start states?\n- How does SAC only perform in the comparisons in Fig. 1?\n- How important is the AIM reward? It is a bit weird to sum the Q value and one-step intrinsic motivation reward. This results in different scales/magnitudes of values, which is why the authors needed to tune the coefficients.\n- To ask the previous question differently, can the AIM reward be substituted with simpler intrinsic motivation rewards like RND [2] or TD-error?\n- It seems SAC + HER would be a lot simpler to use computationally and algorithmically. How does DiCuRL compare against SAC + HER?\n\n\n*References*\n1. Vasan, G., Wang, Y., Shahriar, F., Bergstra, J., Jagersand, M., & Mahmood, A. R. (2024). Revisiting Constant Negative Rewards for Goal-Reaching Tasks in Robot Learning. arXiv preprint arXiv:2407.00324.\n2. Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018). Exploration by random network distillation. arXiv preprint arXiv:1810.12894.\n\nLIMITATIONS:\nPlease suggest how this work can be extended to challenging environments with larger state-action spaces.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}