Continuous Goal Sampling: A Simple Technique to Accelerate Automatic Curriculum Learning
Authors: Mehul Damani
Lerrel Pinto
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
curriculum learning
goal-conditioned reinforcement learning

Abstract: Goal-conditioned reinforcement learning (RL) tackles the problem of training an RL agent to reach multiple goals in an environment, often with sparse rewards only administered upon reaching the goal. 
In this regard, automatic curriculum learning can improve an agent's learning by sampling goals in a structured order catered to the agent's current ability. 
This work presents two contributions to improve learning in goal-conditioned RL environments.
First, we present a simple, algorithm-agnostic technique to accelerate learning by continuous goal sampling, in which an agent's goals are sampled and changed multiple times within a single episode. 
Such continuous goal sampling enables faster exploration of the goal space and allows curriculum methods to have a more significant impact on an agent's learning.
Second, we propose VDIFF, an automatic curriculum learning method that uses an agent's value function to create a self-paced curriculum by sampling goals on which the agent is demonstrating high learning progress.
Through results on 17 multi-goal robotic environments and navigation tasks, we show that continuous goal sampling and VDIFF work synergistically and result in performance gains over current state-of-the-art methods.

Under review as a conference paper at ICLR 2023
CONTINUOUS GOAL SAMPLING: A SIMPLE TECHNIQUE TO ACCELERATE AUTOMATIC CURRICULUM LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Goal-conditioned reinforcement learning (RL) tackles the problem of training an RI. agent to reach multiple goals in an environment, often with sparse rewards only administered upon reaching the goal. In this regard, automatic curriculum learning can improve an agent's learning by sampling goals in a structured or- der catered to the agent's 's current ability. This work presents two contributions improve learning in goal-conditioned RL environments. First, we present a sim- ple, aggorithh-agnosthe technique to accelerate learning by continuous goal sam- pling, in which an agent's goals are sampled and changed multiple times within a single episode. Such continuous goal sampling enables faster exploration of the goal space and allows curriculum methods to have a more significant impact on an agent's learning. Second, we propose VDIFF, an automatic curriculum learn- ing method that uses an agent's value function to create a self-paced curriculum by sampling goals on which the agent is demonstrating high learning progress. Through results on 17 multi-gon robotic environments and navigation tasks, we show that continuous goal sampling, combined with VDIFF or existing curricu- lum learning methods, results in performance gains over state-of-the-art methods.
INTRODUCTION
Recent successes in deep reinforcement learning (DRL) have proven that it can tackle complex sequential decision-making problems for tasks in diverse domains such as robotics, video games, and traffic control (Andrychowicz et al., 2020; Vinyals et al., 2019; Li et al., 2016). Building upon the success of DRL in solving specific robotic tasks, the next step the development of more general-purpose methods that can solve multiple tasks. With this objective in mind, the area of goal- conditioned RL (GCRL) (Schaul et al., 2015) has received increased attention as an extension of standard RL. Here, agents are trained to learn policy that can achieve multiple goals, with sparse rewards being provided when the agent achieves the desired goal (Andrychowicz et al., 2017).
A natural question arises from the GCRL formulation given set distribution of goals, in what order should they be presented to the learning agent? naive strategy is to sample these goals uniformly from the goal distribution. This strategy has two drawbacks. First, humans and other biological agents do not have a random, unstructured order to their learning (Ferster & Skinner, 1957). For example, baby does not start walking before can learn crawl. Using the same intuition, at the start of learning, when an RL agent essentially has a random policy, sampling hard- to-achieve goals will result in no learning signal and wasted samples with high probability. Second, given that the training time is limited, it is often not even possible to adequately cover the entire goal space for high-dimensional problems by random sampling. To address these challenges, eurriculum learning methods structure an agent's learning by organizing the order in which goals or tasks are presented to the agent (Soviany et al., 2022). The fundamental intuition behind many of these methods is to sample goals that are neither too easy nor too hard and are thus maximally informative to the agent. Although driven by the same intuition, the technical objectives used in prior work vary significantly. For example, Matiienn et al. (2019) and Portelas et al. (2020) propose to sample tasks on which an agent has high learning progress (LP), while VDS (Zhang et al., 2020) proposes to sample goals that have high expected epistemic uncertainty.

---

Under review as a conference paper at ICLR 2023
While curriculum learning methods have led to significant improvement vanilla GCRL, they still quite sample inefficient in sparse reward settings, especially for problems with high dimensional goal spaces. One of the reasons behind this is that the standard GCRL setup constrains have a single goal for the entire length of an episode. As a result, episodes where the goal is too hard too easy achieve are non-informative and waste agent interactions. The single-goal framework also restricts curriculum learning methods to sampling goals from the initial state and does not give them the flexibility to change an agent's goal online based on its current state. However, since goals do not affect transition dynamics, this constraint is not required to be enforced but is simply result of the chosen conventional framework. Using this observation and the recent success of curriculum learning methods, this work presents two contributions to accelerate and improve GCRL.
agent to
First and as the main focus of this work, we present Continuous Goal sampling, a simple technique that proposes a novel extension to GCRL which can accelerate a wide range of curriculum learning algorithms. In continuous goal sampling. goals are sampled multiple times in an episode, instead of the standard practice of sampling a goal only at the start of episode. Second, inspired by the success of recent LP-based curriculum methods in multi-task RL (Matisen et al., 2019; Portelas et al., 2020), we reformulate the LP objective to enable it to work in sparse reward settings with random initial states. We achieve this by using a current and a lagged value function to approximate the expected LP of goal. Since value functions are part of most current deep RL algorithms, we reuse them off the shelf for LP computation. Our proposed automatic curriculum learning algorithm. referred as VDIFF, requires no additional learning, has little no computational overhead, and is agnostic to the base RL algorithm.
We present results on set of 14 benchmark manipulation environments (Plappert al., 2018; Gallouédec et al., 2021) and 3 maze navigation environments (Zhang et al., 2020). Through these results, we show that resampling improves the sample efficiency and performance of explicit cur- riculum learning algorithms (VDS and VDIFF), implicit curriculum learning methods (HER), and vanilla RI. algorithms in multiple environments. Finally, we also show that our proposed currieu- lum learning method VDIFF is able to outperform existing explicit curriculum learning methods in GCRL. Anonymized code is available here.
2
BACKGROUND
2.1
GOAL-CONDITIONED RL.
conlicondiionee RL (GCRL) (Schaul et al., 2015) aims train an agent achieve multiple goals in an environment. Formally, it can be described as a finite-horizon Markov decision process (MDP) defined by the tuple (S, G, A, R, T, p. 7). where S is the state space, A is the action space, R: S x A AxG G - IR is the the reward function, T S Sx A x S + [0, 1] is the state transition function, p(so) is the initial state distribution, and y E [o, 1] is the discount factor. At the start of each episode, a goal g is sampled from G and the objective is to learn a policy x(ae ((ae|se.9) g) which maximizes the expected
value V. which is defined as the sum of discounted rewards V ((so, .g) EE 7t R(st, at, g)]- Given the dependence of A on GCRL can also be viewed as the problem of learning a policy π over distribution of reward functions R parameterized by a goal g.
The inherent binary structure of GCRL setup allows for the definition of sparse indicator reward function that indicates an agent has achieved the given goal g (Andrychowicz agent receives reward of ( and is said to have achieved goal g when d(st, )) < e, where d(...) is some distance function in goal space and e the acceptance threshold. all other cases, the agent receives reward of 1. In this work, an episode terminates as soon an agent achieves the goal.
al., 2017). An
2.2 CURRICULUM LEARNING SETUP
In goal-conditioned RL, a goal g is sampled from G according to some probability distribution p(g]so), where 50 is the initial state. The task of curriculum learning method is to design p(g]so) to enable the sampling of meaningful goals that can both improve and accelerate an agent's 's learn- ing. Curriculum learning methods dynamically change p(g]su) an agent's 's policy evolves during training. To make this dependence on policy more explicit, we will denote the goal distribution designed by curriculum method pd (g](s)) Existing curriculum methods generally construct
2

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Continuous Goal Sampling Algorithm
Input: Max Episode Length Tmaz, Resample Frequency R. policy
Initialize: Initial state so ~ p(80), Initial goal Geurvent ~> p² (g]60) [Using equation 6]
for ( = 0, 1...T.Tazz do a2 ~ (Sc Jeurrent)
(1-1, done eneironment.steplo))
done then break
end
mod == 0 then
g' ~ p" (g'))
Di In practice, done using Equation
Jcurrent = g'
end if end for
p" (glsa) & f(s..g), where f is some function encoding the curriculum objective. For example, in VDS (Zhang et al., 2020), f is the epistemic uncertainty of sampling goal g from initial state That is. the objective of VDS is sample goals which have high epistemic uncertainty. Similarly, in ALP-GMM and R-IAC (Portelas et al., 2020; Baranes & Oudeyer, 2009), f is the learning progress (LP). For vanilla goal-conditioned RL (Section 2.1) in which goals are sampled randomly, f reduces to an uniform distribution (g) over the goal space. Finally, note the dependence of y" (glsa) on the initial state 8g. While it possible to drop 8g if the environment always resets a fixed state, such as in GoalGAN (Florensa et al., 2018), that is not the case in many real-world settings.
METHOD
In this section, we first introduce continuous goal sampling, a simple technique to accelerate GCRL, by sampling goals multiple times in an episode. Then, we introduce VDIFF, a new automatic cur- riculum method for GCRL, which utilizes value functions to compute learning progress (LP).
3.1 CONTINUOUS GOAL. SAMPLING
In continuous goal sampling. in addition to setting new goal at the start of an episode (standard practice), we continuously sample and set a new goal every R timesteps in an episode, where R less than the maximum episode length. The main idea behind this is that the standard GCRL framework, which has enjoyed notable success in robotic tasks (Andrychowicz et al., 2017), can be even further extended by removing the constraint of restricting episode single goal. The removal of this constraint is made possible by observing that a goal is parameterization of the reward function and has no effect on environment transition dynamics. We incorporate resampling into our training framework by sampling new goal g' every R timesteps during an episode, where R is referred as the resample frequency. The probability of sampling specific goal g is given by p" (g]ss), where yd (glse) x [( g). described in Section 2.2. The continuous goal sampling algorithm is formally described in Algorithm
3.2 LEARNING PROGRESS BASED CURRICULUM
Existing curriculum methods using learning progress (LP) as an objective have shown great promise in the domain of multi-task RL with dense rewards and low-dimensional tasks (Portelas et al., 2020; Romac et al., 2021). However, they are not directly applicable to goal-conditiona RL, which not only has binary episode returns, but also has random initial states that possibly high dimensional. In this section. drawing on the success LP in multi-task RL, we present VDIFF, an LP-based automatic curriculum learning method for GCRL.
3

---

Under review as a conference paper at ICLR 2023
Algorithm 2 LP-based Curriculum Learning Algorithm
Input: Initial Parameters (æ, (r.VVVV.VT"" 0,.0%,0" Blackbox Learning A Algorithm A, Total Training Episodes Nman. Fit Rate F
Output: Learned policy parameters ΦN Initialize: 8% Bu. 8" -
for i = 0. mm.r do
Sample initial state 50 ~ p(50)
Sample set of N random goals G using (g)
for g E G do
LP(50. ()) = VT (so: gi 0() - VT (so. gi 0*)
end for
Sample ~ p(g]su) [Using equation 6]
Collect trajectory Ts (TT,
Update Parameters:
0i+1 +- "ipdale_Policyy A, di 0, 0')
0,-1 t Update_ (A, A, o2, 0g 0))
8 U pdaae Target__ alue (f++1, 03) [Using equation Generally inbuilt in
if' i mod! F == 0 then
H" 1/
end if
end for Return: ON
3.2.1 CURRICULUM DESIGN
We define the Learning Progress (LP) state and goal tuple (s:.g) as the change in the expected discounted returns of reaching the goal from that state after some specified AT training episodes.
LP(st, g) Vist, g) V'(st.g)
(I)
where V is the current value function and V is the old value function, i.e., the value function AT training episodes ago.
Goals exhibiting high LP under this formulation are the ones in which the agent is progressing the fastest and should be sampled more aggressively. To formalize this notion, we propose to sample goals proportional to their learning progress, i.e. p" (g]ss) x I.P(st,)) g). In addition to allowing self- paced learning for the agent, using value functions to construct a curriculum has little computational overhead since the value functions learned by base RL algorithms can be reused off the shelf.
One downside of using value functions for LP inference is that they can be noisy because i) they are trained over data collected by an agent which is constantly exploring, and ii) because, any iteration, they are trained using a small, random batch of data that could be biased. To address this, we use Polyak averaging (Polyak Juditsky, 1992) to obtain smoothed value functions for LP inference. Smoothed value functions are part of many deep RI algorithms and are referred to as 'target" functions because they are used to obtain the target for temporal difference (TD) updates. LP is now computed as
LP g) VT
VT (8t.g)
where VT is the target value function and VT is the target value function AT training episodes ago. At each training step, the parameters 0' of V are updated using
0' = =al' + (1 a:)
where 9 are the parameters of the value function V, 0' are the parameters of the target value function VT, and a is the Polyak coefficient.
The smoothed target value functions provide cleaner, less noisy LP signal. This does not come for free, as the smoothed update causes V/ to lag behind V. However, in practice, the gains from the reduced noise far outweigh the slowdown from the lag. This is formally defined in Algorithm 2.
Selective Resampling: some situations, the current goal might be optimally challenging for an agent and sampling goal g' will not add value. More formally, it makes intuitive sense
4

---

Selective resampling is Under review as a conference paper at ICLR 2023
sample goal only if the expected gain of sampling new goal is greater than the estimated gain of the current goal. Thus, we sample new goal if
Ba (st, ((t,,'))) g')) > ((
(4)
We use selective resampling where the expected value of sampling a new goal is computed as
N
Es[LP(st,,))
Σ (p(s..g()) x LP(s,,
(5)
z Σ L)P(s LP(s..g0))
[Using Eqn 6]
where is a set of N random goals and is normalization constant. formally described in Algorithm 3.
Our final algorithm, referred to as VDIFF, develops an adaptive, self self-paced curriculum for an agent without any extra learning- -aaedd component or encoded domain knowledge, thus allowing for easy integration into most existing GCRL frameworks.
4 EXPERIMENTAL RESULTS
In this section, we first detail the experimental setup by describing the environments tested, train- ing algorithm and the baseline methods. Next, through series of experiments, we analyze the performance of continuous goal sampling and VDIFF on diverse set of multi-goal environments.
4.1 EXPERIMENTAL SETUP
Environments: We run experiments on 14 manipulation goal-conditioned tasks and 3 maze- navigation environments. These include 12 environments of the OpenAl Fetch and Hand envi- ronments (Plappert et al., 2018) which serve as benchmarks for goal-based RL and 2 tasks (Flipping and Stacking) from the panda-gym environments (Gallouédec et al., 2021). The maze-navigation environments adapted from Zhang et al. (2020). All environments use the sparse reward function detailed in Section 2.1.
Training details:
We use Soft-Actor Critic (SAC) (Haarnoja et al., 2018) as the base RL algorithm for our method. All baseline methods also use SAC with identical hyperparameter settings to enable a fair comparison. VDIFF uses, without alteration, the target value function V' g: 6') of SAC. Additionally, to compute VT" (8,g). we store additional parameters 0" which are synced periodically with 0' described in Algorithm 2. We also augment SAC by incorporating Hindsight Experience Replay (HER) (Andrychowicz et al., 2017), a highly effective relabeling technique that uses future states reached in a trajectory as virtual goals obtain additional learning signal. However, since HER is constrained to work only for off poo-cy algorithms in sparse reward domains, we also present results of SAC with no HER. We train variants of VDIFF:
VDIFF: Uses the LP-based curriculum described in Section 3.2.1, but does NOT use continuous
goal sampling.
VDIFF-R: Uses both the LP-based curriculum and continuous goal sampling.
Baseline methods:
We compare method with the following baseline methods:
Random: Random is simple baseline method in which goals are sampled randomly from the goal space.
HER is a relabeling technique which augments the buffer of base RL algorithms with fictitious data obtain additional learning signal. Since HER is not explicit curriculum method, can be used alongside other explicit curriculum methods. vanilla HER, goals are sampled randomly using uniform distribution U(g) i.e., HER has explicit curriculum.
HER:
5

---

Under review as a conference paper at ICLR 2023
FatchPickAndPiccev1
FetchPush-w1
HandManijuulttellocdo
1gxx
HandMan HandMan ipulstePen-vO
Maze A
199
(Tew 1ee8 (hins
Su00sss RiAA
Sudauss
22
30
05
20
15
3.0
25
300
35
4.0
00
05
20
25
30
35
40
25002 53800 200.0/000000000000000000 20000000000000000000000000000000 www.OOT0000OO0 900000000
Timedtep ep
Timesep
Timedap
Figure I: Results of SAC+HER on 6 selected environments. Note that these plots mainly show the environ- ments on which VDIFF-R improves performance. Complete results can be found in Figure 4. Shaded region represents 95% confidence interval of mean success rate across 6 seeds.
VDS: VDS (Zhang al., 2020) is the state-of-the-ard curriculum method for goal-conditioned RL. It proposes to sample goals with high epistemic uncertainty which is approximated as the disagreement between the outputs of an ensemble of learned value functions. Our implemen- tation follows the official codebase of VDS. However, we implement VDS with SAC instead of DDPG (Lillicrap al., 2015) enable fairer comparison. This is not issue as VDS is independent of the base RL algorithm and treats it as a black box.
4.2 IMPROVEMENTS USING VDIFF AND RESAMPLING
Results of all methods using HER+SAC are presented in Figure 1. In the following discussion, we drop 'HERR' from the names of the explicit curriculum algorithms for brevity. For example, VDIFF actually refers to VDIFF+HER. From the results, we first observe that VDIFF-R shows improved sample efficiency or better performance compared to baseline methods in 7 out of the 17 envi- ronments. In the other 10 environments, VDIFF-R matches the performance of baseline methods. Notably, the performance gains are significant in 3 variants of the challenging Hand Manipulate Block environments. By comparing VDIFF and VDIFF-R, we can conclude that continuous goal sampling is responsible for the observed gains.
We hypothesize that there could be 2 major reasons behind the ineffectiveness of VDS and VDIFF when used alongside HER. First, HER already has powerful implicit curriculum because as an agent's policy improves, the goals it uses for replay naturally shift from simpler to more difficult ones. As generates training data for the agent, it also has a stronger and more direct impact on an agent's is learning. The potential gain of using thus be small. Second, HER augments the buffer through fictitious data which is often biased. Since both VDS and VDIFF use value functions trained this buffer, their generated curricula might be noisy and biased. At the same time, it is worth noting that while HER is constrained to work only in sparse reward settings and with off- off-py methods, both VDS and VDIFF have such restrictions. Hence, it is crucial to understand their performance without HER, which is discussed in Section 4.4.
explicit curriculum method alongside HER could
4.3
CAN RESAMPLING HELP OTHER ALGORITHMS?
Figure presents results comparing continuous resampling variants the baseline methods to their vanilla variants. In the following discussion, drop '+HER' from the names of the explicit curricu- lum algorithms for brevity. For example, VDS-R actually refers to VDS-R+HBR We first observe that both VDS-R and HER-R outperform their corresponding vanilla variants, VDS and HER

---

Under review as a conference paper at ICLR 2023
FatchPickkndPPccew1
FetchPush-v1
HandManipulateBiooci
120
197
10
161
20
Timedep
Timeddep
Timeddp
0.5
Pandaflip-v2
Maze A
SU000 R22A
Sudouss
30
05
20
15
3.0
25
30
35
4.0
00
05
15
20
25
30
35
40
25002 5000000502222000 50000000000000000000000 75000 20003
Timestep ep
Timesep
Timedep
Figure 2: Results for continuous goal sampling variants of HER and VDS on selected environments. Note that these plots mainly show the environments on which continuous goal sampling improves performance over baseline methods. Complete results can he found in Figure 5. Shaded region represents 95% confidence interval of mean success rate across 6 seeds.
6 environments. In the other 11 environments, they match the performances of VDS and HER. This provides further evidence that resampling goals is a general strategy which can accelerate per- formance in goal-conditionss outperform VDS-R R and VDIFF-R. Since HER-R always resamples goals while VDS-R and VDIFF- constant only selectively resample, this leads us the conclusions that i) random resampling frequency is an effective strategy with HER and ii) explicit curriculum methods might have lesser room for improvement in settings with HER (further validated through results in Section 4.4).
RL. Interestingly,
also observe that HER-R is able to marginally
4.4 IS VDIFF EFFECTIVE WITHOUT HER?
In Figure 3, we compare all baseline methods on SAC without HER, We first observe that both VDS and VDIFF are easily able to outperform Random in 6 out of the 12 tested environments. Without the strong implicit curriculum of HER, the role of explicit curriculum methods increases greatly. It is also worth noting that VDIFF is able to outperform VDS multiple environments. Next, and as expected, continuous goal sampling helps further improve the performance of all methods. Finally, it is also interesting to note that not all environments are amenable learning with HER. For example, we observe that though methods using HER generally perform significantly better in most environments, their asymptomatic success rate is lesser than equivalent NO-HER variants for 3 environments (Hand Manipulate Hand Manipulate Pen-Rotate and Fetch Slide). This is possibly because of the biased data that HER adds into the replay buffer. In environments where the performance is affected by such bias, using explicit curriculum learning methods with no HER is the best choice.
4.5
HOW DOES RESAMPLING FREQUENCY AFFECT PERFORMANCE?
To better understand what the optimum resampling frequency (R) is, we run an ablation study over different values for R. These results are depicted Figure 7. Note that Episode Length repre- sents the standard practice of sampling a new goal when the environment is reset. From these results, we first observe that performance is roughly the same for moderate values of R, i.e., Rt = 10, 20. However, performance becomes increasingly unstable as Rt is decreased further and is the worst for the extreme value of R = 1. This is expected for two reasons. First and intuitively, sampling a new goal at every timestep breaks the sequential dacision-making nature of the problem. Second and more technically, if new goal is sampled at some timestep t, then the (s,, o, sy) transition tuple is unsuitable for temporal difference (TD) updates as the reward function has changed. For higher
7

---

Under review as a conference paper at ICLR 2023
FetchPickAndPlace-/
FetchSilde-v1
FetchRush-v1
Agex RasaBrve
07
Success: Rant:
60
20
Timmlep
105
HandW lateEggo-v
52
Su00css
10
Timestep
Timedep
Timestep
Figure Results without HER on all environments. Shaded region represents 95% confidence of mean success rate across 6 seeds. We observe that the explicit curriculum methods VDS and VDIFF easily outperform Random. However, HER is able to outperform other methods on most environments. Interestingly, there are environments in which HER converges to lower success rate compared to VDIFF-R. Full results can he found Figure 6.
values of R, this does not matter as such transitions are relatively rare (even rarer when there is data augmentation from HER which is free of this issue). However, will still be interesting study if removing these rare erroneous transitions from the buffer can further improve learning. We find that setting R 0.2 Max Epiisode Length generally works well for most environments.
4.6 How DOES THE TARGET VALUE FUNCTION OF VDIFF IMPACT PERFORMANCE?
VDIFF uses target value functions compute learning progress (LP) (Equation 2). To understand the impact of using the target function, we run an ablation study in which LP is instead computed using the standard (non polyak-averaged) value functions, which were described in Equation 1. To minimize additional sources of noise and to bring full focus on the explicit curriculum method, we run these experiments without HER. In addition to the success rate, we also plot the estimated LP, which is computed using the appropriate value functions, as described above. The obtained results are depicted in Figure 8. From the results, we observe that standard value functions perform worse compared to target value functions in multiple environments. As discussed in Section 3.2.1, we believe that that this result of the noisiness of the value function. This noise causes significant variation in the LP estimates, which can be directly seen from the LP plots.
5 RELATED WORK
5.1 ORIGIN OF CURRICULUM LEARNING
The origins of curriculum learning are rooted in human and animal cognition. Early experiments showed that animals could be trained more effectively by administering curriculum of programmed learning (also referred to as shaping in cognitive science) (Ferster Skinner, 1957). Inspired by this, Bengio et al. (2009) conceptualized curriculum learning in the context of machine learning by training a supervised learning algorithm on examples of gradually increasing difficulty. This was soon followed by works which developed
active (adaptive) curriculum for supervised learning algorithms (Kumar et al., 2010; Lee & Grauman, 2011; Graves et al., 2017). Due to its parallels with learning in biological agents, RL is one of the most promising domains to apply curriculum learning. With the advent of deep RL, there has been increased interest developing more general- purpose RI. agents that can solve multiple tasks (Narvekar al., 2020; Soviany et al., 2022).
8

---

Under review as a conference paper at ICLR 2023
5.2 CURRICULUM LEARNING IN COAL-CONDITI RL
Curriculum learning for goal-conditioned RI. (GCRL. concerned with the problem of presenting training goals to an agent in an order which is conducive to learning (Schaul et al., 2015; Liu et al., 2022). Florensa et al. (2018) trained a generative adversarial network (GAN) to generate goals that have an intermediate probability of success. Zhang et al. (2020) proposed to sample goals which have high epistemic uncertainty (Section 4.1). Hindsight Experience Replay (HER) is a popular relabeling technique that uses future states reached in trajectory as virtual goals to augment the replay buffer (Andrychowicz et al., 2017). By doing so, it builds an implicit curriculum for the agent as goals used for replay naturally shift from simpler more difficult ones. Curriculum- guided HER (CHER) is an extension of HER which uses curriculum learning to select the virtual goals used by HER for relabeling (Fang al., 2019). Finally, self-play based methods have a multi- agent setup (generally 2 agents) in which one agent proposes increasingly challenging yet achievable goals for the other agent (OpenAI et al., 2021; Sukhbaatar al., 2017; Du et al., 2022; Campero et al,, 2020). Although powerful, self-play based methods are primarily targeted towards open-ended environments, and require a complex setup and a large number of training samples.
5.3 LEARNING PROGRESS BASED CURRICULUM
The fundamental idea behind learning progress (I.P) based curriculum methods is to sample with high frequency the goals/tasks on which an agent is making the most progress. I.P can be considered a form of intrinsically motivated active learning and has been used successfully in diverse applica- tions ranging from developmental robotics (Blank et al., 2005) to classroom teaching (Clement et al., 2013). R-IAC, one of the first active learning algorithms to use LP in sensorimotor learning, used absolute LP (ALP) to split parameter space into sub-regions (Baranes & Oudeyer, 2009). Build- ing on this, Matüienn et al. (2019) presented teacher-studen framework for discrete task spaces in which a teacher (curriculum generator) samples tasks with high LP for the student (RL agent). Portelas et al. (2020) extended this to continuous task spaces by fitting Gaussian Mixture Model (GMM) on (task, ALP) tuples and then sampling a task from a Gaussian chosen proportionally to its mean ALP value. Both these works assume dense reward settings and compute LP using either the nearest neighbor (Portelas et al., 2020) or the change in episode return for a given task (only applicable in discrete task spaces). As result, they are not directly applicable to GCRL, which not only has binary episode returns, but also has random initial states that are possibly high dimen- sional. This work reformulates ALP for GCRL using value functions, as described in Section 3.2. 1. Similar to our method, SPaCE also uses value functions for LP computation (Eimer et al., 2021). However, SPaCE is targeted towards contextual RL and is only applicable in discrete task settings. Additionally, it does not make use of target value functions for smoothing.
6 CONCLUSION AND FUTURE WORK
In this work, presented 2 contributions advance goal-conditioned RL. First, proposed con- tinuous goal sampling, a novel extension of standard GCRL in which goals for an agent are sampled and set multiple times during an episode. Second, we introduced VDIFF, an automatic curriculum learning method for GCRL, which uses learning progress computed through value functions to de- velop self-pacel curriculum for an agent. Through experiments on suite of 17 sparse-reward tasks, we demonstrated the effectiveness of continuous goal sampling as general strategy to ac- celerate learning in GCRL. Finally, we showed that although explicit curriculum methods such as VDIFF can generate meaningful curricula, their role is minimized in settings where Hindsight Ex- perience Replay (HER) (Andrychowicz et al., 2017) can be applied.
Future work will explore different variants and applications of continuous goal sampling. This could include exploring techniques such as dynamically changing episode length and applications such as reset-free RL (Zhu et al., 2020). Another interesting future direction will be to better understand the interaction between explicit curriculum methods and HER, with the ultimate intent of developing explicit curriculum methods that can work harmoniously with HER. Finally, it will also be inter- esting to evaluate explicit curriculum methods such as VDIFF on on-policy algorithms and dense reward settings, and study techniques that can extend VDIFF to multi-task RI. (Romac al., 2021).
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017. 1, 9,
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Adrien Baranes and Pierre-Yver Oudeyer. R-iac: Robust intrinsically motivated exploration and active learning. IEEE Transactions on Autonomous Mental Development, 1(3):155-169, 2009,, 3,
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009.
Douglas Blank, Deepak Kumar, Lisa Meeden, and James B Marshall. Bringing up robot: Funda- mental mechanisms for creating a self-motivated, self-organizing architecture. Cybernetics and Systems: An International Journal, 36(2):125-13) -50, 2005.
Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals, arXiv preprint arXiv:2006. 12122, 2020.
Benjamin Clement, Didier Roy, Pierre-Y Oudeyer, and Manuel Lopes. Multi-armed bandits for intelligent tutoring systems. arXiv preprint arXiv:1310.3174, 2013.
Yuqing Du, Pieter Abbeel, and Aditya Grover. It takes four to tango: Multiagent selfplay for auto- matic curriculum generation. arXiv preprint arXiv:2202.10608, 2022.
Theresa Eimer, André Biedenkapp. Frank Hutter, and Marius Lindauer. Self-paced context evalu- ation for contextual reinforcement learning. In International Conference on Machine Learning, pp. 2998-299222 PMLR, 2021. 9
Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guida hindsight experience replay. Advances neural information processing systems, 32, 2019. 9
Charles B Ferster and Burrhus Frederic Skinner. Schedules of reinforcement. 1957. 1,
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528. PMLR, 2018. 3,
Quentin Gallouédec, Nicolas Cazin, Emmanuel Dellandréa, and Liming Chen. panda-gym: Open- source goal-conditioned environments for robotic learning. In 4th Robot Learning Workshop: Self-Supervised and Lifelong Learning at NeurlPS, 2021. 2,5
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference on machine learning, pp- 1311-1320. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018. 5
M Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. Advances in neural information processing systems, 23, 2010.
Yong Jae Lee and Kristen Grauman. Learning the easy things first: Self Selfep visual category discovery. In CVPR 20/ /, pp. 1721-1728. IEEE, 2011. 8
10

---

Under review as a conference paper at ICLR 2023
Li Li, Yisheng Lv, and Yui-Yue Wang. Traffic signal timing via deep reinforcement learning. IEEE/CAA Journal of Automatica Sinica, 3(3):247-254, 2016. 1
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0299 02977, 2015. 6
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Prob- lems and solutions. arXiv preprint arXiv:2201.08200 2022. 9
Tambet Matisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learn- ing. IEEE transactions neural networks and learning systems, 31(9):3732-3740) 2019. 1, 9, 12
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: framework and survey. arXiv preprint wrXiv:2003.04960 2020. 8
OpenAl OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D'Sa, Arthur Petron, Henrique automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2101.0444 100.:2880..48 2021.
Pinto, Asymmetric self-play for
Matthias Plappert, Marcin Andrychowicz. Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforce- ment learning: Challenging roboties environments and request for research. arXiv preprint arXiv:1802.09444 2018. 5.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838-855 855, 1992.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Fierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference Robot Learning. pp. 835- 853. PMLR, 2020. 1, 3,
Clément Romac, Rémy Portelas, Katja Hofmann, and Pierre-Yves Oudeyer. Teachmyagent: benchmark for automatic curriculum learning in deep rl. In International Conference on Ma- chine Learning, pp. 9052-9063. PMLR, 2021. 3.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxi- mators. International conference on machine learning, pp. 1312-1320. PMLR, 2015. 1, 2,
Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, pp. 1-40, 2022. 1,8
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv: 1703.05077 2017. 9
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value dis- agreement. Advances in Neural Information Processing Systems, 33:7648-7659, 2020. 1, 2, 3,
6. 9
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Ku- mar, and Sergey Levine. The ingredients of real-world robotic reinforcement learning. arXiv preprint 2tt.::2004.1257000 2020. 9
11

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 IMPLEMENTATIONAL DETAILS
This section presents implementational details and minor modifications which enable continuous goal resampling and VDIFF to work effectively in practice.
Sampling Goals:
easily obtain LP(st, g) for any (8c. 9). computing p" p²((g se) over the entire continuous goal space is intractable. To enable sampling from p" (g)st), we discretize the problem by first sampling a set of goals G { n= using (g). the uniform distribution over the goal space G. Then, the probability sampling goal g G
Even though
p (g]st) = (sc.
(6)
where -= E (st, (() is normalizing constant, and f(8,, is the function encoding the curriculum objective (in the case of VDIFF, f(8,, g) ()),)))))..
Multi-task RL is prone catastrophic forgetting, phenomenon where agents forget how to achieve learned goals if the learned goals are not sampled enough or if the agent's policy changes significantly in the process of learning new goals This issue can be exacer- bated by curriculum learning methods because they alter p(g]st) which could result in some goals having very low probability of being sampled. After initial early training for M episodes with LP objective, we samples goals using the absolute learning progress (ALP)-
Catastrophic Forgetting:
pa (glss) [LP(st.)) g)l
We do not use ALP in the early training phase because value functions are initialized randomly to output values close Given that our reward structure has negative rewards, outputs of the value function fall during early training. If we use ALP as an objective from the start, the drop in output values might be falsely detected as catastrophic forgetting, when in fact the initial convergence of the value function. Finally, in actual practice, we find that agents in multi-goal RI are less prone to catastrophic forgetting in comparison to multi-task RL (Portelas et al., 2020; Matiisen et al., 2019). This is expected multi-task RL considers distribution of environments, whereas multi-goal RL more structured and involves a distribution over sparse reward functions in a single environment.
Efficient Updating of V': Since updating VT' at each training step can be expensive, we sync with VT every F episodes, where F is referred to the fit rate.
A.2 ALGORIHTM
A.2.1 SELECTIVE GOAL RESAMPLING
Algorithm Selective Goal Resampling Algorithm
Input: Max Episode Length Tmaz, Resample Frequency R, policy π Initialize: Initial state so p(so). Initial goal Jcurrent p" (g((s) [Using equation 6]
for /. = 0, 1...TTmar maz do a, (8t, Jeurrent)
8t41, Tt-1: done = cnerronmeteeettssselpee)
done then break
end if
if t mod R = 0 then
Sample set of N random goals & using U(g)
[elff(rr g)] Ein [p(sr, (() x f fst. (1)] [[Euaatoo 6]
if Ex(s(st, ()] (ourrent) then
g n" (g'18441)
Jeurrent = end if end if
end for
12

---

Under review as a conference paper at ICLR 2023
A.3 COMPLETE EXPERIMENTAL RESULTS
A.3.1 IMPROVEMENTS USING VDIFF AND CONTINUOUS GOAL SAMPLING
FatchPicdAndPPacaww1
FatchSlide-w1
FotchPush-v1
160
Agert dum VT1 M01
100
Fasaay (1500)
1Tn 1460
CALCOMNC Ride
22
100003
100000
330303
450000
500000
100000
203030
300000
600000
570000
338380
200000
303930
50000
Timestep
Timestop
Timestes
Hlndd www.daaiiilleellleeooeeeeoZZZZ.
20.3
Acar
15.2
1003) Vircuees
Succes
1Tn 1999
12.5
VTC
101
2.5
6.0
0.0
05
1.0
3.5
3.3
3.5
3.0
35
4.0
0.3
0.5
4.
50
35
1.0
1.5
2.5
Timestep
Timeskop
landdWanipultteegg
Aear
62
1in 1499
124 101
40
0.0
0.2
666
1.
6.
0.5
1.0
2.0
25
3.0
3.5
4.0
0.0
0.2
3.5
0.8
linddtanipuuateeorRotteeee-
HlandReach-vo
60
Sycues Rar
002 6.26
202
0.5
Ldd
1.00
210
PandaStack-v2
PandaFlip-22
Maze
100
Succees Sare
60
23003 scose www.raaiceee//eee...... 70000000000
Timestes
Mizze
Maze
02
54000000 Rath
2.0
SuccOssE
15 15
1001 12
112
0.5
Figure 4: Results for SAC+HER on all environments. Shaded region represents 95% confidence interval across 6 seeds. We notice that VDIFF-R improves performance in environments and matches the performances of baseline methods the other environments.
13

---

Under review as a conference paper at ICLR 2023
A.3.2 EFFECT OF CONTINUOUS GOAL SAMPLING ON BASELINE ALGORITTMS
FeteMPickAndMMccew
FetchSSide-wvl
FetchPush--l
100
60
20
HandManipllateBllcc.vvv
SANC0EN Tate
30
05
10
1.5
2-2
200
25
40
25
6.
20
35
16
15
232
255
30
35
40
Timestep
Timeskop
Timestop
#andthorppuateloccooaaasovv
HandManiipllteeEgooo
102
62
100
Timeetep
Tieedep
Timestes
HandMaaiiudatseeenv0
HondReach-vD
Sapccese Sude
40
VECHEE
002
0.35
6.50
3.75
1.00
1.50
202
2.00
0.25
0.50
6.75
1.50
220
PandaStack-v2
Pandafilp--2
Maze A
100
66
Sucres Rade
23
20
25003 10000 n-arsiitice /ot.ciieee../..... 2102
Timmsttop
TimesAop
Timestep
03
2.0
SU000S
22
TimesAop
Figure 5: Results showing continuous goal sampling for HER and VDS on all environments. Shaded region represents 95% confidence interval of mean success rate across 6 seeds. We notice that the continuous sampling variants HER-R RR and VDS-R outperform their corresponding vanilla variants in multiple environments.
14

---

Under review as a conference paper at ICLR 2023
A.3.3 RESULTS IN THE NO-HER DOMAIN
FetchPPisAAnPPacce
FetchSilde-v2
FetchPush-vi
10)
AMM
1948
Succos 8222
VOV
0.03
0.23
0.33
0.33
1.33
2.50
L13
2:0
5.00
3.2
030
0.75
1.00
1.3
123
175
2:00
Timestop
Timestop
Timestop
HandMMaripulttelllakkkooo
HandHManipaitteeiookRtataa
25
3uccmm Patla
Success Sale
166
140
Timestep
HandManipulateeeog VD
SaanccMMnniuussttggggotttac vO
439m
Ager
Landinnd
Rancome
LEE
50
Su00sss R.ZAL
80
025
038
0.35
1.00
1.25
1.50
1
2:50
0.0
0.5
1.0
1.5
2.3
2.5
35
35
c5
Timezop
Timestop
Timestop
HardHManipulateeer--o
HandReach-v0
52
Vtere
Succeb FEL
VC1A
Succos Rat
9754
23
8.00
0.25
030
1.00
3.2
130
375
2.00
Timestop
Timestop
Timestep
Figure 6: Results without HER all environments. Shaded region represents 95% confidence of mean success rate across seeds. We observe that the explicit curriculum methods VDS VDIFF easily outperform Random. However. HER is able to outperform other methods on most environments. Interestingly, there are environments in which HER converges to a lower success rate compared to VDIFF (Hand Manipulate Pen, Hand Manipulate Pen-Rotate and Fetch Slide). Note that in some environments, HER crashes after converging. Although this does matter practice an adverse effect caused by the bias of HER.
save/pick the model the hest success rate, it example of
15

---

Under review as a conference paper at ICLR 2023
A.3.4
ABLATION ON RESAMPLING FREQUENCY
FeschPickAndPlace-v
FetchSlide-v1
FetchPush-v1
100
100
Success Rate
Success Rate
Success Rate
60
Agent
40
Rm20
Rm20
[t=20
R-Ep
R=Ep
R-Ep Lon
400000
500000
400000
300000 400000 500000
Timestep
HandMarijulattellcckkvv
Agent
Agent
Agent
R=:1
60-
R=11
R=
Success Rele
R-5
Succes Rate
R=5
Succeds Rate
R
20
20
ReEp
Timestop
Timestop
Timestep
1e5
HanddWarrppaateBloRRRoaaazzo
HandManipulateegggoo
HandMManipplateEggRooooooo
Success Ratz
Success Rate
R-5
Success Rate
60
R-20
R-10
R-20
I-100
40
Ri
40
Re
R-10
20
R-20
Timestop
Timestop
Timestap
HandMaanipllteePe--v0
HandReach-v0
Success Rate
Success Rate
Success Rate
30
Agest R-
R-1
Agent
Agent
R-1
Rid
Ru5
[1-20
Ep
RaEp Len
0
0
1e6 0.0
0.2
0.4
0.6
0.8
1.0
Timestep
1e5
Timestep
Timestep
166
Figure 7: Ablation analysis on the choice of resample frequency for all environments. All methods use VDIFF+HER with identical hyperparameters. R = Ep Len represents the standard practice of sampling new goal when the environment is reset. Shaded region represents 95% confidence interval of mean success rate across 3 seeds. We observe that performance becomes increasingly unstable R decreased.
16

---

Under review as a conference paper at ICLR 2023
A.3.5 ABLATION ON THE TARGET VALUE FUNCTION OF VDIFF
Success Rate
Estmatted
fantan- VDIT-3100u5
Sandera (eseokampp
F-00 PICAAIAPPPPOO
ITA torn
M
60
20
0:33
0.60
3.03
0.25
0.3
D17
0.30
9.25
830
0:75
1.00
1.25
133
Timessopp
300
Timrassp
Success
Estimated
(181-1116
244
20
325
208
Success Rate
Estimated LP
Agart
100
Fo0h V2
3.02
3.25 0.3
0:30
2.25
030
0:39
1.00
1.25
1.37
3.75
Timestep
166
Timestep
Estimated
2.5
(urn)
0.25
2.5
6.00
6.12
2.4
0000
32
Timeszop
Success Rate
Estimated LP
255
VDaT-3a de holat Aveng no!
ITA the Poyaconmpine
2.5
25
Timmsstpp
30e
Timescpp
Estimated
Sandurvk Vial-- (ual)
255 ma
35
Timestep
Timedep
17

---

Under review as a conference paper at ICLR 2023
300
Apor
Lo Pope ceeaping
120104 Aoraar
Orggn
3.5
3.5
0.5
0.0
05
3.2
4.5
20
30
35
4.0
00
6.5
2.0
2.5
6.0
Timestep
Timeetep
Apmr
2.30
0.25
1201044 Noraaaii
0.25
Orgion
0.23
3.20
0.15
3.00
602
Timestepp
Timestep
Success
Estimated
Apmt
haya Aveeane)
Uirrra Averaciing
Origin
Timestop
Success
Estimated
Agait
Agart
2.4
Averaciing
Timestop
Timeslop
Success
Estimated
Apart
11erra rovel Averacind
Timestop
Timsslop
Success Rate
Estimated
1gel
Apart
6.6
0.
05
1.5
2.0
4.0
00
3.0
2.0
2.5
3.2
3.5
4.0
Figure 8: Ablation analysis the target value functions of VDIFE. Shaded region represents 95% confidence interval mean success rate mean estimated LP across seeds. We observe that standard value functions lead to noisier estimated LP and perform worse compared to target value functions in multiple environments.
18


Reviews:
[{"review_id": "wP1u5WJEHA9", "reviewer": "Reviewer_r5BM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses learning in goal-conditioned RL problems, and presents two contributions. The first is a simple algorithmic technique that continuously resamples goals within an episode, instead of only between episodes, to speed up learning. The second is an automatic curriculum learning method driven by the agent’s value function called VDIFF. VDIFF samples goals for which the agent is demonstrating learning progress. The effectiveness of these techniques, both separately and combined, is supported by empirical experiments in simulated robotic environments.", "summary_of_the_review": "Overall, the paper presents two clear new algorithmic ideas, but the presentation and discussion of the experiments is lacking. Therefore I would argue to reject.\n\nThe paper could be improved by cleaning up the presentation of the experiments and expanding the discussion.\n", "strength_and_weaknesses": "The main strength of this paper is that it presents two clear and testable algorithmic ideas. The ideas are also relatively simple, which would make them easy to implement. Thus, this paper could be of interest to a broad range of RL practitioners.\n\nThe main weakness of this paper is the presentation and discussion of the empirical results.\n\nWith the first set of experiments, for example, the learning curves for only 9 of the 17 environments are shown. In 6 of the 9 shown, VDIFF outperforms the baselines, but in reality VDIFF outperforms the baseline only 6 of 17 environments (after a close look at the appendix). This disparity is misleading, and is not adequately highlighted. However, it is worth noting that VDIFF is never worse than the baselines (except maybe in 1 case.)\n\nIt would be useful to summarize all the data somehow in the main text, instead of including only cherry picked plots.\n\nIf possible, it would be useful to increase the number of seeds used in the experiments to at least 10, but 30 would be ideal. More seeds would help to support stronger conclusions.\n\nHow is the 95% confidence interval used in the plots computed? Is the distribution of the learning curves assumed to be normal? Is this actually the case?\n\nMinor: The colors used in the plots make the lines hard to distinguish. Consider changing the colors to something with more contrast or even label the lines directly instead of using a legend.\n\nIn Figure 1, what is the algorithm labeled Random? It not clear to me from the text.\n\nThe paper uses the term “vanilla” throughout to mean “plain”. I don’t think that this term is useful or descriptive and can be confusing. It would be better to simply label each algorithm based on its components, e.g. “SAC with HER”.\n\nIn the last paragraph of Section 3.2.1 the paper says that using the smoothed updates provide a cleaner, less noisy LP signal, and that in practice the gain outway the slowdown introduced. Could this be further discussed? Are there experiments that compare the two cases? How big is the difference in practice?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "g_3NIbKuY6", "reviewer": "Reviewer_UKQZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed two methods to improve auto curriculum learning strategies in the RL settings where multiple goals are available with no change to the environment dynamics:\n\n1. Continuous goal sampling, where the goal of the agent changes every R < episode length steps\n2. A curriculum based on \"learning progress\", which chooses to sample tasks (described here by a tuple (initial_state, goal) proportionally to the improvement in the value function within dT episodes, and to smooth these VFs using Polyak averaging.\n\nThe method is evaluated on a number of manipulation and maze navigation tasks, with the performance compared with VDS and HER. In the basic version, authors propose to use their method (called VDIFF) together with HER, but they also provide ablations on the effect of using HER with the conclusions that in some of the tasks VDIFF improves over HER and VDS, and in others VDIFF makes the same gains (ie. performance of HER is comparable to VDIFF which is better than the lack of both).", "summary_of_the_review": "The authors propose two improvements to the curriculum learning for goal-conditoned RL. One of them is simple but novel, the other one is a marginal improvement over previous work (SPaCE). The experimental results are solid and convincing. \n\nI think the goal sampling idea would be an interesting addition to the conference, so I lean towards acceptance. The strength of my sentiment is weak due to the second, less novel part and the fact that the idea is something many would have tried even if they didn't read the paper (yet I haven't managed to locate it among previous literature).", "strength_and_weaknesses": "## Strengths\nContinuous goal sampling is a simple, but a reasonable idea. It fits well within the Zone of Proximal Development (ZPD) framework[1] which states that tasks presented to the agent should be neither too easy nor too difficult. I see the standard \"termination at the reward\" as satisfying the \"not too easy\" part of the ZPD motivation, and authors \"change the goal every R steps\" naturally rejecting \"too hard\" tasks (if there is no reward in R steps, the goal gets rejected).\n\nThe experiments are performed in many (current, external to the paper) domains and the baselines of variants of VDS and HER are chosen appropriately. The paper tells a coherent story, supported by many experimental results.\n\n## Weaknesses\nLearning progress seems to only differ from SPaCE through the use of Polyak averaging. Authors claim that \"SPaCE is targeted towards contextual RL and is only applicable in discrete task settings.\", however I think SPaCE could be a reasonable additional baseline to try. Alternatively, the addition of Polyak averaging could be ablated, to understand the impact of the novel part proposed by this paper.\n\n## Smaller suggestions\n\nI would like to see HER performance on the Figure 3. Authors suggest that in many tasks, a further progress is difficult compared to a baseline of HER; it would be good to confirm that (where) the gains of HER are reproduced by VDIFF. It would make it clear whether the conclusion is:\n\n1. Sometimes HER is better, sometimes their method, best to use both, or\n2. VDIFF (with no HER) is always better, but sometimes HER is enough.\n\n## Longer-term, future work suggestions\n\nI find changing goals without resetting the environment only partially symmetric to \"finishing episode at the reward\"[2], as ending the episode on reward resets the environment, whereas for changing the goal the env is not reset. There are two alternatives which would treat both constraints in the same way:\n\n1. Change \"goal sampling\" to reset the environment when there is no reward in R steps, or\n2.  Don't reset the environment at the reward. Rather, continue running the environment, but cut the bootstrapping beyond the first reward, to make the agent only optimize \"time until the first reward\" and not keeping the reward satisfied later.\n\nIt's unclear how the proposed method deals with the situation where some tasks are inherently harder (=> require more time to solve optimally) than others, and thus require different Rs.\n\n[1] Vygotsky, [Interaction Between Learning and Development](https://innovation.umn.edu/igdi/wp-content/uploads/sites/37/2018/08/Interaction_Between_Learning_and_Development.pdf)\n\n[2] I understand authors made no claims to the symmetry here, but for the future work, I find it interesting to analyse the situation in more detail, which would make the results even stronger in my mind.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "gGXt1mGo5L", "reviewer": "Reviewer_aTaV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on goal conditioned RL and introduces two algorithmic contributions:\n* goal resampling within an episode (called continuous goal resampling)\n* a automatic curriculum learning method called VDIFF that samples appropriate goals based on learning progress (defined as the change in predicted (target) value for the same state and goal after some number of updates)\nThe paper demonstrates empirically that goal resampling is beneficial for VDIFF and other goal-conditioned RL methods. VDIFF is similar to other methods when hindsight experience replay (HER) is used but performs better without HER.", "summary_of_the_review": "The paper proposes two orthogonal contributions. To my mind it is written in a slightly misleading way and does not do enough to justify why the setting in which VDIFF-R performs well (no HER) is interesting and relevant. If there is an interesting setting where HER cannot be used then the evaluation should at least partially be in that setting. ", "strength_and_weaknesses": "Strength:\n* goal resampling seems like a generally applicable and useful idea.\n* VDIFF-R is empirically promising in the setting without HER\n\nWeaknesses:\n* There are two algorithmic contributions that are orthogonal. The first figure is misleading and I would prefer it if the results of figures 1 & 2 are combined in one figure. Since the resampling can be applied to other methods those should be present in the first comparison\n* Related to the first weakness, I think the abstract is misleading and contains not enough nuance. \"continuous goal sampling and VDIFF work synergistically and result in performance gains over current state-of-the-art methods\" does not seem quite right when VDIFF-R does not outperform HER-R or VDS-R.\n* I am not particularly convinced that settings without HER are very interesting. What is a setting where HER cannot be used? If that is where VDIFF shines then there should be an evaluation in such a setting directly rather than evaluating in a setting where HER can be used.\n* I think it would be good to include the algorithms in the main text if possible. Similarly, I think the selective resampling mentioned in the appendix ought to be mentioned in the main text as it adds complexity.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "48COz1UAuox", "reviewer": "Reviewer_5Ze8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to accelerate and improve goal-conditioned RL. The main contribution is the continuous goal sampling technique proposed. The idea is very simple. Instead of sampling a goal only at the start of an episode, we can sample a new goal every fixed number of timesteps within an episode. The second contribution is the automatic curriculum learning algorithm proposed, which uses learning progress estimated through value functions to create a self-paced curriculum. These two techniques are both not tied to the base RL algorithms.  ", "summary_of_the_review": "This work is well-motivated and the proposed methods are simple and easy to understand. Based on the experimental results, the main proposed technique continuous goal sampling seems to be quite effective in a variety of tasks tested, even though its idea is so simple. However, I'm not sure if the proposed method is compared against baseline methods in a fair way as the authors claimed, making it hard for me to judge how effective the proposed method really is. Also, I think the limitations of continuous goal sampling need to be clearly discussed. ", "strength_and_weaknesses": "- Strengths:\n  - The proposed continuous goal sampling technique and automatic curriculum learning algorithm are very simple and not tied to the base RL algorithms. \n  - The paper is well-written and easy to follow overall. \n- Weaknesses:\n  - In continuous goal sampling, just sampling a new goal every *fixed* number of timesteps seems to have some obvious drawbacks/limitations (and the paper dose not seem to discuss it). It does not really take into account the agent’s current learning progress or the state it is in, but just blindly changing goals when the time meets the pre-designed fixed condition. I could easily imagine some scenarios where this might hurt the sample efficiency, e.g., when the agent is close to reach the current goal and receive the highest reward, the goal (reward function) is suddenly changed. Then what the agent learns before could be wasted.\n  - The technical novelty of the paper does not seem significant enough (e.g., sampling goals based on the agent's learning progress has been done before as mentioned in the paper). \n  - There seems to be some problems with the hyperparameters settings used in the baseline methods in the experiments. Specifically, in the experiments, the paper mentions \"All baseline methods also use SAC with identical hyperparameter settings to enable a fair comparison.\" I do not understand why the baseline methods have to use the *same* hyperparameter settings as the proposed method to enable a fair comparison (if I understand it correctly). It is common to do separate hyperparameter tuning for different methods when comparing their performance in deep learning. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject