Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learning
Authors: Nicolas Castanet
Olivier Sigaud
sylvain lamprier
Venue: Submitted to ICLR 2023 (2023)
Keywords: Exploration
Goal-conditioned Policies
Automatic curriculum
Stein Variational Gradient Descent

Abstract: Multi-goal Reinforcement Learning has recently attracted a large amount of research interest. By allowing experience to be shared between related training tasks, this setting favors generalization for new tasks at test time, whenever some smoothness exists in the considered representation space of goals. However, in settings with discontinuities in state or goal spaces (e.g. walls in a maze), a majority of goals are difficult to reach, due to the sparsity of rewards in the absence of expert knowledge. This implies hard exploration, for which some curriculum of goals must be discovered, to help agents learn by adapting training tasks to their current capabilities. We propose a novel approach: Stein Variational Goal Generation (SVGG), which builds on recent automatic curriculum learning techniques for goal-conditioned policies. SVGG seeks at preferably sampling new goals in the zone of proximal development of the agent, by leveraging a learned model of its abilities and a goal distribution modeled as particles in the exploration space. Our approach relies on Stein Variational Gradient Descent to dynamically attract the goal sampling distribution in areas of appropriate difficulty. We demonstrate the performances of the approach, in terms of success coverage in the goal space, compared to recent state-of-the-art RL methods for hard exploration problems.

Under review as a conference paper at ICLR 2023
STEIN VARIATIONAL GOAL GENERATION FOR ADAP- TIVE EXPLORATION IN MULTI-GOAL REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-goal Reinforcement Learning has recently attracted large amount of re- search interest. By allowing experience to be shared between related training tasks, this setting favors generalization for new tasks at test time. However, in settings with discontinuities in the goal space (e.g. walls in maze) and when the reward is sparse, majority of goals are difficult to reach. In this context, some curriculum over goals is needed to help agents learn by adapting training tasks to their current capabilities. this work we propose a novel approach: Stein Variational Goal Gen- eration (SVGG), which builds on recent automatic curriculum learning techniques for conliconditioned policies. SVGG samples goals of intermediate difficulty for the agent, by leveraging learned predictive model of its goal reaching capabilities. In that aim, it models the distribution of goals with particles and relies on Stein Variational Gradient Descent to dynamically attract the goal sampling distribution in areas of appropriate difficulty. We show that SVGG outperforms state-of-the-an multi-goal Reinforcement Learning methods in terms of success coverage in hard exploration problems, and demonstrate that our approach is endowed with useful recovery property when the environment changes.
INTRODUCTION
In standard Reinforcement Learning (RL), agents learn a policy to optimally achieve a single task. By contrast, in Multi-goal RL (Kaelbling) 1993), they address set of tasks by having policies conditioned on goals, where each goal corresponds to an individual task. The resulting goal-condin coal-conditioned policies offer an efficient way for sharing experience between related tasks (Schaul et al., 2015 Pitis et al. 2020b Yang et al. 2021). In general, the corresponding agents are equipped with a capability to sample goals from some goal space, but do not know whether given goal can be achieved or not. Besides, the general ambition in the multi-goal RL context is to obtain an agent able to reach any goal from desired goal distribution and to do reliably and efficiently. This is particularly challenging in settings where the desired goal distribution is unknown at train time, which means discovering the goal space by experience and optimizing its coverage without any prior knowledge.
To avoid deceptive gradient issues, multi-goal RL often considers the sparse reward context, where the agent only obtains non-null learning signal when the goal is reached. In that case, the multi- goal framework makes it possible to leverage Hindsight Experience Replay (HER) (Andrychowicz et al. 2017) which helps densify the reward signal by relabeling failures as successes for the goals achieved by accident. However, in settings with discontinuities in the goal space (e.g., walls in maze), or in hard-exploration problems where the long task horizon results in an exponential decrease of the learning signal (Osband et al. 2016), many goals remain hard to reach.
In these more difficult contexts, and without any desired goal distribution at hand, we want to maximize the performance of the agent on all feasible goals, that we call the success coverage. This metric encompasses the capacity of the agent to explore and master every goal in the environment. To do so, the selection of goals must be structured into curriculum to help agents to explore and learn progressively by adapting training tasks to their current capabilities (Colas et al. 2018). The question is: how can organize curriculum of goals that maximize the success coverage

---

Under review as a conference paper at ICLR 2023
A first approach consists in focusing novelty, with the objective of expanding the set of achieved goals Pitis et al. (2020b), Pong et al. (2019). Warde-Farley et all (2018). Nair et al. (2018). This leads to strong exploration results, but success coverage only optimized impliitly. Another strategy is to bias the goal generation process toward goals of intermediate difficulty (GOIDs) that will intuitively provide strong learning signal to the agent Florensa et al. (2017) Racaniere et al. (2019) (Sukhbaatar
et al. (017)) Zhang et al. (2020) By aiming at performance, those methods target more explicitly success in encountered goals, but benefit from implicit exploration.
In this work, we propose a novel method which provides the best of both worlds. Our method, called SVGCT learns model of the probability of succeeding in reaching goals, and targets goals whose success is the most unpredictable. To model such a distribution over the goal space, we rely on set of particles where each particle represents a goal candidate. This set of particles is updated via Stein Variational Gradient Descent (Liu & Wang 2016) to fit our objective of goals of intermediate difficulty. Due to the optimization properties of SVGD, in the absence of goals of intermediate difficulty, the current particles will repel one another and foster exploration. We use this feature to demonstrate that SVGG possesses a very useful recovery property that prevent from catastrophic forgetting and enables the agent adapt when the environment changes during training.
BACKGROUND AND RELATED WORK
In this paper, we consider the multi-goal reinforcement learning setting, defined a Markov Decision Process (MDP) Mg =< S, T, A. 9. Rg >. where S is a set of states, 7' is the set of transitions, A the set of actions and the reward function Rg is parametrized by a goal g lying in the d-dimensional continuous goal space G Rd. In our setting, each goal g is defined as a set of states Sg S S that are desirable situations for the corresponding task, with states in Sg being terminal states of the corresponding MDP. Thus, a goal g is considered achieved when the agent reached at step any state E Sy. which implies the following sparse reward function Ry S + {0; 1] in the absence of expert knowledge. With I the indicator function, Rg(*t, Ge, 8t+1) T(8++1 E S,) for discrete state spaces and R, (8e, Ge: 8t+1) 7(ming. ((S,) || 8" 1l2 < 8) for discrete ones, where 8 is distance threshold.
Then, the objective is to learn a goal-conditioned policy (GCP) π S x G - A which maximizes the expected cumulative reward from any initial state of the environment, given a goal g E 9: x arg max; Egrp, (r) EE= >'r:]. where r Ry as st+1) stands for the goal-conditioned reward obtained at step l of trajectory T using goal g. y is discount factor in J0; 1 and Pa is the distribution of goals over G. In our setting we consider that Py is uniform over S (i.c., no known desired distribution), while the work could be extended to cover different distributions.
2.1
AUTOMATIC CURRICULUM FOR SPARSE REWARD RI.
Our SVGG method addresses automatic curriculum for sparse reward goal-conditioned RL (GCRL) problems and learns to achieve a continuum of related tasks.
Achieved Goals Distributions Our work is strongly related to the MEGA algorithm (Pitis et al. 20205), which (1) maintains a buffer of previously achieved goals, (2) models the distribution of achieved goals via a kernel density estimation (KDE), and (3) uses this distribution to define its behavior distribution. By preferably sampling from the buffer goals at the boundary of the set of already reached states, an increase of the support of that distribution is expected. Doing so, MEGA aims at overcoming the limitations of previous related approaches which also model the distribution of achieved goals. For instance, DISCERN (Warde-Farley et al.) 2018) only uses a replay buffer of goals whereas RIG (Nair et al.) 2018) and Skew-fit (Pong et al.] 2019) rather use variational auto-encoding (Kingma & Welling 2013) of the distribution. While RIG samples from the modeled achieved distribution, and DISCERN and Skew-Fit skew that distribution to sample more diverse achieved goals, MEGA rather focuses on low density regions of the distribution, aiming to expand it. This results in improved exploration compared to competitors. Our work differs from all these works as they only model achieved goals, independently from which goal was targeted when they were
For Stein Variational Goal Generation. Code and instructions to reproduce our results will be made publicly available httpp:///annnymoooss...eeeeeeeee vatarrratttiiaaaaaaitaaall Variational -oal--Generationn- A6F
2

---

Under review as a conference paper at ICLR 2023
achieved, whereas we model the capability of reaching target goals. This makes a strong difference since, while MEGA selects goals at the frontier of what already discovered, nothing indicates that goals g closer to the mode of the distribution can be achieved when they are targeted. MEGA is also prone to catastrophic forgetting and limits exploration to goals present in the replay buffer.
Adversarial Goal Generation Another trend proposes to adversarially learn goal generator, that produces targets that are at the frontier of the agent's 's capabilities. In that vein, GoalGAN (Florensa et al. 2018) simultaneously learns a discriminator to sort out non GOIDs or generated goals from GOIDs in the buffer of achieved goals, and a generator that aims at producing goals that fool the discriminator. While appealing. the approach is prone to instabilities, with a generator that may usually diverge far from the space of reachable goals. Setter Solver (Racaniere et al., 2019) stands as an extension of GoalGAN, where a goal setter is learned to provide goals of various levels of difficulty w.r.t. judge network (similar goals remain close to already achieved goals, and are diverse enough to avoid mode collapse. This approach suffers from relying on invertible networks map from the latent space to the goal space, which severely limits the modeling power, and can reveal problematic for high dimensional goal spaces, e.g. images. Asymmetric self-play (Sukhbaatar al. 2017) is another way to generate goals, with teacher agent seeking to produce goals that are just beyond the capabilities of the student agent. Both teacher and student learn simultaneously, with an equilibrium of adverse rewards determined on their respective time to go. However, this balance is hard to maintain, and many useful areas are missed. Finally, -uriosity-driven exploration (Pathak et al. 2017) could be placed in that adversarial category, with inverse model that aims at learning compact latent representation space from which it is still possible to predict the dynamics of the environment. The agent is then rewarded for exploring areas whose dynamics are hard to predict. Nevertheless, it requires complex model learning and may lead to hallucinations for the agent which can remain stuck in difficult areas of the search space, by contrast with our simpler model-free setting.
success predictor, see next section). The provided
Skills Discovery Many recent works propose to deal with the problem of exploration based on a quantization of the state space distinct skills, that are supposed to correspond to homogeneous sets of states that can be achieved with similar policies. Based on variational empowerment (Gregor) (et al. 2016 Choi et al.) 202T) that approximates a posterior skill distribution via variational inference and mutual information maximization, DIAYN (Eysenbach et al.) 2018) leverages discriminator network to identify for each trajectory the skill that the agent is currently using. Extensions such as Kamienny al. (202T) propose to sequentially chain skills to deal with hard environments. However, these approaches are unstable, and the full exploration of the environment is not guaranteed.
Population-basee Exploration Population-baut basedd RL. using evolutionary strategies to explore environments, is receiving increasing attention for hard exploration problems. Methods such as Novelty Search or Quality Diversity (Conti et al., 2018 Liu et al. 2018 Paolo et al., (02T) deal with populations of agents that evolve to maximize diversity objectives. However, such approaches are usually greatly less sample efficient than GCRL ones. Our SVGG approach can be considered as a trade-off between classical GCRL and evolutionary methods, by dealing with a population of goal particles, ensuring diversity and benefiting from sample efficiency of gradient- based optimization.
2.2
STEIN VARIATIONAL GRADIENT DESCENT
Our method builds on Stein Variational Gradient Descent (SVGD) (Liu & Wang 2016) to approximate the distribution of goals of interest. SVGD is powerful non- parametric tool for density estimation, when the partition function of the target distribution p to approximate intractable. It stands as an efficient alternative to MCMC methods, which are proven to converge to the true distribution but are usually too slow to used in complex optimization processes, It also stands as an alternative to variational inference of parametric neural distributions g. which are restricted to pre-specified families of distributions (c.g.. Gaussian or mixtures of Gaussians) that may not fit target distributions. Instead, it models g as set of particles {z: V all belonging to the support of p.
The idea behind SVGD is to approximate the target distribution p with g by minimizing their KL- divergence: min, KL (gg lp). This objective is reached by iterative deterministic transforms small perturbations of the identity map, on the set of particles: T(x) = x + cő(x), where Ø is a smooth transform function that indicates the direction of the perturbation, while is the magnitude.
3

---

Under review as a conference paper at ICLR 2023
The authors draw connection between KL-divergence and the Stein operator by showing that V.KL(qqr)ll) k= = -E (trace(Ap where 4T) is the distribution of particles after the transformation T, Apo(r) = v(r)V, log p(x)² + V=0(z) being the Stein operator. The KL minimization objective thus directly linked the Stein Discrepancy, defined as: S(q. p) max Eae Earuy (isewg/ttaceeAApppl @EF
Note that S(q. p) = 0 only if q pp
Minimizing Stein Discrepancy being intractable such, Liu et al. (016) and Chwialkowski et al. (2016) introduce the Kernelised Stein Discrepancy (KSD) where the idea is to restrict to projections that belong the unit ball of reproducing kernel Hilbert space H (RKHS), for which there is a closed form solution. The KSD is defined as S(g. p) = max ( (trace
s.t || S I},
whose solution is given by: ()).) = [L>alAAp((t .)], where k(z, ) is the positive definite kernel
of the RKHS H. The RBF kernel k( x²) exp( - x'11²) is commonly used.
Therefore, the steepest descent on the Kl.-obeective is given by the optimal transform:
3j f Ii + ex (xre);
Vi = 1....,, n. where
Φ (2,)
k( 13 Vz, logp(n,) Va, Va,k(a),
T2
attractive fonce
repulsive forve
The "attractive force" in the update T drives the particles toward high density areas of the target p. The "repulsive force" pushes the particles away from each other, therefore fosters exploration and avoids mode collapse. Note that
1, the update in ( corresponds a Maximum Posteriori.
SVGD has already been successfully explored in the context of RL. The Stein Variational Policy method Gradient (SVPG) (Liu al. 2017) employs SVGD to maintain distribution of useful agents as particles. It strongly differs from our approach, since we consider particles as behavior goal candidates, while SVPG aims at capturing the epistemic uncertainty about policy parameters. A mix between this orthogonal work and SVGG could be considered future work, to deal with policy and models uncertainty in addition to a useful distribution of GOIDs. Chen et al. (202T) also relies SVGD build strategy to generate goals to agents, but in very simplified setting without the attractive force from (I. which does not allow to fully benefit from this theoretical framework. Notably, such kind of approach is particularly sensitive catastrophic forgetting.
STEIN VARIATIONAL GOAL GENERATION
In this section we introduce our Stein Variational Goal Generation (SVGG) algorithm. Our aim is to obtain a curriculum to sample goals of appropriate difficulty for the RL. agent, where the curriculum is represented as an evolving goal sampling probability Pguuls- To do so, we maintain model of the agent's 's skills or goal reaching capability and sample goals with the highest entropy, resulting in a distribution Pakills which assigns probability mass to areas of goals of appropriate difficulty, relying on a model of the agent's 's current skills. Additionally, with simple one class SVM, we learn a prior probability Pvalid preventing the particles from moving too far away from the previously achieved goals, and thus from ending up in non-featee neasibiib areas of the goal space. Thus we sample goals from the following target distribution:
Pgunts( P-ails (g)aaai((())
Formal definitions of these two components are given below.
Model of the agent's skills The probability Paills is modelled Neural Network D3 whose parameters & are learned by gradient descent on the following Binary Cross Entropy (BCE) loss:
L. =
Σ
sf (log Do(g') + (1 - ('og(o( - Do(g)))),
EE0
where O= , $ agent in the environment. The sampled goals are those whose predicted probability of success is neither too high nor too low (i.e., we avoid Do(g) Re 1 or Do(g) 0).
1=1 is batch of (goal, success) pairs coming from recent trajectories of the
4

---

Under review as a conference paper at ICLR 2023
Approximate Paoale with particles q {or [= via Stein Variationnal Gradient Descent
Replay
Trajectories
Goals Success
Uniformly sample (m
T1
g1
X
g g~
Goal Conditionned Policy (GCP) TTpl- (g)
Env
Supervised
Learning
Model of the agent's skills Noural Network that predict the probability of the agent for a goal Do(g) Ply & 7/r ~ B))
Prior Distribution:
min, [L Peonts)
distribtuion of valid goals:
Pprior Fg g)
Target goal distribution
Proals Pokills Poctor
Distribution of goals appropriate difficulty
Pakillk (g) s
Figure 1: Overview of the SVGG method. The interaction of agents with their environment is stored in replay buffer (top right) and used learn Da. a model of its abilities to reach goals (bottom right). We build on this model to compute a distribution of goals of appropriate difficulty Pskills leveraging a prior Porior to stay inside the valid goal space. The obtained target goal distribution Pgoals is approximated with set of particles {gs l using Stein Variational Gradient Descent (SVGD).
To build Paiilss based on the prediction of D&, we use a beta distribution whose maximum density point mass is determined according to the output of Dø. by two hyper- parameters and 3 that shape the distribution,
target goal's difficulty
Mediumm: a=
Hard: 155
illus- trated in Figure 2 We report results for these 5 depicted settings in Appendix C.2 (unless specified otherwise, the setting we use the Medium one from the figure).
fgl.).D(gg))
Easy: 3=3,8=1.
Very hard: =
Very easy: a 5.p
We define the distribution pains as an energy- -ased density whose potential is the output of the beta distribution f:
00
10
Pains (g) x exp (f B, Dz(g)).
(4)
Dy(g): model prediction
Prior distribution To keep the sampled goals in the valid goal space of the environment G, we need to define prior distribution over the set of particles. We assume that the set of valid goals G is not given. However, the set states already reached by the agent are stored in archive R.
Figure 2: Modulation of goal's target difficulty with beta distributions.
We would ideally have prior that considers the posterior probability that goal belongs to g given R: (g) F(g GJR). We progressively build such distribution with One Class SVM (OCSVM). This model is mainly designed for outlier or novelty detection in absence of labeled data. Given dataset X E Rdd it defines boundary of the data support in Rd while keeping small portion of the data points out of that boundary. Therefore, we use a calibrated version of OCSVM model. using goals from R the prior distribution:
(g) x P(g E ÿ]R) (g)
(5)
where (g) the output of the OCSVM model, with parameters ub and I is the set of states that the agent is was able reach until the current iteration.
As the agent progresses and expands its set of achieved states through training, it eventually reaches the environment boundary. In this case, Pratia (g) x Pg E ğiR)= = PPg E R.).
The pseudo-code of SVGG is given in Algorithm I After random initialization of the particles set Ω the approach collects data by running the agent with sampled goals corresponding to particles from Ω which used to feed learning of success predictor Dg and reachability predictor Rg. From
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 RL with Stein Variational Goal Generation
1: Input: a GCP To, parameterizable environment M, number of particles m, success predictor
Dø. reachability predictor Ryg buffers of transitions B, reached states and success outcomes
O, a kernel k, numbers n, 867) f(m), {60) ((r), g(m) and ((P),
2: Sample set of particles 52 = {o { Ji=1
uniformly from states reached by pre-runs of TTa
3: for n epochs do
>> Data Collection
4: for (1r) iterations do
5: Sample batch {gr ((r) /i=1
from Ωi
6: for i from 1 to ((T) do
7:
T Rollout To (o - g");
8: Store all (st, aq. S141, 2.99 9') from in B and every 8, from in R;
9: Optionally (HER): Store relabeled transitions from T;
Store outcome (1)) R g¹) in O:
11: end for
end for for f(mm) iterations do
13:
t Model Update
Sample batch {gg, s²) 1 from O;
((m))
14:
15:
Update model Da (e.g.. via ADAM), with gradient of Lo (3)
16:
end for
17:
Update model Rs according to all states in R;
< Prior Update Particles Update
18:
for t(p) iterations do
Compute the density of the target Peoals for the set of particles using 2
20:
Compute transforms: o (a.)) m j= [k(I, v4)
Vaz v:),, log Pgouls (x)
+ V ((,,
21:
Update particles 2; + 2; + cØ" (x)), Vi 1.....mm
22:
end for
23:
Improve agent with any Off-Policy RL algorithm (e.g., DDPG) using transitions in B;
D Agent Improvement
24:
end for
these adjusted models, Pyouts is used compute transforms of the particles in S. At the end of each epoch, an off-Policy RL algorithn is used to improve the agent.
Theorem 1. Recovery property: Let us denote as G+ the set of goals such that R > O and C CE Rd its convex hull. Assume that, at a given iteration I, Dø(g) = for every g (i.e., the whole set G considered as mastered by Do(g)), and that, on that set, R&g is well calibrated: R9) se P(g E G). Assume also that we use a kernel which ensures that the Kernelized Stein Discrepancy KSD(4, Pgonts) of any distribution q with Pgaats
only if q weakly converged to Pgoats Then, with no updates of the models after iteration I and a number of particles m > 1, any area w C C n G with diameter diam(w) 2 Vd diami() 4m-1) eventually contains at least one particle, whenever KSD((2*) (e))((()))))) Vi = 0 after convergence of the particle updates.
The above theorem (proof in Appendix A ensures that, even if the success model over-estimates the capacities of the agent for some area w (e.g., due to changes in the environment, catastrophic forgetting or success model error), some particles are likely to go back to this area once every goal in G looks well covered by the agent, with an increasing probability for more particles. This way, the process can reconsider over-estimated areas, by sampling again goals in them, and hence correcting the corresponding predictions, which leads to attracting attention of Pskills back to these difficult areas. Approaches such as MEGA do not exhibit such recovery properties, since they always sample at the boundary of G. which is likely to incrementally grow towards G. The particles of our approach can be seen attention trackers which remodel the behavior distribution and mobilize the effort on useful areas when needed. This is much better than uniformly sampling from the whole space of achievable states with small probability, which would also ensure some recovery of forgotten areas but in a very inefficient way. This theoretical guarantee of SVGG is empirically validated by the
2 We use DDPG (Lillicrap et al. 2015 in our experiments
As assumed for instance in Liu 2017
This is not always true, but gives strong insights about the behavior (017) for more discussions about KSD.
of SVGD. Please refer to Gorham Mackey

---

Under review as a conference paper at ICLR 2023
experiment from Figure 4 which shows the good recovering property of our approach, after sudden change in the environment.
4 EXPERIMENTS
To mcasure SVGG's performance and compare it to baselines, we evaluate the resulting policy on the entire space of valid goals G in environment. We consider success coverage metric, defined as (TT) = ল Jg P(TT reaches g)dg. The goal space being continuous, we evaluate the policy on a
1
finite uniform subset 9 C G. Then our objective reduces to:
161
|ô| Σ
S(T) = 9
F(t reaches gi)
E [1 ET, min
||s - s ll2 < <d}].
(6)
ES
To build G. we split 9 into areas following a regular grid, and then uniformly sample 30 goals inside each part of the division.
We investigate the following questions: 1) Does SVGG maximize the success coverage objective compared to recent intrinsic motivation work? 2) What target difficulty (i.e., beta distribution as described above) more efficient maximize coverage? Is SVGG robust environment changes? 4) Is SVGG Robust to catastrophic forgetting?
Our main results are based on a Point-Maze environment: an agent moves point without mass within 2d maze with continuous spaces of states and actions. Due to the walls, the goal space presents many discontinuities. In the absence of expert knowledge, introduced via reward shaping or a given curriculum, the success coverage objective is very hard to maximize. To conduct our experiments, we rely on the modular RL code base (Pitis et al. 2020a).
4.1 COMPARED APPROACHES
As baselines, choose methods from the literature that span the existing trends in unsupervised Goal-GANNFloeensaa RL with goal conditioned policies (GCP), MEGA Pitis et al. (202bb) and Goal-GAN Florensa et al. (0188). In addition, the Random baseline randomly selects the behavior goal among past achieved states. We also perform two SVGG ablations: a No Prior version, which considers Pgouts X Pakills and Only Prior version, where Pooals X Pealid- All compared approaches use DDPG and HER with mixed strategy described in the appendix. In every version of SVGG, we use n = 100 particles to approximate the target distribution. Implementation details about all considered architectures are given in Appendix C
4.2
MAIN RESULTS
Our main results in shows the success coverage of baselines over 4 different maze environ- ments. SVGG is the only method that discovers and reaches all valid goals in 4M agent steps in all considered environments.
Due to the known stability issues in GAN training, Goal-GAN the least efficient of baselines. Another explanation of the failure of Goal-GAN is that the generator is likely to output goals outside the valid space, which is not the case for MEGA or SVGG. MEGA chooses the behavior goal in Replay Buffer of achieved goals, while the prior distribution Pealid considered in SVGG keeps particles inside valid areas. Nevertheless, we observe that Goal GAN performs better in environments with linear structures, such as Mazes 3 and 4. On the other hand, it has hard time to generate goals of intermediate difficulty when the structure is random, as in Mazes and
The minimum density heuristic of MEGA efficiently discovers all feasible goals in the environment, but our results show that its success plateaus in almost all the considered environments. MEGA's intrinsic motivation only relies on state novelty. Thus, when the agent has discovered all feasible states, it is unable to target areas that the agent has reached the past but has not mastered yet. Figure also reports results two classical environments: AntMaze and hard version of FetchPickAndl Place.. Place. SVGG also outperforms MEGA on AntMaze, although the considered maze is simpler than those considered in our main experiments, but still induces goal space with discontinuities involved by
7

---

Under review as a conference paper at ICLR 2023
Maze
Maze
SVGG
MEGA
Goal-GAN
No prior
Only prior
Random
0.8
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
Ery
Ant Maze
FetchiicsAndFlice
Success coverage
96
351
250
500K
155
stops
Eny steps
Figure 3:
Success coverage evaluated over 4 different PointMazes, AntMaze and PedlaaeckAnnPlace (4 seeds each). SVGG outperforms recent intrinsic motivation algorithms (MEGA and Goal-GAN), as well as naive baselines.
walls. On FetchPickAndPlace, where the goal space fully smooth, our approach obtains comparable results to its competitors.
Recovery Property In Figure 4 we demonstrate the advantages of SVGG's intrinsic motivation MEGA's in changing environment. Walls suddenly added in the mazes during the training process, after the methods had solved the entire goal space (dot red line from the figure). Curves show that MEGA is unable fully adapt to the new maze setting, which corresponds to Maze from Figure I (although MEGA solved Maze in IM steps in that experiment with constant conditions). On the other hand, SVGG is able discover new difficulties due to the environment modification and focuses on those to finally solve the entire maze in less than 1.5M steps.
We also observe that the advantages of our method over MEGA in terms of recovery ability more significant when changes in the vironment are more drastic (i.c., when starting from maze A).
Maze A
Maze
Final Maze
SVGG
MEGA
SVGG analysis To gain further intuition on how SVGG maximizes the success coverage, we show in Figure the evolution of the particles throughout training. As the agent progresses and reaches novel and harder goals, the D& model updates its predictions. Thus, the target distribu- tion Proals is updated accordingly (background color of the 2nd row of the figure). The parti- cles g = {g: ) are then moved toward new areas of intermediate difficulty through SVGD to minimize K (Poallg)
Start Maze 8
Start Maze A
Succers coverage
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
500K
3M
1M5
500K
1M
2M5
Env steps
Friv steps
Figure 4: Success coverage evolution in changing environment for MEGA and SVGG (4 seeds each).
Figure alsso highlights the recovery property of SVGG. When the model detects that the agent has nothing else to learn in the environment, Pgeals reduces to Pvalid that at this point uniform on the entire goal space. Therefore, the particles spread uniformly the environment and thus prevent
8

---

Under review as a conference paper at ICLR 2023
Step 10K
S0K
BOK
120K
250K
400K
GOOK
1M2
Model
Partiel
Cov, %
0,29
0.56
0.64 %
0.79 %
0.87
0.87 %
0.98
Covergge
Figure 5: Parallel visualization of the model's prediction (1*t row), the particles evolution (2nd row) and of the coverage (3nd row) throughout training in Maze 1.
SVGG from catastrophic forgetting, as the model rediscovers areas that the agent has forgotten how reach (cf. rightmost column in Figure 5). Additional analyses SVGG given in C.2
CONCLUSION
This paper introduces approach for multi-goal reinforcement learning in deterministic environ- ments. Our algorithm, SVGG, leverages Stein Variational Gradient Descent to monitor model w.r.t. its goal reaching capabilities. Using this model, the agent addresses goals intermediate difficulty, resulting in an efficient curriculum for finally covering the whole goal space. Morcover, SVGG can recover from catastrophic forgetting by means of SVGD particle optimization, which is a classic pitfall in intrinsically motivated RL.
Studying the impact of the number of particles is left for future work. Actually, the target distribution being in constant evolution, the KL divergence minimization objective is hard to reach at all times, which makes it difficult to claim that using more particles always better. Furthermore, a previous work Angelo & Fortuin (02T) spotted exploration failures in SVGD, and suggests that periodically annealing the attraction force in particle optimization (T) is required to enable particles to cover non-trivial distributions, c.g. in multimodal settings or in high dimensions.
Some limitations should be addressed in future work. The environments used our experiments have low dimensional goal space, which facilitates learning the particles and the agent's 's model. When the agent's 's observation and goal space will be images, the agent should learn compact latent space of observation like in PPng al. (2019) Yarats al. (2021) and Liu & Abbeel (2021), using various representation learning techniques like contrastive learning, prototypical representations or variational auto-encoders. In future work, such procedures could be added to SVGG in order learn latent goal space from observations, where both particle and model's skills learning could be performed. This would result in end-to-end algorithm learning to discover and reach all possible goals in its environment from pixels.
Besides, we also envision to address larger and stochastic environments, where additional uncertainty estimation should be added to the goal generation process, to prevent the agent getting stuck in uncontrollable states (like TV screen showing white noise) like Chua al. (2018) and |Pathak et al. (2019). Methods such as model disagreement between multiple agents could be explored in future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Picter Abbeel, and Wojeiech Zaremba. Hindsight Experience Replay, arXiv preprint arXiv:/777.0044: 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Jiayu Chen, Yuanxin Zhang, Yuanfan Xu, Huimin Ma, Huazhong Yang, Jiaming Song, Yu Wang, and Yi Wu. Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems, 2021. URL /t..htsss://arxiiorrrg
Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu. Variational empowerment representation learning for goal-based reinforcement learning. arXiv preprint arXiv:2106.01404, 2021.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement
learning in a handful trials using probabilistic dynamics models, 2018. URL https://
arxiv.orge abs 1805 12114
Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton. A kernel test of goodness of fit. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd In- ternational Conference on Machine Learning, volume 48 of Proceedings of Machine Learn- ing Research, pp. 2606-2615, New York, New York, USA, 20-22 Jun 2016. PMLR. URL html
http://cceeeeenes...eeeeeeeeeeee mlr ptt.p//pppee//////aaaaaaaaoooooa
Cédric Colas, Pierre Fournier, Olivier Sigaud, and Pierre- Oudeyer. CURIOUS: intrinsically motivated multi-task multi-goal reinforcement learning. In ICML, 2018.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of govelty-seeking agents. Advances in neural information processing systems, 31, 2018.
Francesco D' D'Angolo and Vincent Fortuin. Annealed stein variational gradient descent. CoRR, abs/2101.09815, 2021. TRLl/ccpss//axxx1..rrr//////////
09815
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine Diversity is you need: Learning skills without a reward function. arXiv preprint arXiv:1802.00070, 2018.
Benjamin Eysenbach, Xinyang Geng, Sergey Levine, and Ruslan Salakhutdinov. Rewriting history with inverse RI. hindsight inference for policy improvement. CoRR, abs/2002. 1089, 2020. URI.
https::/erxiie org/ abs 2002. 11089
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference robot learning, pp. 482-495. PMLR, 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528. PMLR, 2018.
Jackson Gorham and Lester Mackey, Measuring sample quality with kerels.. In International Conference on Machine Learning, pp. 1292-1301. PMLR, 2017.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint (ooe:/611.0757 2016.
Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, pp. 1094-1099. Citeseer, 1993.
Pierre-Alexandre Kamienny, Jean Tarbouriech, Alessandro Lazarie, and Ludovic Denoyer. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. CoRR, abs/21 10. 14457, 2021. URL www.:xiwwww..///////ee/
14457
10

---

Under review as a conference paper at ICLR 2023
Diederik P Kingma and Max Welling. 1rxiv:13/2.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:/509.02977 2015.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training, 2021. URL
https: //rxivv wwtp:////gobb//...oooo/ 04555
Qiang Liu. Stein variational gradient descent gradient flow. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL
Qiang Liu and Dilin Wang. Stein variational gradient descent: general purpose bayesian inference algorithm. arXiv preprint urXiv:1608.0447) 2016.
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-ff tests. In Maria Florina Balcan and Kilian Q. Weinberger (eds.). Proceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 276 New York, New York, USA, 20-22 Jun 2016. PMLR. httpsttt:::///ooeeee ings ings m wwwpppwwsss..eeepp 1tt./1.bb161..1 html
Qihao Liu, Xiaofeng Liu, and Guoping Cai. Pns: Population-guiided novelty search learning method for reinforcement learning. arXiv preprint arXiv: 10264, 2018.
Yang Liu, Prajit Ramachandran, Qiang Liu, and Jian Peng. Stein variational policy gradient. arXiv preprint arXiv:1704.023383 2017.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. arXiv preprint arXiv:1807.04742) 2018.
Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions. In International Conference on Machine Learning, pp. 2377-2386. PMIR, 2016.
Giuseppe Paolo, Alexandre Coninx, Stéphane Doncieux, and Alban Laflaquière. Sparse reward exploration via novelty search and emitters. arXiv preprint arXiv:2102.033140 2021.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2778-2787. PMLR, 06-11 Aug 2017. URL /ttpps://proceeenngs..
press//10/pat
html
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement, 2019. URL /tttpps//arxivva
abs 1tt.../966..... 04161
Silviu Pitis, Harris Chan, and Stephen Zhao. mrl: modular rl. mrl
2020a.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7750-7761. PMLR, 13-18 Jul 2020b. URL https::/prroeeddngs. http://eccaasss...pppeee/////aap
Vitchyr H Pong. Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Sitw-fit: State-covering supe-suue reinforcement learning. arXiv preprint (tt:::9900036990 2019.
Sebastien Racaniere, Andrew Lampinen, Adam Santoro, David Reichert, Vlad Firoiu, and Timothy Lillicrap. Automated curriculum generation through setter-solver interactions. In International Conference Learning Representations, 2019.
11

---

Under review as a conference paper at ICLR 2023
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Interaational Conference on Machine Learning, pp. 1312-1320. PMLR, 2015.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint MoXiv:1703.05007 2017.
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin lonescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint
2018.
Deyu Yang. Hanbo Zhang, Xuguang Lan, and Jishiyu Ding. Density-based curriculum for multi- goal reinforcement learning with sparse rewards. CoRR, abs/2109.08903, 2021. URL https
/arxii.orr//ab///009..
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with
prototypical representations. arXiv, 2021. doi: 10.45550ARXXV.2202.11777 URL https
//arxiv. //arxiv.orre/bb//2002.111
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement, 2020. URL ptttpps:/arxxv..orm http:////aaaioooo/aa///////0// 00060000020000644111
A PROOF OF THEOREM
Theorem 1. Recovery property: Let us denote G the set of goals g such that R4(g) 0 and C Rd its convex hull. Assume that, at given iteration Dz(g) se for every G (i.e., the whole set g+ is considered as mastered by Do(g)), and that, on that set, Rg is well calibrated: kernel which ensures that the Kernelized Stein
R9 P(g E 9). Assume also that Discrepancy (SDD(g.Pootss) of any distribution with Paents is only if weakly converged to Pgoats Then, with no updates of the models after iteration I and a number of particles m > 1, any area G n. G with diameter diam(w) /dd diam(().
( (m-11) eventually contains at least one particle, FE=1: Pgonts) = 0 after convergence of the particle updates.
whenever KSD((x)) }
Proof. In settings where the KSD of distribution from a target is 0 only if weakly converged
p. Liu (017)) previously proved that, for any compact set, the empirical measure μ² μr, computed on a set of particles, converges weakly towards the target p when sufficient number of particles is used. Thus, under our hypotheses, the set of particles of SVGG appropriately represents Pgeats after sufficient number of steps of Stein transforms.
Now, we know that particles cannot leave G+ since Rgg = outside, and so does Pgoats. Since Rag is well calibrated on G+, we also know that Rg. = on every area of G only containing achievable goals. Thus, since Pekslls = 1 in g+, Pyoals is maximal and constant for any area w E G n g. This implies that the concentration of particles in any were uniformly spread over C. In that case, for any particle x from the set ( we know that P(x E w| K Dg {ar; 1= Pgoats) 0) > P(x KSD (2, 1-1 ((Y)- 0), with X' an hypercube of dimensions with side length equal to diam(() and U(2') the uniform distribution over X.
of g+ n G is greater than if particles
Next, if KSD(g = (a. U(x)) = 0. we know that particles are spread as grid over each dimension of X. Thus, in each dimension of X' any particle is separated from its closest neighbor by at most a difference of diam(C)/( vmm 1) in the worst case. Thus, any area w with diameter greater than Vd diam(() is guaranteed to contain particle in that case, which concludes the proof. (4m-1)
12

---

Under review as a conference paper at ICLR 2023
SVGG
MEGA
Goal-GAN
Random
Final Achieved Goals Behavioural Goals
0.96 %
0.92 %
0.61 %
0.42 %
Coverage
Figure 6: Visualization of behavioral goal (1 row), achieved goals (2nd row) and success coverage
(3rd row) for 1M steps in the Maze environment.
B VISUALIZATION OF GOALS
We visualize behavioral and achieved goals Maze (Figure and Maze 2 (Figure 7 . in parallel with the success coverage after training. The main advantages of our method lie in the capacity to target difficult areas and avoid catastrophic forgetting, which results in nearly optimal success coverage. We observe that MEGA efficiently discovers the environment but is unable detect where the agent has not mastered yet goal in order to target it. This also leads to catastrophic forgetting and lack of adaptation when the environment changes, a case studied
main paper.
One can see that the generation of GOIDs in Goal-GAN is very unstable and tricky in such discontin- uous goal space, especially as the generator susceptible to output goals outside the environment boundary, which SVGG avoids with the prior distribution.
C ADDITIONAL EXPERIMENTS AND IMPLEMENTATION DETAILS
C.I ENVIRONMENTS
Pointmaze We use 2D pointmaze environment where the state space and goal space are (x. y) coordinates (the agent cannot see the walls), and the agent moves according its action, which is 2-dimensional vector with dimensions constrained to |-0.95, 0.95]. All methods are tested on 4 x mazes with different shapes and a highly discontinuous goal space. The maximum number of steps in one trajectory is set to 30. We argue that the difficulty of these environments does not lie in their size, but rather in the complexity their goal space and thus the trajectories adopted by agent to reach these goals.
As assumed for instance in Liu 2017 This is not always true, but gives strong insights about the behavior of SVGD. Please refer to Gorham Mackey (017) for more discussions about KSD.
13

---

Under review as a conference paper at ICLR 2023
SVGG
MEGA
Goal-GAN
Random
Final Achleved Goals Behavloural Goals
Episodes (Thousands)
0.97 %
0.82
0.64 %
0.35 %
1.2
1.2
3.
1.
1.5
3.0
2/0
2.0
1.0
6G
0.23
0.29
0.0
30
D0
20
0.0
Coverage
QLP
12
CD0
10
10
0.4
0.7
0.88
0.57
0.22
0.1
60
0.37
95
0.72
30
50
2.1
0.1
Q3
1.
1.
10
1.5
1.0
2.0
1.0
6.1
3.0
10
10
10
10
1.0
200
2.0
1.0
000
10
1.0
0.26
1.0
3.0
0.59
1.0
69
10
10
10
3.0
822
35
1.0
022
3.90
10
1.8
0.7
3.0
59
1.0
06
0.57
100
100
3.0
0.1
32
0.3
0.0
Figure 7:
Visualization of behavioral goal (1*t row), achieved goals (2nd row) and success coverage (3nd row) for 4M steps in the environment on Maze 2. Our method (SVGG) the only one achieve nearly perfect success coverage.
Antmaze An Ant has to move around U-shape hallway maze of size 20x4, the goal space remains (x.y) coordinates of the Ant in Pointmaze. However, the exploratory behavior is much simpler than in the considered pointmazes environments, as the maze is simply U-shaped. The difficulty lies in the control of the Ant dynamics, as it must move its legs with 8-dimensional action space and the observation space is a 30-dimensional vector corresponding to the angles of the legs.
Fetch Pick and Place (Hard version) To test the versatility of our method, we conduct additional experiments hard version of FetchPiickndPPaceee The agent controls a robotic arm which must pick and place a block to 3D desired location. In the hard version, the target goals are placed between 20 and 45cm in the air, while the original version, 50% of target goals are on the table and the remaining ones are placed between O and 45cm in the air.
from OpenAl gym [Brockman al. (2016).
The distance reach the goals greater difficulty in terms of control (the action dimension is 4 and the state dimension is 10 which correspond to the various positions of the robot joint), the goal generation component is significantly easier as the goal space is smooth, there is no obstacle for the agent to bypass. As Figure 3 shows, SVGG solves the environment within 1.5M steps, but does not significantly outperform MEGA.
environment
0.1. While this environment presents
We argue that the interest of our goal generation algorithm resides in environments with highly discontinuous goal space as the action control largely supported by the RL algorithm (e.g. DDPG). Therefore, the smaller difference between MEGA and SVGG in this environment SVGG is mainly designed to target non-smooth goal spaces, and to avoid pitfalls such as catastrophic forgetting. Experiments environments that are complex both in terms of goals and control are left for future work.
expected, as
C.2 CONTROL OF THE SAMPLING DIFFICULTY
Using beta distributions reveals which goal difficulty an intrinsic motivation algorithm should aim at for agent efficiently explore and control environment. In Figure 9 compare 5 versions
14

---

Under review as a conference paper at ICLR 2023
target goal's difficulty
Medium: 2=2.8=2
Hard: a= 1.5.
fla, P. Dz(g))
Easy: a=3,
Very hard: ==1.8=5
Very easy: a 5
0.0
0.2
10
Dę(g): model prediction
Figure 8: Modulation of goal's target difficulty with beta distributions. Dø(g) being the model's s predicted probability of the agent reaching the goal g.
Medium
Hard
Easy
Very hard
Very easy
Maze
Maze
Moze 3
Maze
3.0
1.0
1.0
Suttens
0.4
0.4
022
0.2
0.2
0.0
0.0
0.0
2M Env stops
3M
41
3M
2M
Env steps
Erw Stops
Env stops
Figure 9: Success coverage evolution for target difficulties as intrinsic motivation for SVGG, mazes and seeds each.
of SVGG using different distributions from Figure 8 on 4 previously considered mazes. One can observe that extreme targets difficulties are the least effective ones, especially the Very easy, which is too conservative to efficiently explore new areas of the space. On the other hand, SVGG performs very well with Medium and Hard distributions. This suggests that the optimal goal difficulty is somewhere between medium and hard.
C.3 BASELINES
The authors of MEGA Pitis et al. (2020b) train a GCP with previously achieved goals from a replay buffer. Their choice of goals relies on minimum density heuristic, where they model the distribution of achieved goals with a KDE. They argue that aiming novel goals suffices to efficiently discover and control the environment. We use the original implementation of the authors, the pseudocode is described in Algorithm 2
15

---

Under review as a conference paper at ICLR 2023
Algorithm 2 MEGA
1: Input: GCP parameterizable environment M, reached states R, a KDE model Pas of the achieved states, numbers c, n, N, ((r) and ((r)
2: Initialize with states reached by pre-runs of me;
3: for epochs do
Data Collection
4: for ((()) iterations do
[ 5: Sample batch { /==1 uniformly from R;
6: Eliminate unachievable candidates if their Q-values are below cutoff
( 7: Choose goals {g' Ji=1 - (gi
8:
for from 1 to ((7) do
9:
T 4 Rollout To Te 9²);
Store all (st ag St+1, re: 9") from 7 in B and every s from in R; 11: Optionally (HER): Store relabeled transitions from т:
Store outcome (s)) # g¹)) in O;
13:
end for
end for
15:
Update KDE model Pas with uniform sample from R;
D KDE Update
16:
Improve agent with any Off-Policy RL algorithm (e.g., DDPG) using transitions in B;
D Agent Improvement
17: end for
Algorithm 3 Goal-GAN
1: Input: a GCP Te, parameterizable environment M, goal Generator Ga, Discriminator Do. success predictor D&, buffers of transitions B, reached states R and success outcomes O, numbers n, Pminn Pmnaz ((r), z6r), 1(m), k(m), ((9) and k(0))
2: Initialize Gg, and Dæ, with pre-runs of TT:
3: for epochs do
Data Collection
4: for t(r) iterations do
5: Sample noise {= (0) (0,
(r) [-
6: generate {g-)(-m [=
Ga, ({=))/
7:
for i from 1 to ((7) do
8:
Rollout ==(-l..9=9') 9¹);
Store all (st, at, Sc+1, Tt: g') from T in B and every s; from in R; 10: Optionally (HER): Store relabeled transitions from ;;
9:
11:
Store outcome (g', [(sr) x g¹)) in O;
end for
13:
end for
14:
Sample batch {(g) ( from O;
GAN training
15:
Label goals (GOID or not) with model D& {M,
MM 1(00
{Pmin Delgi) < P Pmoo li=1
(19)
16: for (()) iterations do
Update Go, and Da, (e.g. with ADAM) with gradients L.SGNN losses;
18:
end for
19: for (mm) iterations do
D Model Update
20:
Sample batch {g s² > from O;
((m)
Update model D& (e.g., via ADAM), with gradient of Ls (agent model loss described in
the paper):
end for
23: Improve agent with any Off-Policy RL algorithm (e.g., DDPG) using transitions in B;
Agent Improvement
24: end for
Goal-GAI Loal-GANII Florensa et al. 2018 uses a procedural goal generation method based on GAN training. As our SVGG, it aims at sampling goals of intermediate difficulty, which they define GGoD Pmin < (g(g) < Pmman J. (g) being the probability for the policy и to reach goal g, Pmin and P. maa hyperparameters. A Discriminator Da, trained to distinguish between goals in GGom and
16

---

Under review as a conference paper at ICLR 2023
other goals, while generator Gg, is trained to output goals in GGoD by relying the discriminator outputs. They optimize Gg, and Da, in a manner similar to the Least-Squares GAN (LSGAN) with the following losses
min v(Da.) ES~R [1 (D) 1)2 + (1- (DD, +1
min V(G,)) [G--(())) De, (=))²),
where y, is the label that indicates whether belongs to GGOID or not. In Florensa et al. (0118), the authors Monte Carlo sampling of the policy estimate Vy- For efficiency reasons, learned model of the agent's 's capabilities as in SVGG. The pseudocode is described Algorithm
C.4 HINDSIGITT EXPERIENCE REPLAY
The original goal relabeling strategy introduced HER Andrychowicz al. (0117) is the future, which consists in relabeling given transition with goal achieved on the trajectory later on. This is very effective in sparse reward setting learn GCP, However, many works suggested that relabeling transitions with goals outside the current trajectory helps the agent generalize across trajectories. For example, one can use inverse RL determine the optimal goal to relabel a given transition Eysenbachi al. (2020). We use naive version of this method. As in Pitis al. (2020b), relabel transitions for DDPG optimization using a mixed strategy. All methods presented in this work use the same strategy.
10% of real experience are kept while 40% of the transitions are relabeled using the future strategy of HER. We relabel the remaining 50% transitions with goals outside of their trajectory, with randomly sampled goals among the past behavioral and achieved goals. The latter part of the strategy helps share information between different trajectories that often contains similar transitions.
17

---

Under review as a conference paper at ICLR 2023
C.5 ARCHITECTURES AND HYPERPARAMETERS
Hyper-Parameter
Value
DDPG
Batch size for replay buffer
2000
Discount factor y
0.99
Action L2 regularization
le-1
Action noise maximum std (Gaussian noise)
0.1
Warm up steps before training
2500
Actor learning rate
le-3
Critic learning rate
le-3
Target Network update fraction for soft parameters update update
0.05
Activation for actor and critic networks
Gelu
Actor dense layers sizes
(512. 512, 512)
Critic dense layers sizes
(512. 512, 512)
Replay buffer size
max training steps
SVGD
Number of particles
100
Optimization every k agent's steps
20
Number of particle moves for one optimization k(P))
Bandwidth for RBF kernel k(-,-) for RKHS
Learning rate
1e-3
Agent's 's skill model D
Dense layers sizes
(64, 64)
K gradient steps per optimization
10
Learning rate
le-3
Training batch size (())
100
Training history length (trajectories)
1000
Optimize every (steps)
4000
Number of training steps (-(m)
100
Activations
Gelu
OCSVM prior
Bandwidth a for RBF kernel k(.,.) for RKHS Optimize every (agent's steps)
4000
MEGA
Bandwidth of KDE RBF kernel
0.1
KDE optimize every (agent's 's steps)
Number of state samples for KDE optimization
10000
Number of sampled candidate goals N
100
Q-vauue cutoff
-3
Goal-GAN
Generator input dimension (gaussian noise)
4
Generator dense layers sizes
(64, 64)
Discriminator dense layers sizes
(64, 64)
Optimize every (agent's steps)
2000
GAN training batch size (19)
200
Number of GAN steps for one optimization k(9)
100
Discriminator/
Learning rate
le-3
P inin minimum probability for GOID
0.1
P. mo maximum probability for GOID
0.9
18

---

Under review as a conference paper at ICLR 2023
C.6 RESSOURCES
Every seed of each experiment was run on GPU in the following list (Nvidia RTX A6000, Nvidia RTX A5000, Nvidia TITAN RTX, Nvidia Titan Xp, Nvidia TITAN V). We estimate the amount of training time to be 4,000 hours, most of it executed in parallel, as we had full time access to 12 GPUs.
19


Reviews:
[{"review_id": "vCdeV32tw9", "reviewer": "Reviewer_ipWS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors describe an automatic curriculum learning method for goal-based (reward is to reach a goal or not) multi-task RL they call SVGG. It keeps a set of current goals as particles, estimates their density using Stein Variational Gradient Descent, and keeps updating the particles to keep presenting interesting challenges to the agent without leaving already-learned goals behind to prevent catastrophic forgetting.\n\nThe estimation of the density relies on two models: D, behaving like a value function, estimating success rate for reaching a particular goal, and R, a more conservative model, not allowing to sample goals far away from what was achieved so far (with the motivation behind R being to not allow sampling invalid tasks).\n\nAuthors motivate the work with HER being prone to local optima when using Euclidean distance as the metric between goals which live in a discontinuous space.\n\nAuthors claim their approach could be easily adapted to settings where goals don't correspond to states to be reached. I would like the authors to expand on their statement, as it's not obvious for me how to extend the work.\n\nAuthors compare their method to MEGA and GoalGAN on 4 mazes from PointMaze, AntMaze,, and robotic-arm task FetchPickAndPlace.", "summary_of_the_review": "The paper presents a new method. I have numerous concerns about its ability to meet the claims made in the paper, and the experimental results are not encouraging. Before these issues are addressed, I discourage publishing the paper at ICLR.", "strength_and_weaknesses": "## Strengths\n\nOn the high level, the mechanism of storing tasks in the buffer and attaching weights to them based on some criteria to maximize learnability makes sense.\n\nDevelopment of models which encourage similarity to already solved tasks as a method of preventing overestimation is an interesting idea (even though I'd love it to happen in a single model, see below).\n\n## Weaknesses\n\n### Splitting D and R\nD and R have a similar role: to estimate the density for whether a task is good for training. I don't find the motivation presented for splitting them (that R distinguishes \"unachievable\" tasks) convincing, as the tasks the paper tests on: PointMaze, and AntMaze don't have such tasks.\n\nIt seems to me that the actual role of R is to correct for overestimation behavior of D, but it should be possible to combine them into a single model (eg. with loss being a combination of conservative R and value-accurate D), which would make the method simpler.\n\nFurthermore, the theorem giving intuition about why the method works assumes that R behaves exactly like D (is 0 where $D<0$ and 1 elsewhere).[1]\n\n\n### Catastrophic forgetting\nThe main advantage of the work over MEGA the authors seem to stress is the ability of SVGG to escape catastrophic forgetting, as no goal will ever be \"forgotten\" when it's already learned, but will be continuously revisited. Overall, that's a desiderata that is simple to satisfy by just uniformly sampling previously achieved goals with some fixed probability $\\alpha$. In fact, the MEGA paper[2] proposed such a variant, calling it OMEGA (where they also estimate $\\alpha$). It seems appropriate to compare against that version of their work.\n\nFurthermore, I don't understand why $D=1$ would not lead to catastrophic forgetting for SVGG. As $q$, expressed by the particles means to approximate $p\\_{}goals = exp(f(D)) * R$, and $f(1) = 0$ for most of the tested $f$s (which makes most sense, we don't want to be oversampling solved tasks), the weight of tasks with $D=1$ would be $0$, even if they are present in $\\Omega$, leading to catastrophic forgetting.\n\n### Zone of Proximal Development\nChoosing $\\alpha$ and $\\beta$ fixed over training corresponds to prioritizing tasks of a particular level of difficulty (percentage of being solved). Eg. authors' choice of $\\alpha=\\beta$ corresponds to tasks with $D=1/2$ having the highest weight throughout the training, even though at the beginning the distribution of task difficulty (as measured by $D$) may be centered around 0.01 (meaning that tasks with $D=1/2$ would be way too hard, or most likely: having their difficulty overestimated). \n\nThis seems unintuitive, as the Curriculum Learning methods typically attempt to sample tasks of medium difficulty for the agent's current abilities, and requires a further explanation.\n\n### Exploration\n\nDespite the claim about working in non-Euclidean spaces, R seems to be strongly discouraging exploration: if the initial set of particles would be chosen in a deep corner of the task-space (because the initial policy is not able to explore well), the method will have to slowly go through the goals based on what OCSVM predicts as the boundary of the dataset, even if the policy, after a couple of steps, would be able to explore much wider and generate the new particles covering the space more uniformly.\n\n### Experiments\nThe experiments are not convincing enough. The domains are simpler than, e.g. the ones shown in MEGA (smaller mazes, less tasks). There are two baselines: one of them is 4 years old, and another one a the simpler version of the method presented in the MEGA paper. Ideally, I would like to see a comparison with 2>= current methods, eg. SPaCE [3] and OMEGA or fixed, $\\alpha>0$ version of MEGA.\n\nDespite the maze tasks (on which the improvement is shown) are simpler than the ones used in MEGA, the reference scores of MEGA are much worse (see their figure 3 in [3]), suggesting that something is wrong with the experimental protocol.\n\n## Suggestions\n\nI would like to see a discussion on why not embed the tasks in a better metric space than the Euclidean one, as it seems easier than the proposed method.\n\n[1] By the way, it seems that the proof says that $R=1$ on whole $G^+$, but the theorem assumes $R=1$ only on $\\tilde{G}$.\n\n[2] [Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning](https://arxiv.org/pdf/2007.02832.pdf)\n\n[3] [Self-Paced Context Evaluation for Contextual Reinforcement Learning](https://arxiv.org/pdf/2106.05110.pdf)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "qV5MIigR6-", "reviewer": "Reviewer_ZXEG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel approach Stein Variational Goal Generation (SVGG) to build on recent automatic curriculum learning techniques to address the difficulty of discontinuities in the state or goal spaces for goal-reaching tasks. Experiments show that SVGG achieves state-of-the-art results for hard-exploration RL tasks in several environments with low-dimensional state spaces and deterministic transition models.", "summary_of_the_review": "I have some concerns and thus do not recommend acceptance. ", "strength_and_weaknesses": "The paper is well-written and the idea is simple but novel. However, I have some concerns and thus do not recommend acceptance. \n\nQ1. SVGG seems to be the direct combination of stein variational gradient and goal generation, which weakens the novelty of the paper.\n\nQ2. Recently, many related works [1-7] have been proposed to solve goal-reaching tasks. Some of them [4-6] learn a goal space capturing the inter-state reachability, which can address the difficulty of discontinuities in the goal spaces. More discussions about the comparison between SVGG and these methods are required.\n\nQ3. The environments used in the experiment section seem to be simple. How well does SVGG perform on RL tasks of complex environments?\n\nReferences:\n\n[1] Huang, Z., et al. Mapping state space using landmarks for universal goal reaching. NeurIPS, 2019.\n\n[2] Ghosh, Dibya, et al. \"Learning to reach goals via iterated supervised learning.\" ICLR 2021.\n\n[3] Hoang, Christopher, et al. \"Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning.\" Advances in Neural Information Processing Systems 34 (2021): 26963-26975.\n\n[4] Zhang, Tianren, et al. \"Generating adjacency-constrained subgoals in hierarchical reinforcement learning.\" NeurIPS, 2020.\n\n[5] Zhang, Tianren, et al. “Adjacency constraint for efficient hierarchical reinforcement learning”, IEEE TPAMI, 2022.\n\n[6] Chane-Sane, et al. \"Goal-conditioned reinforcement learning with imagined subgoals.\" ICML, 2021.\n\n[7] Kim, Junsu., et al. Landmark-guided subgoal generation in hierarchical reinforcement learning. NeurIPS, 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_zijSLTSzT", "reviewer": "Reviewer_BQng", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problem of goal conditioned reinforcement learning (GCRL), where the authors propose a new way of sampling goals during the data collection phase. Specifically, the authors propose to sample goals with probability proportional to two factors: the prior distribution of goals and the intermediate difficulty of goals. The intermediate difficulty probability is measured by a beta distribution which is parameterized by a learned classifier to give high weight on goals that the agent can achieve only some of the time. This ensures that the generated goals stay within the valid regions, and are neither too easy nor too difficult. The authors then propose to learn this distribution via Stein variational gradient descent (SVGD) to obtain a goal generator model.\n\nThe authors evaluate the proposed method empirically on maze, ant maze and simulated robot pick and place environments, and the results suggest that the proposed method outperforms prior methods in terms of goal success rate and sample efficiency.\n", "summary_of_the_review": "While I think the paper presents an interesting new technique of sampling goals, I believe that the proposed method is not sufficiently justified and the empirical performance is not very strong. Hence I cannot recommend acceptance of this paper in its current state.\n", "strength_and_weaknesses": "Overall I think the paper presents an interesting new technique of sampling goals. The idea is presented clearly and the evaluations are easy to follow. However I do have some concerns.\n\n\n### Strength\n\nThe proposed method is clearly presented and well motivated. I find the idea of weighting the desired goal probability with the intermediate difficulty and prior distribution very intuitive, and I can imagine a class of different methods building on top of this idea. I also find the ablation study and visualization of the proposed method is very informative. The behavior of the proposed method can be directly understood from the generated goals.\n\n\n### Weaknesses\n\nFrom my understanding of the paper, it seems like the core novelty of the paper is the particular goal distribution jointly weighted by the intermediate difficulty and goal prior. This objective is intuitive as it gives high weight to valid goals that have intermediate difficulty that the policy can only only some of the times. However, it is one of many other objectives that encourages similar things (for example, the GOID objective from Goal-GAN), so it is not clear to me why this particular choice is better. It would be important for the authors to justify the objective either theoretically or empirically.\n\n\nMoreover, once the desired goal distribution is formulated, it seems to me that SVGD is just a particular choice of generative models used to learn this goal distribution. Given an unnormalized probability density, many other techniques, such as MCMC, Langevin dynamics and variational inference, can be used to generate samples from this distribution, and some of them are much simpler than SVGD. Especially in the low-dimensional goal space that the authors experiment with, numerical integration can be accurately calculated to normalize the goal density. The mentioned recovery property can also be obtained by simply mixing the learned goal distribution with a uniform prior distribution.  Hence, it is not clear to me why SVGD is a good choice here.\n\n\nThe empirical performance of the proposed method is also not very strong. Among the domains the authors experiment with, only in 3 out of 6 domains the proposed method outperforms prior methods significantly.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8veSqJ8WY-", "reviewer": "Reviewer_TSmE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis work considers a multi-goal learning setting where the agent learns across a diverse set of goals by building its own representation model of goal space. The work tackles exploration in a multi-goal learning setting since being able to learn across diverse goals during train plays a vital role for generalization. However, since the goal space requires exploration - the work proposes a new mechanism based on stein variational goal generation to tackle the goal exploration problem. \n", "summary_of_the_review": "\nPlease see above - I think the contributions of the work were not clear enough and the overall model is too complicated to be of actual impact. ", "strength_and_weaknesses": "\t- This paper addresses the important problem of exploration in goal spaces, since to design a curriculum of goal generation for better generalization, it is important to learn a model that can help with exploration of new goals. The work is primarily based on context of automatic goal curriculum generation \n\t- One minor thing about the paper in general though is it provides too much not connected information in the write-up, making it hard to fully understand the context of the paper. It would be better if the paper could be written more clearly, with the main contributions focussed on. From the introduction itself, it was not clear what are the primary contributions of this work. \n\t- The work adapts from using SVGD in context of approximating the goal distribution for curriculum generation. Section 2.2 providing context of this, and stating out related work that uses SVGD is useful to see. \n\t- Section 3 in my opinion could have been written more clearly; and figure 1 could be explained better. The overall approach seems too complicated for actually being impactful for a general community. The work primarily focuses on hard exploration problem, and my understanding is that SVGD based approach is required for approximating a goal distribution, such that this learned model can then be used for novel goal generation? \n\t- I do not understand the context of equation 2? Why is the distribution of goals related to skills and valid goals? It would be helpful if authors could provide more context of why the distribution of goals that we are trying to approximate, indeed relies on these two? The text below equation (2) provides some context to this, and p_valid (g) is understandable; but how does the learned skills also depend on p(g) is perhaps not clear? My understanding is, we want to approximate goals but at the same time make sure these goals are achievable based on the learned skills? Is this the correct way to interpret this?\n\t- Equation 4 and 5 then provides a tractable approach to approximating these two terms, which are again dependent on learning two separate models?\n\t- Could the authors explain the coverage metric in more details? It seems like there is an approximation being made for this coverage metric? The problem seems intractable, and is the same as trying to approximate the state distribution for coverage? \nExperimental results in my opinion does not seem entirely convincing. The most difficult task seems to be the FetchPickandPlace and it is not clear if the improvements are significant, especially given the fact that the overall proposed model in this paper seems overly complicated (see below)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject