C3PO: Learning to Achieve Arbitrary Goals via Massively Entropic Pretraining
Authors: Alexis D. Jacq
Manu Orsini
Gabriel Dulac-Arnold
Olivier Pietquin
Matthieu Geist
Olivier Bachem
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement Learning
Exploration
Goal-conditioned Policy
Continuous Control

Abstract: Given a particular embodiment, we propose a novel method (C3PO) that learns policies able to achieve any arbitrary position and pose.  Such a policy would allow for easier control, and would be re-useable as a key building block for downstream tasks.  The method is two-fold: First, we introduce a novel exploration algorithm that optimizes for uniform coverage, is able to discover a set of achievable states, and investigates its abilities in attaining both high coverage, and hard-to-discover states;  Second,  we leverage this set of achievable states as training data for a universal goal-achievement policy, a goal-based SAC variant. We demonstrate the trained policy's performance in achieving a large number of novel states. Finally, we showcase the influence of massive unsupervised training of a goal-achievement policy with state-of-the-art pose-based control of the Hopper, Walker, Halfcheetah, Humanoid and Ant embodiments.

Under review as a conference paper at ICLR 2023
C3PO: LEARNING TO ACHIEVE ARBITRARY GOALS VIA MASSIVELY ENTROPIC PRETRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Given particular embodiment, we propose novel method (C3PO) that learns policies able to achieve any arbitrary position and pose. Such a policy would allow for easier control, and would be re-useable as a key building block for downstream tasks. The method is two-fold: First, we introduce a novel exploration algorithm that optimizes for uniform coverage, is able to discover a set of achievable states, and investigates its abilities in attaining both high coverage, and hard-to-discover states; Second, we leverage this set of achievable states as training data for a universal noal-achievement policy, goal-based SAC variant. We demonstrate the trained policy's 's performance in achieving large number of novel states. Finally, we showcase the influence of massive unsupervised training of a actleeenievement policy with state-of-the- pose-base based control of the Hopper, Walker, Halfcheetah, Humanoid and Ant embodiments.
INTRODUCTION
Reinforcement learning (RL) has shown great results in optimizing for single reward functions (Mnih et al., 2013; Silver et al., 2017; Vinyals et al., 2019), that is when controller has to solve a specific task and/or the task is known beforehand. If the task is not known a priori, or is likely to be often re-configured, then re-training a new policy from scratch can be very expensive and looks as waste of resources. In the case of multipurpose systems deployed in contexts where they will likely be required to perform a large range of tasks, investing significant resources into training a nigh-performance general goal-based controller beforehand makes sense. We propose an approach allowing for training a universal goal achievement policy, a policy able to attain any arbitrary state the system take. coal-conditioned RL is such a setting where single policy function can be prompted to aim for particular goal-state (Kaelbling, 1993; Schaul et al., 2015). One important issue with goal-conditioned RL that goals that are useful for training the policy are generally unknown, and even in the case of humans this is a key part of learning general controllers (Schulz, 2012; Smith and Gasser, 2005). Several approaches exist in the literature. Adversarial methods build out goal- bosee curriculum (Mendonca et al., 2021; Eysenbach et al., 2018: OpenAl et al., 2021; Florensa et al., 2018) through various ad-hoc 2-player games. Other recent approaches (Kamienny et al., 2021; Campos et al., 2020) explicitly optimize for uniform state coverage with the goal of learning a general goal-conditioned policy, but are still tied to learning a policy function to actually implement the exploration strategy in the environment. Although not explicitly geared towards based learning, many reward-free RL (Laskin et al., 2021) approaches are geared towards learning policies that provide good state coverage (Bellemare et al., 2016; Ostrovski et al., 2017; Burda et al., 2018; Houthooft et al., 2016; Badia et al., 2020), however primarily with the intent of fine-tuning the learnt exploration policy rather than leveraging its state coverage. Our proposed approach, Entropy- Based Conditioned Continuous Control Policy Optimization (C3PO), is based on the hypothesis that disentangling the exploration phase from the policy learning phase can lead to simpler and more robust algorithms. It is composed of two steps:
generating set of achievable states, as diverse possible to maximize coverage, while being as uniform as possible to facilitate interpolation.
Goal Discovery:
Con-iCondi Training: leveraging these states learn to reach arbitrary goals.

---

Under review as a conference paper at ICLR 2023
4452
0.3
Figure I: Left: Examples of the C3PO (in beige) achieving a pose (in red). Right: Goal Achievement of C3PO vs. other methods, averaged over Walker, Hopper, Halfcheetah and Ant (see Section 3.3).
To address the goal discovery step in C3PO, we propose the Chronological Greedy Entropy Maximiza- tion (ChronoGEM) algorithm, designed to exhaustively explore reachable states, even in complex high dimensional environments. ChronoGEM does not rely on any form of trained policy and thus doesn't require any interaction with the environment to learn to explore. Instead it uses highly- parallelized random- brannchirnnhingggg policy to cover the environment, whose branching tree is iteratively re- re-pruned to maintain uniform leaf coverage. This iterative pruning process leverages learnt density models and inverse sampling to maintain a set of leaf states that are as uniform as possible over the state space. Training the goal-conditioned policy is then performed by leveraging the uniform states generated by ChronoGEM as dataset of goals that provide well-distributed coverage over achievable states. We perform two types of experiments to illustrate C3PO's ' benefits over similar methods: First, we evaluate entropy upper bounds on ChronoGEM'ss generated state distribution compared to other reference exploration methods such as RND (Burda et al., 2018) and SMM (Lee et al., 2019) as described in Section 2.1.2. Second, we compare the full C3PO approach to ablated versions that leverage datasets generated by SMM and RND and random walk. We do this by performing cross-valin cross-validating of goal-conditioned policies across methods. By training policy dataset and evaluating its coul-achievement capabilities on datasets generated by other methods, observe which of the methods gives rise to the most general policy. Through these two empirical studies, we illustrate the superiority of ChronoGEM compared to RND and S.M. Finally, we investigate C3PO's abilities in achieving arbitrary poses on five continuous control environments: Hopper, Walker2d, HalfCheetah, Ant and Humanoid. Videos of the resulting behaviours reaching the goal poses are available as gif files in our supplementary material.
method's
2 CONDITIONED CONTINUOUS CONTROL POLICY OPTIMIZATION (C3PO)
The optimal universal goal-achievement policy for a given embodiment should allow an agent to achieve any reachable position and pose in the environment as quickly as possible. Learning such a policy necessarily requires significant amount of exploration and training to both find and learn to attain a large enough number of states to generalize across goal-states. However, in the context of simulator, which allows for both parallelization and arbitrary environment resets, covering massive amount of the state space is doable. In our case, we consider 2 parallel trajectories. that are re-sampled every step to maintain high coverage of the reachable space. Once such large coverage of the state space is achieved, learning a goal-achievement policy can be done with a relatively traight-forwarde learning algorithm that aims at attaining goals from this high-coverage set of states. If the states are well-enough distributed, and if the learning algorithm is sufficiently efficient, we may expect the final policy to achieve universal goal-achievement.
2.1 MASSIVELY ENTROPIC PRE-TRAINING
As described above, the first step is to discover the set of achievable goals. This collection will be the key of the effectiveness of the resulting policy: We want it as uniform possible such that reachable region is neglected. Therefore, without any prior, the ideal set of goals should be uniformly sampled from the manifold of states that reachable in given number of steps (T). Since the shape
2

---

Under review as a conference paper at ICLR 2023
of that manifold is totally unknown and can be arbitrarily complex, such sampling is impossible. However, it is possible to approximate such a sampling if enough states are simultaneously explored able to sample N states that approximate the uniform distribution at time T - 1. Then, from each of these states, playing K' uniform actions to obtain N K next states would not lead to a uniform sampling over the possible next states. However, with N large enough, it would at least induce good coverage of the set of reachable next states. Let pr be the distribution induced by these N next states. Since the of achievable states in I steps is necessarily bounded (at least in any realistic environment), and given that we are able to closely estimate pr. we can sub-sample with a probability weighted by the inverse of the density 1 in order to approximate a uniform sampling. We prove Appendix ?? that such a sub-sampling approximates uniform sampling when the number of sampled states is large enough. This suggests a recursive approach to approximate a uniform sampling of the reachable states in T steps, by starting with states sampled from the environment's 's initial distribution Po. playing uniform actions, sub-sampling get an highly covering set that approximates a uniform distribution, re-exploring actions from that new set, and then again, for T iterations. We call this process ChronoGEM (for Chronological Greedy Entropy Maximization) since at a given step, it only focus on maximizing the entropy by directly approximating uniform distribution over the next step, without further planning. Algo 1 summarizes ChronoGEM.
at the previous time step (T' 1). Assume
Algorithm 1 Chronological Greedy Entropy Maximization (ChronoGEM)
1: Sample N states S. = { Po.
2: for = 1 to I do
3: Sample K uniform actions for each state of S,-1-
4: Obtain KN next states.
5: Estimate Pt using density model fitted on the distribution of these KN states.
6: Sample N states with probabilities p(s) x p.(*) to get S...
7: end for
8: return ST
ChronoGEM requires exactly KNT interactions with the environment, which makes easily control- lable in term of sample complexity. At each time step, the N sampled states being supposed to be independent also simplifies the implementation, allowing to parallelize N jobs that only consume KT interactions, significantly shortening the time complexity (that also depend the density model fitting).
2.1.1 RESETTABLE STATES ASSUMPTION
Like other diffusion- based algorithms (see related works 4.4), ChronoGEM needs to explore many actions from single given state. While many approaches would forbid such assumptions to better approximate real-wwor world situation, we argue that in our case, simulated environments are used for these precise types of simplification, also allowing to run many jobs in parallel and to safcly explore dangerous trajectories. In this paper, physics engine offering a collection of continuous control environments similar to MuJoCo (Todorov et al., 2012). Brax is designed to acceleration devices and massive parallelization, and allows resetting a trajectory at any given state. In the unfortunate case where this assumption would not be available, different equivalent alternative of ChronoGEM could be imagined. With short horizon and small branching factor K, the easiest one being to start with KT N states, and sub-sampling with factor K at each step. until it ends up with N states at time T. In this paper, we stick to the case where this assumption is satisfied.
every simulations using Brax (Freeman et al., 2021),
2.1.2 DENSITY ESTIMATION
Many choices of density estimation in high dimensional space exist, from simple Gaussian estima- tors to neural networks-bart methods such as autoregressive models or normalizing flows. The performance of these models may vary given the type of data: some models are more suited for images while others are better for text or lower dimensions. In our case, we wanted to find the most accurate model for the special case of state distribution in continuous control environments.
3

---

Under review as a conference paper at ICLR 2023
For this, we implemented candidate models, including Gaussian models (Multivariate, Mixture), autoregressive networks (RNade (Uria et al., 2013), Made (Germain et al.. 2015)), and normalizing flows (real-NVP (Dinh al., 2016), Maf (Papamakarios al., 2017), NSF (Durkan al., 2019)). Each of them was trained by maximum likelihood estimation over different continuous control environments, using sets of states obtained from pre-trained agents solving the task. We used various hyper parameters configuration for each model, and selected the best ones based on AUC criteria, then compared the different models when trained with their best hyper parameters. We found that normalizing flows performed significantly better than other models, and among normalizing flows NSF worked slightly better than other flows. This experiment detailed Appendix ??.
2.1.3 CONTINUOUS MAZE
As a tool scenario to test ChronoGEM, we implemented two-dimensional continuous maze in which actions dy, dy steps bounded by the [-1, 1]2 square, and the whole state space is bounded by the [-100, 100]² square. This environment, illustrated in Figure 2a similar to the maze used by Kamienny et al. (2021). except it adds a significant difficulty induced by the presence of two narrow corridors that needs to be crossed in order to reach the top-left room. The main goal of this experiment to verify that even in challenging tool game, ChronoGE. still manages to in- duce uniform distribution over the whole state space. In order to emphasize the relative difficulty of the exploration of this maze, we also run a random walk, SMM and RND to compare the obtained state coverages. We implemented ChronoGEM with N = 27 paralleled environments and branching factor know that if is large enough, all achievable states just every point in the maze, ChronoGEM could be uniform on the maze given that we let run for enough time (for instance, = 1000). However, in that given episode length, both RND and SMM failed at exploring beyond the first corridor, and random walk did not even explore the whole first room, as illustrated in Figure 3.
(a)
(b)
Figure 2: (a) Maze environment, the red star corresponds to the starting point and black volumes are walls. (b) Example of ChronoGEM trajectories in the Maze.
In this setup we
2.2 GOAL-CONDITIONED TRAINING
To build C3PO, we modified Brax' implementation of SAC to take set of goal as input and train to reach them. The reward the opposite of the maximum of the euclidean distance between body (e.g. arm, the torso, leg) and its goal position. As result the policy is encouraged to move to the correct location and then match the correct pose. The goal added to the observation as a relative position to the current position. agent's state and the goal is smaller that a tolcrance threshold e. In other terms, an episode & is successful when its closest state the goal is close enough: success(E)gg We set the environment to stop an episode as soon as it successful, or when exceeds the number of allowed steps. We initially the tolerance be high (1.) and slowly anneal it down when the success rate reaches 90% on the training data. As a result SAC learns first to move towards the
say that goal is reached if the Euclidean distance between the
min.ee ||s gl| <
(a) Random walk
(b) SMM
(c) RND
(d) ChronoGEM
Figure 3: fog-frequencies of the discretized states visitation when taking the last states from 4000 episodes, sampled according to a random walk (a), SMM (b), RND (c) and ChronoGEM (d), averaged over 10 seeds. Only ChronoGEM managed to visit states in the top-left room.
4

---

Under review as a conference paper at ICLR 2023
target and then to match the exact position. We call C3PO the resulting procedure that combines ChronoGEM for the training data collection and goal-conditioned SAC with tolerance annealing, as described in Algorithm 2.
Algorithm 2 C3PO
Require: Initial tolerance
1: Collect training goals G with ChronoGEM.
2: for enough steps do
3: Draw goals from G.
4: Run full episode rollout with drawn goals, considering an episode is successful the distance to the goal is less than
5: Add rollouts to the replay buffer.
6: If the success rate is above 90%, multiply by .99
7: Train networks using SAC losses on transitions drawn from the replay buffer.
8: end for
9: return Trained policy.
3 EXPERIMENTS
This section has two functions: 1) to quantify the superiority of C3PO compared to haselines in which the training data was collected using different exploration methods (SMM, RND and random walk) and 2) to illustrate the accuracy of the resulting goal-achieving policy after reaching asymptotical training performances, on various continuous control tasks, including Humanoid. To collect training data, ChronoGEM was with N = 2 paralleled environments and branching factor K 4 in all following experiments, except for Humanoid in which N = 2 and K = 64. We detail C3PO and all baselines (SAC+RND, SAC+SMM and SAC+random walk) implementations in Appendix ??. For each baseline, we separately tuned the hyper parameters in order to obtain the best performance in all different environments.
3.1 CONTINUOUS CONTROL TASKS
We used the control tasks from Brax (Freeman al., 2021) high dimensional environments. To compare the entropy and the richness of the obtained set states with bonus-based exploration baselines, we focused on four classical tasks: Hopper, Walker2d, Halfcheetah and Ant. Since we focus on achieving arbitrary poses and positions of the embodied agents, we modified the environments observations so they contain the (x, y. 2) positions of all body parts. All measures (cross entropy in section 3.2 and reaching distances in section 3.3) are based on that type of state. To get reasonable trajectories (as opposed to trajectories where HalfCheetah jumps the sky), we explore the environment within the low energy regime by putting a multiplier on the maximum for Hopper, .1 for Walker, ,01 for HalfCheetah and 1. for Ant. In the two
action. The multiplier considered episodes of length T 128. So the physical time horizon is following subsections, similar on all tasks, we added an action repeat of 6 for Hopper and Walker. All episode end conditions (because the torso is too low or too high for example) have been removed, have prior.
3.2 ENTROPY UPPER-BOUNE
Given a set of points ..... &x sampled from distribution with unknown density p. can estimate an upper bound of the entropy of given by the cross entropy (pp P) where ý is an estimation of pz
Hpp P) -Ez~~ ((x) = H(p) + KL(P))A) 2 H(p).
The estimation being trained by maximum likelihood specifically on the set of points, it directly minimises the cross entropy and closely approximate the true entropy. The KL term becomes negligible and only depends the accuracy of the trained model on the observed set of points, which supposedly does not differ given the different exploration method that generated the points. Consequently, comparing the cross entropy is similar comparing the entropy of the distribution
5

---

Under review as a conference paper at ICLR 2023
halfcheetah
hopper
walker2d
Cross entropy
-10
-20
-50
-20
40
nardon walk
RND
SMM
ChronaGEM
randon walk
RND
SMM
ChoroGEM
random walk
RND
ChronoSEM
random walk
RND
SMM
ChroneGEM
Figure 4: Distribution over 10 seeds of the cross entropies of the state visitation induced by Chrono-
GEM, RND, SMM and random walk, different continuous control tasks.
ChronoGEM
Random
SMM
20
20
20
10-
10-
10-
10-
-10
-10
-10
-20
-20
-20
10 -20
20 -10
20
-10
20
-10
10
Figure 5: Sky-view of the discretised spatial log-frequency covered by Ant with the different exploration methods. SMM has the largest scope but contains empty zones even close the origin. Both RND and ChronoGEM share similar range covering all directions, and ChronoGEM is visibly much more uniform, while other methods are concentrated on the origin. Note that this only represents the spatial positions, while both poses and positions are being explored.
induced by the exploration. In this experiment, we used this upper-bound to compare the efficiency of ChronoGEM compared RND, SMM and random walks. Figure 4 displays box plots over 10 seeds of the resulting cross entropy measured on the sets of states induced by different algorithms, on the 4 continuous control tasks. As expected, the random walk has the lowest entropy, and RND and SMM have, in average over the environments, similar performances. ChronoGEM has the highest entropy on all environments, especially on HalfCheetah, where it was the only method to manage exploration while the actions were drastically reduced by the low multiplier (see 3.1). In order to illustrate the fact that ChronoGEM induces a distribution that is close to the uniform, we measured the spatial coverage based discrete gird of the x-y plan: if the distribution is uniform over both the possible poses and positions, it should be in particular uniform over the positions. Figure 5 shows the resulting log-freen frgfeeuuyccy on the x-y grid visitations and if ChronoGEM not the method that induces the largest scope of exploration, it however has the most uniform coverage. We also report in appendix ?? the grid visitation in 2D environments (Hopper, Walker2d and Halfcheetah).
3.3 COMPARISON OF EXPLORATIOON METHODS VIA GOAL-CONDITIONED TRAINING.
an exploration method is good, drawing from the states it explored should good approximation of drawing from all achievable states. The state distribution induced by exploration algorithm can be used both as a training set of goal, but also as an evaluation set of goals. In the next experiment, for each environment, we every of the four examined exploration methods (ChronoGEM, Random Walk, SMM and RND) with 3 seeds to build 3 training goal sets per method and 3 evaluation goal sets per method. Training goal sets have 4096 goals and evaluation goal sets have 128 goals. We plot the success rate with regard to the tolerance, for each environment and evaluation goal set. Figure shows that evaluated on ChronoGEM goals, only C3PO which is trained on ChronoGEM gets good results while evaluated on goals from other methods. This is a good hint that the diversity of ChronoGEM goals is higher than other exploration methods. C3PO performs well on other evaluation sets as well, in particular in the low distance threshold regime (see Hopper and Walker). This can be explained by the fact that C3PO learns to reach a high variety of poses, since being able to achieve poses with high fidelity is what matters for low distance threshold regime. However, these achievement rates alone are still hardly interpretable: for example, being good at reaching goals generated by the random walk is less important than achieving harder goals, especially those from the highly entropic distributions (like ChronoGEM goals on Halfcheetah or SMM goals on Walker). We hence summarized the results by collecting all the areas under the curve (AUC), and weighting

---

Under review as a conference paper at ICLR 2023
them proportionally to the exponential of the evaluation goals entropy in Figure 7. Indeed, if a set very diverse, it means more to be able to achieve all of its goals, and vice-versa: a set is not diverse don't want to give too much importance to it, achieving always the same goal is not so interesting. The exponential of the entropy quantifies the number states in the distribution. We call this metric Entropy Weighted Goal Achievement (EWGA):
E Eeeevaluttion sets exp(entropyl (5) AU (methho on s)
EW G A(method)
Extevalation sets erp(entropy(s))
Success rate when evaluated on
1.0
ChronoGEM
1.0
Random
1.0
0.0
0.5
0.5
0.5
0.5
0.0 0.
072
04
06
0.8
1.0
0.0
05
0.25
0.50
0.75
1.00
0.0
0.0
02
0.4
06
08
3.0
0.0
0.2
6.4
076
08
17
0.000
0.06.00
0.0
0.5
0.5
0.5
0.5
0.0 0. 0.2
044
0.6
0.8
150
0.0
0.0
0.2
0'4
08 thresheld
D6
1.0
0.b
0.2
0/4
08 05
3.0
0.0 0.0
0.
6.4
0/6
03
Ustance
Distance thrsshold
Distance
C3PO tours]
RAT-Random
RAC+
Figure 6: For each environment (lines) and each set of evaluation goals (columns), success rates as function of distance thresholds obtained by SAC when trained on the different of training goals (ChronoGEM, Random Walk, SMM, RND). Each exploration algorithm was run over 3 seeds to collect evaluation and training goals, and each SAC training was also run over 3 seeds, so resulting curves are actually averaging different values.
3.4 MASSIVE GOAL-CONDITIONED TRAINING.
Now that established that ChronoGEM is the best exploration method for the purpose of producing training goals for a goal-conditioned setup, we will only use this method. We know allow ourselves train for massive amount of steps, and what is the best policy high parallelization and efficient infrastructure, it is possible to run 30G steps in couple days. We also add Humanoid our environments. By default, ChronoGEM would mostly explore positions where the humanoid is on the floor. However, was simple to modulate the algorithm to only explore uniformly in the space of state where the humanoid is standing. For example, can just associate
achieve. Thanks to Brax's 's
Ant
Halfcheetah
Hopper
Walker2d
0.2
0.2
0.2
EWGG
0.1
0.0
0.0
0.0
Walk
SACVRND
SAC+
CaPO loung)
Randon Walk
SAC+RND
SAC SAC+3MM
CIPO lours)
SAC Remdom Halk
SAC+RMO
sec SMM
CaPD lours)
S4C Random Halt
SAC+RND
SAC+SMM
CaPO lours)
SAC+ BAC-Rardon Rardom
Figure 7: Entropy Weighted Goal- Achieveeeev (EWGA). This estimates the ability policy to
achieve goal sets that better covers the space (for example, policy like C3PO that reaches larger variety of states has an higher EWGA than policy like SAC trained on the Random Walk, that only reaches states that close to the origin).

---

Under review as a conference paper at ICLR 2023
zero weight to undesired states during the sa-samp step. That way, avoided states in which the torso go under the altitude of .8 (the default failure condition). ChronoGEM is modified to not draw states where the humanoid is too low. The coal-conditioned learner gets high penalty for going too low too. The visual results of a policy able to achieve 90% success at .25 tolerance are represented in leverage it steer the exploration and
Figure 8. This shows that when we do have prior, policy learning.
(goe) in red)
Figure 8: For each environment, we drew goals from the ChronoGEM distribution and ran the policy trained on ChronoGEM goals. This figure represent the frame that is the closest from the goal for each episode, overlayed with the goal in red.
4 RELATED WORKS
This work is situated between various fields. Although effectively a goal-conditioned policy opti- mization algorithm, C3PO is enabled by the ChronoGEM exploration algorithm. We will first look at similar goal-conditioned learning setups, and then discuss more in-depth related works in the domain of exploration.
4.1 GOAL-CONDITIONED REINFORCEMENT LEARNING
Goal-conditioned RL (Kaelbling, 1993; Schaul et al., 2015) is the general setup of learning goal- conditioned policy instead of a specialized policy. We are particularly interested in goal-based setups where there is a-priori reward function. Although well known works such HER (Andrychowicz et al., 2017) demonstrate methods for learning goal-conditioned policies with minimal explicit exploration, more recent works (Pitis al., 2020; OpenAl al., 2021: Mendonea al., 2021) demonstrate the importance of having a good curriculum of goals to train from. MEGA (Pitis et al., 2020) extends style relabeling and answers the exploration problem by iteratively sampling goals according to a learnt density model of previous goals. ABC (OpenAl et al., 2021) demonstrates the importance of an adversarial curriculum for learning more complex goal-conditioned tasks, but is concentrated on specific tasks instead of arbitrary goal achievemnt. LEXA (Mendonca al., 2021) builds on Plan2Explore (Sekar et al., 2020), and demonstrates the importance both good exploration mechanism, well the of significant amounts of (imagined) data for learning arbitrary achieveeit policy. DIAYN (Eysenbach et al., 2018) uses a two-part mechanism that encourages the agent explore novel areas for a given latent goal, while at the same time learning
8

---

Under review as a conference paper at ICLR 2023
goal embeddings for different areas of the state space. While some of the above methods consider notions of density for exploration (Eysenbach et al., 2018), C3PO uses a more principled exploration mechanism, and is particularly interested in high-precision goal-achievement from full states.
4.2 BONUS-BASEE EXPLORATION
Although generally not concerned with condiiondi RL, there is a large family of exploration methods that are manifest as reward bonuses, with the intent of training a policy faster, or to be more performant. One family of approaches uses state-visitation counts that can be approximate to create an associated bonus for rarely-visited states (Bellemare et al., 2016; Ostrovski al., 2017). Prediction-eed bonuses use the residual error predictions of future states as a reward signal to approximate novel states, these includes methods such as RND (Burda et al., 2018) which leverages the prediction eπror of random projections of future states, SPR (Schwarzer et al., 2020) and BYOL- Explore (Guo et al., 2022), which make use of the seff-prediction error of the network with a frozen version of itself. Model-basee methods often optimise for next-state novelty, either by looking at the KL-Divergence between sampled states and likely states, such as in VIME (Houthooft et al., 2016) by explicitly driving towards states with high model ensemble disagreement such in Plan2Explore (Sekar et al., 2020). RIDE (Raileanu and Rocktäschel, 2020) and NGU (Badia et al., 2020) use episodic memory in which the bonus reflects the variety of different states covered in single trajectory.
4.3 ENTROPY MAXIMISATION
Some exploration algorithms, such as ChronoGEM. are constructed in order to maximize the entropy of the state visitation distribution. Most of them however, focus on the distribution induced by the whole history buffer (instead the just T-th states of episodes in ChronoGEMV), generally based on the behavior of trained policy. This is the case of MaxEnt (Hazan et al., 2019), GEM (Guo et al., 2021), SMM (Lee et al., 2019) and CURL (Geist et al., 2021). ). In APT (Liu and Abbeel, 20),b), instead of using density model estimate the entropy, they on the distance with the K nearest neighbors in a latent representation of the state space. APS (Liu and Abbeel, 2021a) combines APT's entropy bonus with an estimation the cross-entropy based on successor features to maximize the mutual information I (wr; s) between latent skill representations w and states.
non-parametric approach based
4.4 DIFFUSION-BAN BASED EXPLORATION
ChronoGEM is based on tree-structured diffusion, that makes selection of states, randomly explore from these states and then reselect states, etc. Go-Explore (Ecoffet et al., 2019), share the same steps, then selecting set of 'interesting' states, then
approach, by running random policy for going back these states and start again random explorations from them. The main difference with ChronoGEM is that we skip the "go back' part and we only perform one step of random actions before the new state selection. Also, the selection of states in ChronoGEM is provably converging uniform distribution over achievable goals, and does not need any additive prior about the state importance. Another close work also using a diffusion approach is UPSIDE (Kamienny et al., 2021). finds a set of nodes along with set of policies that connect any node the closest ones, looks for new nodes by random exploration from the existing ones, and remove non necessary nodes that are reached by the less discriminable policies. UPSIDE converges network of nodes that efficiently covers the state space.
5 CONCLUSION
We designed ChronoGEM, an exploration method that generates high entropy behaviors, in theory (Theorem ??) and in practice (Figure 4), outperforming baseline algorithms. All the skills discovered by an exploration algorithm can be used to train goal-conditioned policy. We showed that training ChronoGEM goals results in the most potent policies compared to other exploration methods. On Hopper, Walker, HalfCheetah, Ant and Humanoid, visuals and metrics show that the policy we trained able to achieve a large variety of goals by moving to the correct position and then matching the pose with high fidelity.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. (2017). Hindsight experience replay. Advances in neural information processing systems, 30.
Badia, A. P. Sprechmann, P.. Vitvitskyi, A., Guo, D., Piot, B., Kapturowski. S., Tieleman, O.. Arjovsky. M.. Pritzel, A., Bolt, A., al. (2020). Never give up: Learning directed exploration strategies. arXiv preprint N0VV:2002.06038
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2018). Exploration by random network distillation. arXiv preprint 1rXiv: 128994.....
Campos, V., Trott, A., Xiong, C., Socher, R., Giró-i Nieto, X., and Torres, J. (2020). Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317-1327. PMLR.
Dinh, L.,, Sohl-Dickstein, J., and Bengio, S. (2016). Density estimation using real nvp. arXiv preprint arXiv:1605.0880.
Durkan, C., Bekasov, A., Murray, I,, and Papamakarios, G. (2019). Neural spline flows. Advances in neural information processing systems, 32.
Ecoffet, A., Huizinga, J., Lehman, J.. Stanley, K. O., and Clune, J. (2019). Go-explore: new approach for eud-eexploration problems. arXiv preprint 1rXiv:1901.10995.
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, (2018). Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: arXiv:1802.06070.
Florensa, C., Held, D., Geng, X., and Abbeel, P. (2018). Automatic goal generation for reinforcement learning agents. International conference on machine learning, pages 1515-1528. PMLR.
Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O. (2021). Brax differentiable physics engine for large scale rigid body simulation.
Geist, M., Pérolat, J., Laurière, M., Elie, R., Perrin, S., Bachem, O., Munos, R., and Pietquin, O. (2021). Concave utility reinforcement learning: the mean-field game viewpoint. arXiv preprint arXiv:2106.03777.
Germain, M., Gregor, K., Murray, I., and Larochelle, H. (2015). Made: Masked autoencoder for distribution estimation. In International conference on machine learning, pages 881 881-889. PMLR.
Guo, Z. D., Azar, M. G., Saade, A., Thakoor, S., Piot, B., Pires, B. A., Valko, M., Mesnard, T., Latti- more, T., and Munos, R. (2021). Geometric entropic exploration. arXiv preprint arXiv:2101.02055.
Guo, Z. D., Thakoor, S., Pislar, M., Pires, B. A., Altché, F., Tallec, C., Saade, A., Calandriello, D.. Grill, J.-B., Tang, Y., et al. (2022). Byol-explore: Exploration by bootstrapped prediction. arXiv preprint arXiv:2206.08332
Hazan, E., Kakade, S., Singh, K., and Van Soest, A. (2019). Provably efficient maximum entropy exploration. In International Conference Machine Learning, pages 2681-2691. PMLR.
Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. (2016). Vime: Variational information maximizing exploration. Advances neural information processing systems, 29.
Kaelbling, L. P. (1993). Learning to achieve goals. In IJCAI, volume 2, pages 1094-8. Citeseer.
Kamienny, P.-A., Tarbouriech, J., Lazaric, A., and Denoyer, L. (2021). Direct then diffuse: In- cremental unsupervised skill discovery for state covering and goal reaching. arXiv preprint arXiv:2/10. 144557
10

---

Under review as a conference paper at ICLR 2023
Laskin, M., Yarats, D., Liu, H., Lee, K., Zhan, A., Lu, K., Cang, C., Pinto. L., and Abbeel, P. (2021). Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:2110.11111.
Lee, L., Eysenbach, B., Parisotto, E., Xing, E.,, Levine, S., and Salakhutdinov, R. (2019). Efficient exploration via state marginal matching, arXiv preprint
Liu, H. and Abbeel, (2021a). Aps: Active pretraining with successor features. In International Conference Machine Leurning, pages 6736-6747. PMLR.
Liu, H. and Abbeel, (2021b). Behavior from the void: Unsupervised active pre-training, Advances in Neural Information Processing Systems, 34:18459-18477
Mendonca, R., Rybkin, O., Danilidis, K., Hafner, D., and Pathak, D. (2021). Discovering and achieving goals via world models. Advances in Nearal Information Processing Systems, 34:24379- 24391.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning.
OpenAI, O., Plappert, M., Sampedro, R., Xu, T., Akkaya, I., Kosaraju, V., Welinder, P., D'Sa, R.,
Petron, A., Pinto, H. P. d. O., et al. (2021). Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2 101.0482.
Ostrovski, G., Bellemare, M. G., Oord, A., and Munos, R. (2017). Count-based exploration with neural density models. In International conference on machine learning, pages 2721-2736 PMLR.
Papamakarios, G., Pavlakou, T., and Murray, (2017). Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30.
Pitis, S., Chan, H., Zhao, S., Stadie, B., and Ba, (2020). Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pages 7750-7761. PMLR.
Raileanu, and Rocktäschel, T. (2020). Ride: Rewarding impact-driven exploration for procedurally- generated environments. arXiv preprint arXiv:2002. /2292.
Schaul, T., Horgan, D., Gregor, K., and Silver, D. (2015). Universal value function approximators. In International conference machine learning, pages
PMLR.
Schulz, L. (2012). Finding new facts; thinking new thoughts. Advances in child development and behavior, 43:269-294.
Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., and Bachman, P. (2020). Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929,
Sekar, R., Rybkin, O., Danillidis, K., Abbeel, P., Hafner, D., and Pathak, D. (2020). Planning to explore via self-supervised world models. In International Conference Machine Learning, pages 8583-8592. PMLR.
Silver, D., Schritwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017). Mastering the game of go without human knowledge. nature, 550(7676):354-355
Smith, L. and Gasser, M. (2005). The development of embodied cognition: Six lessons from babies. Artificial life, 11(1-2):13-29.
Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE.
Uria, B., Murray, I., and Larochelle, H. (2013). Rnade: The real-valued neural autoregressive density-estimator. Advances in Neural Information Processing Systems, 26.
Vinyals, O., Babuschkin, I., Chung, J., Mathieu, M., Jaderberg, M., Czarnecki, W. M., Dudzik, A., Huang, A., Georgiev, P., Powell, R., et al. (2019). Alphastar: Mastering the real-time strategy game starcraft ii. DeepMind blog, 2.
11


Reviews:
[{"review_id": "qz-iOzqqAi", "reviewer": "Reviewer_4RNg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method that performs massive uniform exploration of simulated environments and then trains goal-conditioned SAC with an annealed success condition to achieve states discovered during exploration. They demonstrate this exploration procedure to be superior on locomotion environments to other methods.", "summary_of_the_review": "While some of the results are nice, I have 3 main issues with this paper, as detailed above: 1) Framing, 2) The formatting, and 3) The experiments.\n\nEach of these is important and the formatting is critical (there are many papers already desk-rejected for incorrect formatting), and as such, I cannot recommend anything better than a strong reject.", "strength_and_weaknesses": "## Strengths\n\n**Clarity:** \n\n- The figures are, for the most part, very nice and clear.\n\n**Experiments/Results:**\n\n- ChronoGEM does work well as an exploration procedure, and GIFs in the supplementary demonstrate that their goal-conditioned policy can reach many arbitrary poses.\n- C3PO is able to do well on goal distributions generated by other methods, verifying the fact that ChronoGEM is a good exploration algorithm.\n\n## Weaknesses\n\n**Clarity:** \n\n- Quite a few grammatical hiccups throughout. Please fix (some that I found are listed in the minor details heading)\n\n****************Framing:****************\n\n- I somewhat doubt the utility of this work. It requires a specific setting (a cheap simulator that can be sampled en masse in parallel, and reset to arbitrary states), and has extremely poor sample efficiency due to this assumption. Essentially, ChronoGEM tries to visit all of the states in the simulator. In anything other than locomotion tasks, (e.g. robotic manipulation) these states will mostly be useless and having so many states to try to achieve can hurt learning.\n- What is the point of being able to achieve arbitrary positions and poses? The authors state in the abstract that it would allow for easier control and be re-usable as a key building block for downstream tasks. But they don’t show this. The experiments simply demonstrate that Goal-conditioned SAC can reach the position and poses given.\n- Conclusion: “In the real world, no reward function is provided.” THis is true, however in the real world we also can’t run ChronoGEM. This goes back to how I think the paper is not framed too well.\n\n**Experiments:** \n\n- Related to the above point about framing in achieving arbitrary positions and poses, the authors should show experiments where their method allows for better finetuning performance on some downstream tasks (maybe goal conditioned, maybe not) to validate the claims made in the abstract and intro.\n\n****************************Minor details:****************************\n\n- Grammar\n    - Intro last paragraph: “its’ → its”\n    - Sec 2.1: “necessary bounded” → “necessarily bounded”\n    - “up-left room” → “top-left room”\n    - “let it enough time” → “let it run for enough time”\n    - “we says” → “we say”\n- Questions\n    - In chronoGEM, how exactly do you sample the ****next**** states? Do you just reset BRAX simulation to those exact states and continue from there?\n\n**********************Formatting:**********************\n\n- The formatting is incorrect. The margins are way too small. This is unfair to other papers who fit everything into the page limit without the small margins. Regardless of whether this was intentional or on accident, I am giving a strong reject no matter what, until the authors fix this issue during the rebuttal.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ac2Y6lon4c4", "reviewer": "Reviewer_nd1d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "this paper is proposing a new unsupervised reinforcement learning scheme for being able to train a more taskable agent that can reach a large base of goals in the environment. the motivation for this work is to not only collect the data and have the agent be able to explore but at the same time be able to train a goal condition to policy such that if we wanted the agents to reach a particular state in the world, we could specify that during test time. the method proposes to do a type of incremental search outward from the initial State distribution and slowly add states that the agent can learn how to reach in this case in an effort to be able to sample from the state distribution such that goals can be reached uniformly across the state space. the method does perform some analysis showing that qualitatively the agents appear to learn and display a more uniform type of visitation strategy across the state space while visiting goals. and the method is compared across a handful of mujoco-based robotic simulations.", "summary_of_the_review": "The paper proposes a helpful algorithm to allow for training a goal-based agent to reach many goals. The work requires further details and comparisons to prior methods to understand if it is an improvement over prior work.", "strength_and_weaknesses": "pros\n- The paper proposes finding a more helpful policy that can be trained inside the environment compared to prior methods. Most prior curiosity-based methods do help the agents learn to discover different states that are more difficult to reach via exploration bonuses, but this method also seems to train a goal condition policy making it more taskable as well as prior methods.\n- They are also able to show through some illustrative examples that their proposed method can or at least appears to explore the environment in a more uniform manner.\n\ncons\n- There are some limitations in the comparison and analysis that should be addressed. In particular, some prior methods are mentioned in the paper that would be important to compare to understand better if the proposed method is better than prior work. this also notes and is based on the concept that they're training a goal condition policy to explore the state distribution and not just a curiosity-based method.\n- This method appears only to be applicable to simulators. As the authors note, it's also extremely data-hungry.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y0BSNKXc8L4", "reviewer": "Reviewer_3jQz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method called Entropy-Based Conditioned Continuous Control Policy Optimization (C3PO) that tackles the problem of learning a general goal-conditioned policy in two stages. In the first stage, using an exploration algorithm called ChronoGEM, an exploration dataset consisting of diverse goals is collected. In the second stage, the goals collected in the first phase are used as goal targets in goal-conditioned RL with soft-actor critic (SAC) as the base algorithm. The paper demonstrates the method’s effectiveness on Gym environments: walker, hopper, halfcheetah, and ant. It demonstrates that C3PO is not only able to reach a wider distribution of states compared to RND and SMM, training goal-conditioned policies on these states improves the ability of these policies to solve a range of goals.", "summary_of_the_review": "The proposed method has promising empirical results, but I find the current version of the theoretical justification unconvincing and the analysis of why the method works well could be more thorough. Therefore I do not think the paper should be accepted in its current form, but if the authors can address many of my concerns above, that would be very helpful.\n\nI also want to leave a note that this paper seems to be improperly formatted: the margins are much smaller than in the original template, the ICLR header is missing, and the title / author list formatting is nonstandard.\n", "strength_and_weaknesses": "Strengths:\n- The ChronoGEM algorithm appears empirically to generate more uniform state distributions compared to random walks, SMM, and RND exploration strategies on the maze and ant environments.  It is also conceptually simple to understand and implement.\nThe experimental results training goal-conditioned RL on the goals collected by ChromoGEN outperforms policies trained on the other goal distributions, which is a positive signal that the distribution of goals for ChromoGEM is relatively wide.\n\nWeaknesses:\n- The theoretical argument in section 2.1 does not seem convincing to me. The statement seems to be that the state visitation will eventually converge to be uniform over the entire state space, but the theorem in Appendix A only demonstrates that the states selected for the next step of the greedy procedure can be approximately uniform from KN sampled states. As the authors mention, playing uniform actions would not necessarily lead to a uniform sampling over the possible next states. So it seems like the “inductive” step here is missing. So I don’t agree with the statement in the conclusion that the method generates “high entropy behaviors, in theory (Theorem 1)”. \n- I think it would help the paper to have a comparison to a method like Skew-Fit[1] that has a similar exploration strategy to the one proposed in the paper by rebalancing the distribution of states to set as goals.\n- There isn’t an ablation study conducted on the impact of the tolerance annealing component from SAC to see how much of the improved performance comes from that compared to the goals from ChronoGEM.\n- The EWGA metric is an interesting way to evaluate the performance across evaluation sets that may have varying difficulties, but why not evaluate using another strategy like uniformly sampling goals across the [x, y, z] space? That seems like it would be less noisy, and that observation space has already been constructed.\n\n[1] Skew-Fit: State-Covering Self-Supervised Reinforcement Learning (Pong et. al, 2019)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject