Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning
Authors: Jinxin Liu
Donglin Wang
Qiangxing Tian
Zhengyu Chen
Venue:  (2021)
Keywords: unsupervised reinforcement learning
goal-conditioned policy
intrinsic reward

Abstract: It is of significance for an agent to learn a widely applicable and general-purpose policy that can achieve diverse goals including images and text descriptions.  Considering such perceptually-specific goals, the frontier of deep reinforcement learning research is to learn a goal-conditioned policy without hand-crafted rewards. To learn this kind of policy, recent works usually take as the reward the non-parametric distance to a given goal in an explicit embedding space. From a different viewpoint, we propose a novel unsupervised learning approach named goal-conditioned policy with intrinsic motivation (GPIM), which jointly learns both an abstract-level policy and a goal-conditioned policy. The abstract-level policy is conditioned on a latent variable to optimize a discriminator and discovers diverse states that are further rendered into perceptually-specific goals for the goal-conditioned policy. The learned discriminator serves as an intrinsic reward function for the goal-conditioned policy to imitate the trajectory induced by the abstract-level policy.  Experiments on various robotic tasks demonstrate the effectiveness and efficiency of our proposed GPIM method which substantially outperforms prior techniques. 

Under review as a conference paper at ICLR 2021
LEARN GOAL-CONDITIONED POLICY WITH INTRINSIC MOTIVATION FOR DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
It is of significance for an agent to learn widely applicable and general-purpent policy that can achieve diverse goals including images and text descriptions. Con- sidering such perceptually-speccin
goals, the frontier of deep reinforcement learn- ing research is learn a goal-conditioned policy without hand-crafted rewards. To learn this kind of policy, recent works usually take as the reward the non- parametric distance to a given goal in an explicit embedding space. From a different viewpoint, we propose novel unsupervised learning approach named goal-conditioned policy with intrinsic motivation (GPIM), which jointly learns both abstract-level policy and goal-conditioned policy. The abstract-level policy is conditioned on a latent variable to optimize a discriminator and discov- ers diverse states that further rendered into prrceptuall--speciff goals for the goal -ondiiondt policy. The learned discriminator serves as an intrinsic reward function for the goal-conditioned policy to imitate the trajectory induced by the abstract-level policy. Experiments on various robotic tasks demonstrate the effec- tiveness and efficiency of our proposed GPIM method which substantially outper- forms prior techniques.
INTRODUCTION
Reinforcement learning (RL) makes it possible to drive agents to achieve sophisticated goals in
complex and uncertain environments, from computer games (Badia et al. 2020
Berner et al. 2019)
to real robot control (Lee et al. 2018 Lowrey et al. 2018 Vecerik et al., 2019
Popov et al. 2017).
which usually involves learning a specific policy for individual task relying on task-specific reward. However, autonomous agents are expected to exist persistently in the world and have the ability to solve diverse tasks. To achieve this, one needs to build a universal reward function and design a mechanism to automatically generate diverse goals for training. Raw sensory inputs such as images have been considered as common goals for agents to practice on and achieve (Watter et al.) 2013 Florensa al. 2019 Nair et al. 2018 2019), which further exacerbates the challenge for designing autonomous RL. agents that can deal with such perceptually-speciff inputs.
Previous works make full of goal-achievement reward function available prior knowl- edge (Pong et al., 2018). such as Euclidean distance. Unfortunately, this kind of measurement in original space is not very effective for visual tasks since the distance between images does not cor- respond to meaningful distance between states (Zhang et al.) 2018). Further, the measure function applied in the embedding space, where the representations of raw sensory inputs are learned by means of using a latent variable model like VAE (Higgins et al. 2017b et al. 2018) or using the contrastive loss (Sermanet al. 2018, Warde-Farley et al. 2018). We argue that these approaches taking prior non-parametric reward function in original or embedding space above may limit the repertoires of behaviors and impose manual engineering burdens (Pong et al.) 2019).
In the absence of any prior knowledge about the measure function, standard unsupervised RL meth- ods learn a latent-conditioned policy through the lens of empowerment [Salee et al.) 2014 Ey- senbach et al. 22018) y Sharma et al. (2019) or the self-consistent trajectory autoencoder (Co-Reyes et 20.1 Hausman et al. (018)) However, the learned policy is conditioned on the latent variables rather than cereptuully-speciiie goals. Applying these procedures to goal-reaching tasks, similar to parameter initialization or the hierarchical RL, needs an external reward function for the new tasks; otherwise the learned latent-conditioned policy cannot be applied directly to user-specified goals.

---

Under review as a conference paper at ICLR 2021
Different from previous works, novel unsupervised RL scheme is proposed in this paper to learn goal-con coalicond policy by jointly learning an extra abstract-level policy conditioned on latent variables. The abstract-level policy is trained to generate diverse abstract skills while the goal- conditioned policy is trained to efficiently achieve perceptually-speciff goals that are rendered from the states induced by the corresponding abstract skills. Specifically, we optimize discriminator in an unsupervised manner for the purpose of reliable exploration (Salge et al. 2014) to provide the intrinsic reward for the abstract-level policy. Then the learned discriminator serves as an intrinsic re- ward function for the goal- colliiondiio policy to imitate the trajectory induced by the abstract-level policy. In essence, the abstract-level policy can reproducibly influence the environment, and the goal-conditiones policy perceptibly imitates these influences. To improve the generalization ability of coal-conditioned policy dealing with perceptually-specifi inputs, a latent variable model is further considered in the goal-conditioned policy to disentangle goals into latent generative factors.
The main contribution of our work is an unsupervised RL method that can learn perceptually-spent goal-conditioned policy without the prior reward function for autonomous agents. We propose novel training procedure for this model, which provides an universal and effective reward function for various perceptual goals, e.g. images and text descriptions. Furthermore, we introduce latent variable model for learning the representations of nigh-dimensional goals, and demonstrate the po- tential of our model generalize behaviors across new tasks. Extensive experiments and detailed analysis demonstrate the effectiveness and efficiency of our proposed method.
PRELIMINARIES
RL: An agent interacts with an environment and selects actions in RL so as to maximize the ex- pected amount of reward received in the long run (Sutton & Barto 2018), 8), which can be mod- eled as a Markov decision process (MDP) (Puterman 2014). An MDP is defined as a tuple M (S, A, p. r), where S and A are state and action spaces, p( a) gives the next-state distribution upon taking action a in state 8, R(s, 8, is a random variable representing the reward received at transition s - s', and 7 E [0, 1) is a discount factor.
Intrinsic Motivation: RL with intrinsic motivation obtains the intrinsic reward by maximizing the mutual information between latent variables w and agent's 's behaviors r: I[w;;T, where the specific manifestation of T can be an entire trajectory (Achiam et al., 2018). an individual state (Fysenbach) et al. (018)) or final state (Gregor et al. 2016); and the specific implementation includes reverse and forward forms (Campos et al. 2020). Please refer to Aubret et al. (2019) for more details.
Disentanglement: Given observation with dimension of N. VAE (Kingma & Welling 2013 Higgins et al. 201 is a latent model that pairs a top-down encoder q(>[æ>) with bottom-up de- coder network p(z] 2) by introducing latent factor where dim(z) < NN. To encourage the in- ferred latent factor a to capture the generative factor of x in a disentangled manner, an isotropic unit Gaussian prior p(z) N(0; T) is commonly used to control the capacity of the information bottleneck (Burgess et al. 2018) by minimizing the KI. divergence between g(2 (z) and p(2.
THE METHOD
In this section, we firstly formalize the problem and introduce the framework. Secondly, we elab- orate on the process of how to jointly learn the goal-conditioned policy and abstract-level policy. Thirdly, disentanglement applied in our setting to improve the generalization ability.
3.1 OVERVIEW
Given perceptually-sseccfic goal g, our objective is to learn a goal-conditioned policy no (alš, g) that inputs state ă and g and outputs action a as shown in Fig. The abstract-level policy n (a(a,, w) takes as input state and a latent variable w and outputs action a, where w corresponds to diverse latent skills. The discriminator 9e is firstly trained at the abstract-level for reliable exploration, then it provides the reward signal for the goal-conditioned policy to imitate the trajectory induced by the abstract-level policy. The abstract-level policy is able to generate diverse states s that are further rendered as diverse perceptually-specffc goal g Render(s). On this basis, mo (als, g)
2

---

Under review as a conference paper at ICLR 2021
T State c+1
Skill w~ o~p(w)
Abstract-level Policy ar-Hy Tp (ar S5: w))
Environment
Discriminator s E St+11
Goal--onditioned Palicy ar (4g 15p.51p.. St)
Environment
Yt
State 5t+1
Learned
Goal @
Fixed
Non-differentiable render
Figure I: Framework of GPIM. We jointly train the abstract-level policy n and the discriminator Ya to understand skills which specify task objectives (e.g., trajectories, the final goal state), and use such understanding to reward the goal-conditioned policy for completing such (rendered) tasks. State St+1 (c.g.. position of agent) induced by म. is converted into perceptually-speciff goal ge (e.g., image showing the position of the agent) for πι. Note that the two environments in the figure are identical, and the initial states Sp of n. and &y of my are sampled from the same distribution.
conditioned on the rendered goal interacts with the reset environment under the instruction of the reward function Qg. We the non-tilde and the tilde á to distinguish between the states of two policies respectively. Actually, and come from the same distribution.
3.2 PROPOSED GPIM METHOD
In order jointly learn the abstract-level policy (a(sssw) and goal-conditioned policy нa (a|3, g). we maximize the mutual information between the state s and latent variable for ти, and simultaneously maximize mutual information between state and goal g for Ty. Consequently, the overall objective function to be maximized can be expressed follows
Th. env.
Render
Tgg,
Figure Graphical model.
Flu, I(s;w) + ((;gg)
For clarification, Fig. 22 depicts the graphical model for the latent variable w, state induced by Tu goal g rendered from s, and state ø induced by иT. As seen, the latent variable ~ p(w firstly used to generate state via the policy मH interacting with dynamic environment. Then, we use the s to generate percepuually-speeffe goal g. After that, the goal-conditioned policy To outputs action a to interact with the environment to obtain the state at the next time step. In particular, w is expected to generate diverse behavior modes through TT while To behaving like š to imitate this behavior by taking input the rendered goal g. Based on the context, the correlation between & and g is no less than that between ş and w. so that I(;:g) 2 I(3:w) (Beaudry & Renner 2011). Thus, can obtain the lower bound of F(u, 0):
F(,,0) > I(s:w) + I(3:w)
2H(w) - H(wl(s) - H(w(3)
E.,. [log poupp(wa (s) + ((w)(()))) 2 logp(w)).
Since it is difficult to exactly compute the posterior distributions p(wls) and p(w(3), Jensen's In- equality (Barber Agakov| 2003) further applied for approximation by using a learned discrimi-
Too further clarify the motivation, we conduct the ablation study compare our method with that just maximizing I(s;w) that just maximizing Z(3; g) in Appendix A.1
"p(w) denotes the prior distribution. Fixing p(w) instead of learning it is prevent T. from collapsing to sampling only handful of skills. In experiment, use categorical distribution for p(w) following DIAYN.
3

---

Under review as a conference paper at ICLR 2021
Algorithm Learning process of our proposed GPIM
1: while not converged
13:
Step
2: Sample skill ~ p(w)).
14:
Sample initial state &0 ~ Po(3).
3: # *Step I
15:
for f t= 0, 0,11, 1,... T - steps do
4: Sample initial state so ps)).
16:
Sample action
(a: g+)
5: for 0, T steps do
17:
Step environment: #c+1 18 a.
Sample action de ~ Np. (as St, (.)
18:
Compute reward Fg for policy Ry using 77
7: Step environment: Sc41 p( (8, at).
19:
Update policy Ty to maximize Ft with SAC
8:
Render goal: 9r = Render(s141).
(0 {ūg, 0cf when considering the dis-
9:
Compute reward for policy nu using 5
entanglement).
Update policy na to maximize ra with SAC.
20:
Update ig and dp by maximizing Doss [ using SGD,
11: Update diseriminator (ga) to maximize
21:
end for
log to (w18x41) with SGD.
22: end while
12: end for
nator network qo (wl-) (see Appendix B.1 for the derivation). Thus,
F(u, 0) E.,,, à [log (o(w)s) + log (w(l)) 2 logg((w] & [H, 6,
where it is worth noting that the identical discriminator qø with the parameter ó is used for the variational approximation of p(wls) and p(w]5). For the state induced by skill w and originat- ing from g. the shared discriminator ds assigns a similarly high probability on w for both states and associated with the same w. Intuitively, we factorize the abstract-level policy and learn it purely in the space of the agent's embodiment (i.e., the latent variable w) separate from the percepuullly-specifee
goals (e.g., images and text descriptions). The space of goals these two spaces has different characteristies due to the underlying manifold spaces. Therefore, 9s can be regarded as a reward network shared by the abstract-level policy T (a((ass,) w) and goal-conditioned policy mo (als.g). g)-
According to the objective function [A, o, in we propose an alternating optimization be- tween T and Ty as follows:
Step I: Fix To and update Tp and qø. In this case, 0 is not a variable to update and J(H s 0) becomes
T(, Ø) = Eu,, [log (o(w())) + Eu,s [log qo (w][) - 2 log p(w)]
(4)
Variable independent
According to (4) I I(, can be thus optimized by setting the reward at time step i for Ty.
rz log (w] (w]()(1) log p(w),
(5)
where the term - log p(w (w) is added for agents to avoid artificial termination and reward-hacking issues (Amodei al. 2016). We implement this optimization with soft actor-critic (SAC). On the
other hand, the reward network ds can be updated with SGD by maximizing Ew., [log (w][s))
Step II: Fix Ta and g to update тe. In this case, f and á are not variables update any more and T(H, o, 0) can be simplified as
I Eu,s [log go (w]]] + En,. [log yo (w|s) - 2 log p(w))
(6)
Variable independent
According to (6). [0) can thus be optimized by setting the reward at time step / for Te as
F = log go (w[3 - log p(w),
where the term - ((w) is added for the same reason as above and we also implement this opti- mization with SAC. These two steps are performed alternately until convergence. The overall GPIM summarized in Algorithm [ which includes the disentanglement (red) in the next subsection.
3.3 IMPROVE GENERALIZATION VIA DISENTANGLEMENT
The goal 9r can be further disentangled (Higgins et al. 201 7a) so as to improve the generalization ability of the goal-conditioned policy ma.
4

---

Under review as a conference paper at ICLR 2021
As shown in Fig. decompose To into two components: the encoder network E parameter- ized by 1 and the generative network G pa- rameterized by 0g. So we have Pao (ax|8;, 40g (=lg+))-
52-
ag
By (a 15p, gr)
Forward
To encourage this disentangling property in the inferred gög (zl9:), we adopt an prior p(z) N (0; 1) to control the capacity of the informa- tion bottleneck by minimizing the KL diver-
Backprop.
Figure 3: Disentanglement of policy Tg.
gence between q0 (z1gc) and p(z). Hence, the overall objective function of goal-conditioned policy Ty becomes
max 0R.0G.0D
Bu,s, [Ft] + Egg, [pop (9e
3 BBg
Dis
where a and are two hyperparameters. is worth noting that the update of the encoder is based on gradients from both ma and Dis_loss With this disentanglement, the overall GPIM method is given in Algorithm D as mentioned above.
RELATED WORK
Investigating goal distribution: Many prior methods (Schaul et al. 2015: Andrychowicz et al. 2017
Pong et al. 2018: Hartikainen et al., 2019) assume an available distribution of goals during explo- ration for rewrse-reward problems, where the resampled goals are restricted to states encountered along the trajectory (Andrychowicz al. 2017 Levy al. 2017) in an embedding space (Nair eet al. 2018). 'Te setting of goals for learning policy has been discussed in Baranes & Oudeyer
(2013). Recently, several works (Colas al. 2018 Warde-Farley al. 20188 Florensa al. a 2001 Nair et al. 2018 Péré et al. 2018) adopt heuristics to design goal distribution based on previously visited states. However, many of these (Nair et al.) 2018 Pong et al., 2019 Warde-Farley et al. 2018) assume an ideal matching of the goal space with the state space.
Learning goal-conditioned policy: Several recent works propose various methods for agents to au- tomatically learn policies by maximizing the mutual information between behaviors (e.g., latent variables) and tasks (e.g., goals). DISCERN (Warde-Farley al.) 2018) simultaneously learns a coal-conditioned policy and goal achievement reward function that measures how similar state is to the goal state. Pong et al. (2019) concurrently trains a goal-reaching policy and maximizes the entropy of the generated goal distribution. Given the difficulty of vision-based RL, Pong et al. (019)) resorts to RIG (Nair et al. 2018), which obtains reward using prior non non-parametric measure function in embedding spaces. Other unsupervised methods (Sharma et al. 2019 Campos et al. 2020), capable of learning diverse skills, are difficult to apply directly to user-specified tasks.
Learning disentanglement representation: Many prior works utilize unsupervised learning in RL to acquire disentanglement representation (Bengio et al. 2013) Higgins et al. 2017a Chen et al.) 2016
Burgess et al. 2018). Some works take the disentang lement representation as a substitute for states
mememtt
(Higgins et al. 2017b; Ha & Schmidhuber] 2018) Jonschkowski et al.) 2017 Finn et al. 2016 Lange
20122 Watter al. [2015] /rrinias et al.) 2018) Nair et al.] 2019) Srinivas et al. 2018 while
others consider a non parametric metric in latent space to acquire a reward function (Nair et al.,
2018 Sermanet et al. 2018). Besides, a few works directly embed manipulation tasks to generalize
to long-horizon tasks (Goyal et al. 2019. Hausman et al. 2018 Mandlekar et al. 2020).
Hindsight, self-play and knowledge distillation: Our method is similar spirit to goal relabeling methods like hindsight experience replay (HER) (Andrychowicz et al. 2017) which replays each episode with a different goal in addition to the one the agent was trying to achieve. However, HER requires prior reward function and hand-crafted goal space. By contrast, GPIM is unsupervised and able to find its own goal space. The self-play (Sukhbaatar et al. 2017 2018) is closely related to our scheme, leading to emergent autocurriclla by pitting two versions of the same agent against one another. However, these approaches generally require prior reward functions. As for knowledge distillation (Xu et al., 2020), we aim at extracting the relationship between different tasks.
5

---

Video is available under Under review as a conference paper at ICLR 2021
(1A,244
(a) 2D navigation
(b) Ohject manipulation
(c) Atari games
(d) Swimmer
(c) cheetah
(f) Fetch
Figure 5: Goals and learned behaviors by GPIM: Dots in 2D navigation (left subfigure, x-y goal) and atari games denote different final goal states, and curves with same color represent corresponding trajectories; Goals in 2D navigation (right subfigure, color-shape goal) and object manipulation are described using the text at the top of the diagram, where the purple lines imply the behaviors; In mujoco tasks, the first (swimmer, half cheetah and fetch) and third (swimmer and half chectah) rows represent the expert trajectories, and each row below represents the corresponding behavior.
EXPERIMENTS
Extensive experiments are conducted to evaluate our proposed GPIM method, where the follow- ing four questions will be considered in the main paper: 1) By using the "archery" task, we clarify whether is provide effective reward function on learning the goal-conditioned pol- icy тT. Furthermore, more complex tasks including navigation, object manipulation, atari games, and mujoco tasks introduced to answer: 2) Does model learn effective behaviors condi- tioned on a variety of goals, including high-dimensional images and text descriptions that are het- erogeneous states? Does the proposed GPIM learning goal-conditioned policy outper- form baselines? 4) Does the learned reward function produce better expressiveness of tasks, com- pared with the prior non-parane parammerei function the embedding space? https://itess.oogll... wtt.g//.....o..wwwg.i.....
Visualizing the learned reward function. We start with simple "archery" task to visualize how the learned reward function (discriminator (a) accounts for goal-conditioned behaviors in dynamics. The task shown in Fig. 4 requires choosing angle at which we shoot an arrow to the target. The left upper subfigure shows that in a deterministic environment, given three different but fixed targets (with different col- ors), the arrow reaches the corresponding target successfully under the learned reward function da- The reward can find that the learned reward functions resemble convexes in terms of the distance between final states to targets, where vertexes appear in the positions of the corresponding targets. Specifically, the maximum value of the learned reward function is achieved when the final state is close to the given target. The farther away the agent's final state is from the target, the smaller this reward value is. Similarly, the same conclusion can be drawn from the stochastic environment in the left lower subfigure, where the angle of the arrow has a 50% chance become a mirror-symmetric angle. We see that the learned reward function substantially describes the dynamics and the corresponding tasks, both in deterministic and stochastic environments. This answers our first question.
Deterministic
Deterministic
Stochastic
1.0
mecee
Targets
Stochastic
archery
Sive
0.4
Targets
migle
Final state
Final state
Figure 4: "Archery" tasks and learned rewards.
function of the final location of arrows in three tasks is shown on the right. We
'Due the page limitation, experimental analysis and results included in the Appendix.

---

Under review as a conference paper at ICLR 2021
Scaling to more complex tasks. To answer our second question, we now consider more complex tasks as shown in Fig. [ 1) In 2D navigation tasks, an agent can move in each of the four cardinal directions. We consider the following two tasks: moving the agent to specific coordinate named x-y goa and moving the agent to a specific object with certain color and shape named color-shape goal. 2) Object manipulation considers moving agent in 2D environment with one block for manipulation, and the other block as distractor. The agent first needs to reach the block and then move the block the given location, where the block described using color and shape. In other words, the description of the goal contains the color-shape goal of the true target block and the x-y goal of the target coordinate. 3) Three atari games including seaquest, berzerk and montezuma revenge require agent to reach the given final states. 4) We use three mujoco tasks (swimmer, half cheetah, and fetch) taken from OpenAl GYM (Brockman et al. 2016) to fast imitate given expert trajectories. Specifically, the goals for Ty in 2D navigation, object manipulation and atari games are the rendered final state ST induced by abstract-level policy Ty² 9t = (ender(sTT) and the goals for Ty in mujoco tasks are the rendered trajectories induced by TT, {go. 91.... = Render([ (2, s2..).).
The left subtigure of Fig. ((a) shows the learned behaviors of navigation in continuous action space given the x-y goal which denoted as the small circle, and the right subfigure shows the trajectory of behavior with the given color-shape goal. More results on discrete navigation can be found in Appendix [D2] As observed, the agent manages to learn navigation tasks by using GPIM. Further,
2D navigation with color-shape goal (Fig. [ and object manipulation tasks (Fig. 5(b) show
the effectiveness of our model facing heterogeneous goals and states. Specifically, Fig. 5( shows
the behaviors of the agent on object manipulation, where the agent is asked to first arrive at a block (i.e.. blue circle and green square respectively) and then push it to the given location inside a dark circle (i.e., [6.7, 8.0] and [4.8, 7.9] respectively), where the red object exists as a distractor. Fig. 5(c) shows the behaviors of agents that reach the final states goal) space on seaquest, berzerk and montezuma revenge respectively. Fig. d-ff shows how the agent imitates expert trajectories of swimmer, half cheetah and fetch.
higher dimensional (action, state and
By learning to reach diverse goals generated by the abstract-level policy and then disentangling the goals, the agent learns the ability to infer new goals later encountered by the agent. For example, as in Fig. 5(a) (right,, learning three behaviors with the goal of red-square, red-circle or blue-square in a gray background makes the agent accomplish the new goal of blue-circle in a white background. We further conduct the ablation study in Appendix AA.3 to show how the disentanglement affects the learned behaviors, and use more experiments to show the generalization of the coal-conditioned policy unseen goals in Appendix A.4
Solving complex temporally-extended tasks is long-standing RL problem (Jiang et al. 2019). For this purpose, we further make agents im- itate the given composite behaviors (i.e., ex- pert behaviors) and show their performance in Fig. [ Specifically, we consider the fetch task as in Fig. 5() where the gripper of the robot arm is used to render a series of 3D coordinates as goals for imitation in training phase. Dur- ing test, we employ two parameterized complex curves for the gripper to follow: (I, y, z) (tnitt)) /50, t cts 1/50, - -lo((t + 1)/5) (left) and (x. y. 2) = (t/5, cos(t)/5 - 1/5, (t)/t) (right). It is worth noting that during training the agent is required to imitate large number of simple behaviors and has never seen such complex goals before testing. It is observed from Fig. thtt the imitation curves are almost overlapping with the given expert trajectories, indicating that the agent using GPIM has the potential to learn such compositional structure of goals during training and generalize to new composite goals during test.
Expert behavior
Learned behavior
Start
Fetch
Figure 6: Expert behaviors and learned behaviors.
Comparison with baselines. For the third question, we compare our method to three baselines: RIG (Nair et al. 2018), DISCERN (Warde-Farlcy et al.] 2018), and L2 Distance. L2 Distance measures the distance between states and goals, where the L2 distance -I|s sg / is con- sidered with a hyperparameter Opizet- Note that 2D navigation with the color-shape goal and object manipulation using text description makes the dimensions of states and goals different, so L2 can- not be used in these two tasks. In RIG, we obtain rewards by using the distances in two embedding
DDfinitions of x-y goal, color-shape goal, and environment details are given in Appendix B.2
7

---

Under review as a conference paper at ICLR 2021
0.4
1.
Mantonerna Rleoogg
Switmnrr
HolChhetth
hotch
Figure 7: Performance (normalized distance to goals vs. training steps) of our GPIM and three baselines, where GPIM generally achieves smaller distances to goals in comparison to baselines.
spaces and learning two independent VAEs, where one VAE is encode states and the other is code goals. We use the normalized distance to goals 5 as the evaluation metric, where we randomly sampled 50 samples (tasks) from the goal space.
We show the results in Fig. 7 by plotting the normalized distance to goals as a function of the number of actor's steps, where each curve considers 95% confidence interval in terms of the mean value across three seeds. As observed, our GPIM consistently outperforms baselines in almost all tasks except for the RIG in 2D navigation (x-y goal) due to the simplicity of this task. Particularly, as the task complexity increases from 2D) navigation (x-y goal) to 2D navigation (color-shape goal) and eventually object manipulation (mixed x-y goal and color-shape goal), GPIM converges faster than baselines and the performance gap between our GPIM and baselines becomes larger. Moreover, although RIG learns fast on navigation with x-y goal, it fails to accomplish complex navigation with color-shape goal because the embedding distance between two independent VAEs has difficulty in capturing the correlation of heterogeneous states and goals. Especially in high- figh-dimensional action space and more exploratory tasks (atari and mujoco tasks), our method substantially outperforms the baselines.
To gain more intuition for method, record the distance (Ar) between the final state induced by Te and the goal rendered by Ty.
Ar: Distance Rendered Goals
not update update ne with aa
0.4
throughout the training process of the 2D navi-
gation (x-y goal). For this purpose, in this spe- cific experiment, we update Ty and ga but ig- nore the update of my before 200 k steps to show
0.2
na successfully imitates Ry.
0.5M
1M
1.5M
2M
the exploration of TT at the abstract level. As shown in Fig. g Ar steadily increases during the first 200 k steps, indicating that the abstract- level policy T explores the environment (i.e., goal space) to distinguish skills more easily, and as a result, generates diverse goals for training тT. After around 1.5 M training steps, Ar al- most comes to 0, indicating that the goal-conditioned policy my has learned good strategy reach the rendered goals. In Appendix [A.2 we visually show the generated goals at the abstract level in more complex tasks, which shows that our straightforward framework can effectively explore the environment without additional sophisticated exploration strategies.
Actor Steps
Figure 8: Abstract-level policy Tp gradually ex- plores the environment, generating more difficult goals. ls encourages To to gradually mimic TH.
Expressiveness of the reward function. Particularly, the performance of unsupervised RL meth- ods depends on the diversity of autonomously generated goals and the expressiveness of the learned reward function, which conditioned on the generated goals. We have shown that our straightfor- ward framework can effectively explore the environment, achieving competitive performance with baselines (see appendix A.2 The next question is that: with the same exploration capability to generate goals for training, does our model achieve competitive performance against the haselines? Said another way, will the obtained reward (over embedding space) of baselines taking the prior non- parametric function limit the repertoires of learning tasks in a specific environment? Our next exper-
Definition and further descriptions be found in Appendix B.3
8

---

Under review as a conference paper at ICLR 2021
GRM
+DISCEW
DVAYN RYS
maze task Dey goal)
1.0
Start:
Gool
DiSCi Ri: Leaming Pronece of the rewserd Tunetion
0.75
-0.4
0.5
0.2
RIG: Reward funcioon
GPMM Leariinng Proceus (he reward Lanstion
3M 2M
Figure 9: (Left) The maze environment and reward functions. The heatmaps depict (the learning process of) the reward function conditioned on the specific task reaching the left-bottom star (RIG and DISCERN) or "imitating" the trajectory induced by abstract-level policy (GPIM). Specifically, the learning process of DISCERN's reward function refers to the learning process of embedding state. Note that the reward functions of baselines are conditioned on the goals, while GPIM's reward function is conditioned on the skill So, the induced trajectory by GPIM conditioned on the same skill refines over training steps (at the abstract-level), as shown in the bottom. (Right) Learning curves for GPIM and the enhanced baselines (DIAYN+RIG and DIAYN+DISCERN, Compared with our model, baselines ignoring the dynamic of the maze environment exhibit poor performance.
iment studies the expressiveness of the learned reward function. For better graphical interpretation and comparison with basclines, we simplify the complex Atari games to a maze environment shown where the middle wall poses bottleneck state. Campos et al. 2020) shows that the canon- ical information-theoreti skill discovery methods suffer from a poor coverage of the state space. Here, borrowing the idea from state marginal matching (Lee al. 2019). we set the reward for the abstract-level policy as (Jabri et al. 2019 r = X [log q (c) 8
in Fig. [
log p(w)]] +A log (u (s++
where qu a density model, and E [o, ] can be interpreted as trade off between discriminability of skills and task-specific exploration (here we set À = 0.5). Note that we modify rς for improving the exploration on generating goals, and we not change the reward for training the goal-conditioned policy Tg. To guarantee generation of the same diverse goals for training goal-conditioned poli- cies of baselines, we adopt DIAYN taking the modified reward rf to generate goals for RIG and DISCERN, denoted as DIAYN+RIG and DIAYN+DISCERN respectively.
In Fig. | we show the visualized learned reward on a specific task reaching the left-bottom star, and the learning curves the maze task, where the testing-goals -galls are random sampled. We can see that the learned reward functions of RIG and DISCERN produce poor signal for the goal-conditioned policy, which makes learning vulnerable local optima. Our method builds up the reward function after exploring the environment, the dynamic of which itself further shapes the reward function. In Fig. (left), can see that model provides the reward function better expressiveness of the task by compensating for the dynamic. This produces that, even with the same exploration capability to generate diverse goals, our model sufficiently outperforms the baselines, shown in Fig. 9 (right).
CONCLUSION
We propose a novel GPIM method learn goal-conditioned policy in an unsupervised manner. Specifically, we optimize a discriminator in an unsupervised manner for the purpose of reliable exploration to provide the intrinsic reward for the abstract-level policy. The learned discriminator then serves as an intrinsic reward function for the goal-conditioned policy to imitate the trajectory induced by the abstract-level policy. Experiments on a variety of robotic tasks demonstrate the effec- tiveness and efficiency of our proposed method which substantially outperforms prior unsupervised techniques.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 1807. 10299, 2018.

---

Under review as a conference paper at ICLR 2021
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint 1xxiv:1606.066666 2016.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In Advances in neural information processing systems, pp. 5048-5058, 2017.
Arthur Aubret, Laetitia Matignon, and Salima Hassas. A survey on intrinsic motivation in reinforce- ment learning. arXiv preprint 1eXiv:/908 06976, 2019.
Adria Puigdomenech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. arXiv preprint arXiv:2003. 13350, 2020.
Adrien Baranes and Pierre- Yves Oudeyer. Active learning of inverse models with intrinsically mo- tivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49-73, 2013.
David Barber and Felix V Agakov. The im algorithm: variational approach information maxi- mization. In Advances in neural information processing systems, pp. None, 2003.
Normand J Beaudry and Renato Renner. An intuitive proof of the data processing inequality. arXiv preprint /rXiv: 100.00000000.. 2011.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1788)117888 2013.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1912.00912.06000 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins. and Alexander Lerchner. Understanding disentangling in beta-vae. arXiv preprint arXiv: 2018.
Victor A. Campos, A. Trott, Caiming Xiong, R. Socher, Xavier Giro Nieto, and Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. ArXiv, abs/2002.03647,, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp- 2172-2180, 2016.
John D Co-Reyes, YuXuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajec- tory embeddings. arXiv preprint arXiv: 1806.011800002888 2018.
Cédric Colas, Olivier Sigaud, and Pierre-Y Yves Oudeyer. Gep-pg: Decoupling exploration and ex-
ploitation in deep reinforcement learning algorithms. arXiv preprint wwXXiv1802.05033 2018
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint I (00.::80000000 2018.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 512-519. IEEE, 2016.
Carlos Florensa, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, and Martin Ried- miller. Self-supervised learning of image embedding for continuous control. arXiv preprint arXiv: PeXivv1901.000 1901100903..... 2019.
10

---

Under review as a conference paper at ICLR 2021
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the in- formation bottleneck. arXiv preprint arXiv:1901.10002 2019.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv: 1.07507, 2016.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv: 2018.
Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. In International Conference on Learning Representations, 2019.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. Iclr, 2(5):6, 2017a.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 1480-1490. JMLR. org, 2017b.
Allan Jabri, Kyle Hsu, Abhishek Gupta, Ben Eysenbach, Sergey Levine, and Chelsea Finn. Un- supervised curricula for visual meta-reinforcement learning. In Advances in Neural Information Processing Systems, pp. 10519-10531, 2019.
Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstrac- tion for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems, pp. 9419 99-91,, 2019.
Rico Jonschkowski, Roland Hafner, Jonathan Scholz, and Martin Riedmiller. Pves: Position- velocity encoders for unsupervised learning of structured state representations. arXiv preprint /rXiv:1705.09800 09805, 2017.
Rawal Khirodkar, Donghyun Yoo, and Kris M Kitani. Vadra: Visual adversarial domain randomiza- tion and augmentation. arXiv preprint arXin:1812.0049), 2018.
Diederík Kingma and Max Welling. arXiv:1312.61 14,61111400011 2013.
Auto-encoding variational bayes.
arXiv preprint
Sascha Lange, Martin Riedmiller, and Ame Voigtländer. Autonomous reinforcement learning raw visual input data in a real world application. In The 2012 international joint conference on neural networks (ICCNN) pp. 1-8. IEEE, 2012.
Lisa Lec, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint arXiv:/906.000 2019.
Youngwoon Lee, Shao-Hua Sun, Sriram Somasundaram, Edward S Hu, and Joseph Lim. Com- posing complex skills by learning transition policies. In International Conference on Learning Representations, 2018.
Sergey Levine. Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.016A 2020.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint arXiv: 1712.0095 2017.

---

Under review as a conference paper at ICLR 2021
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. arXiv preprint arXiv:1811.01848, 2018.
Ajay Mandlekar, Danfei Xu, Roberto Martin-Martin, Silvio Savarese, and Li Fei-Fei. Learn- ing to generalize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.06083 2020.
Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong, Glen Berseth, and Sergey Levine. Contextual imagined goals for self-supervised robotic learning. arXiv preprint arXiv:1910.1100 2019.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Vi- sual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pp. 9191-9200, 2018.
Alexandre Péré, Sébastien Forestier, Olivier Sigaud, and Pierre-Yves Oudeyer. Unsupervised learn- ing of goal spaces for intrinsically motivated goal exploration. arXiv preprint arXiv: 00781, 2018.
Vitchyr Pong. Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model- free deep for model-based control. arXiv preprint arXiv: 0908,, 2018
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.000 2019.
Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Ve- cerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. Data-efficient deep rein- forcement learning for dexterous manipulation. arXiv preprint arXiv: 10X.:v/70400073: 2017.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley Sons, 2014.
Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment-an introduction. In Guided Self-Organization: Inception, pp. 67-114. Springer, 2014.
Tom Schaul, Daniel Horgan, Karol Gregor. and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320, 2015.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1134-1141. IEEE, 2018.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: 1907.01677 2019.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks. arXiv preprint arXiv:1804.0064 2018.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint 17Xiv:1703.05407, 2017.
Sainbayar Sukhbaatar, Emily Denton, Arthur Szlam, and Rob Fergus. Learning goal embeddings via self-play for hierarchical reinforcement learning. arXiv preprint 181 2018
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23-30. IEEE, 2017.
12

---

Under review as a conference paper at ICLR 2021
Mel Vecerik, Oleg Sushkov, David Barker, Thomas Rothörl, Todd Hester, and Jon Scholz. A prac- tical approach to insertion with variable socket position using deep reinforcement learning. In 2019 International Conference Robotics and Automation (ICRA), pp. 754-760. IEEE, 2019.
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through parametric discriminative rewards. arXiv preprint arXiv:1811.11359, 2018.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller, Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, pp. 2746-2754, 2015.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. arXiv preprint arXiv:2006.0m 001.0.001.1... 2020.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018.
13

---

Under review as a conference paper at ICLR 2021
A
ADDITIONAL EXPERIMENTS
A.1
COMPARISON WITH DIAYN AND ITS VARIANT
In this section, we expect to clarify the difference and connection with DIAYN (Eysenbach et al. 2018) experimentally, and indicate the limitations of maximizing I(s;w) and maximizing [(s; separately on learning goal-conditioned policy.
(a) DIAYN-Imitator: (();w). (b) DIAYN-Goal: [(s:g).
(c) GPIM: Z(s;w) + +((;g)
(b) DIAYN-Goal. and (c) our proposed GPIM. In (a), (b) and (c-left), the open circle denotes the user-specified goal, and the line with a solid circle the end is the corresponding behavior. Specifically, the white lines in (a) the learned skills in training, where we show 10 diverse skills. We can see that neither DIAYN-Imitator nor DIAYN- Goal approaches the corresponding goals, while our model succeeds in reaching the goal as shown in (c-left). In (c-right), we also show the generative factor by disentangling goals in (c-left), where the open circles with different colors (c-right) correspond the goals in (c-left).
Figure 10: 2D navigation with (a)
In DIAYN, authors show the ability of the model to imitate an expert. Given the goal, DIAYN uses the learned discriminator estimate which skill was most likely have generated the goal g:
arg max (w]gg)
Here we call this model DIAYN-Imitator We also directly substitute the perceptually-speeine goals for the latent variable in DIAYN's objective to learn goal-conditioned policy. We call this model DIAYN-Goal:
max I(s; g),
(10)
where g is sampled from the prior goal distribution p(g). Please note that do not adopt the prior non-parametric distance as in DISCERN (Warde-Farley et al. 2018) to calculate the reward. We
obtain the reward as in normal DIAYN using the variational inference: Tr (o 184+1)
Fig. TO shows the comparison of our GPIM with DIAYN variants including DIAYN-Imitator and DIAYN-Godd where the 2D navigation task is considered. As observed, Imitattr can reach seen goals but not unseen goals in Fig. [0(a), because it cannot effectively accomplish the interpo- lation between skills that are induced training. And behaviors generated by DIAYN- cannot guarantee consistency with the preset goals in Fig. [10b]] The main reason is that this objective states generated by y (or w) are different. However, there is no guarantee that (or w) and the state generated by the current y (or w) have semantically consistent behavior information. Our proposed GPIM method, capable of solving interpolation and consistency issues, exhibits the best performance in this 2D navigation task.
only ensures that when g (or w) is different,
Moreover, when the user-specified goals are heterogeneous to the states, the learned discriminator Is in DIAYN is unable to estimate which skill is capable of inducing given goals. Specifically, when the goals are visual inputs, and the states in training is feature vectors (e.g.. joint angles), the learned discriminator is unable to choose the skills due to lack of models for converting high-dimensional figure into low-dimensional feature vectors. On the contrary, there are lots of off-the-shelf models render low-dimensional feature vectors into perceptually-spenet high-dimensional inputs (Tobin et al. 2017 Khirodkar et al. 2018).
A.2 AUTOMATED GOAI-GGENERATION FOR EXPLORATION
In general, for unsupervised RL, we would like to ask the agent to carry out autonomous "practice" during training phase, where do not know which particular goals will be provided in test phase.
14

---

Under review as a conference paper at ICLR 2021
(a) Decoded latent goals (RIG).
1.00
Normaoeed Deoeeee to
0.25
GPM
DISCERM
0.00
Stops
(b) Goals (DISCERN).
(c) Goals (GPIM).
(d) Performance.
Figure 11: Distribution of sampled goals for the goal-conditioned policy, where the initial state is [5,2] (the red star) and the actor step is for each rollout. (a) The goals (images) are obtained by decoding 30 sampled latent goals 2g in VAE framework; (b) The goals (colored dots) are sampled from the agent's behaviors by random exploration: (c) The goals (colored dots) are rendered from the states induced by abstract-level policy. (d): Evaluation reaching user-specified goals, where GPIM significantly outperforms baselines.
In order to maximize the state coverage, the ability to automatically explore the environment and discover diverse goals crucial. this section, we will further analyze the goal distribution of three methods (RIG (Nair et al. 2018), DISCERN (Warde-Farley et al., 2018 GPIM) in new 2D obstacle navigation task as shown in Fig. II(a) Fig. [i(b] and Fig. II(c) The size of the environment is 10 x 10, the initial state is set [5, 2]. and there are two obstacles that prevent the agent from passing through each of which 3 x 6 size.
DISCERN samples goals during training by maintaining fixed sized buffer G of past observations. We simply mimic the process goal generation by taking random actions for 20 environment steps after initialization of the method. As in Fig. [Tbb we generate 100 goals with different colors. We can see that the majority of goals locate between the two obstacles, which limits the further exploration of the environment.
RIG samples representation (latent goals 2n)) from the learned VAE prior, which represents distribution over latent goals and state observation. The policy network takes the representation as substitute for the user-specified goal. For a clear visualization of the distribution of the sampled latent goals, we further feed the sampled latent goals into the decoder to obtain the real goals in the ussr-specified space. The decoded latent goals are shown in Fig. [1(a) where sample 30 goals. It shown that the majority of goals are also between the two obstacles because goals for training the VAE prior come from the same distribution as in DISCERN.
Our method, GPIM, obtains goals from the behaviors induced by the abstract-level policy. Maximiz- ing I(s;w) w) encourages different skills induce different states that further rendered goals. This objective ensures that each skill individually is distinct and the skills collectively explore large parts of the state space (Eysenbach et al. 2018). As shown in Fig. TI(c) our method provides better coverage of the state space than DISCERN and RIG.
Fig. I[dd shows the performance of the three methods, where randomly sample goals from the whole state (or goal) space at test phase. We can see that our method significantly outperforms the baselines. The most common failure mode for prior methods is that the goal distribution col-
15

---

Under review as a conference paper at ICLR 2021
lapses (Pong al. 2019), causing that the agent reach only fraction of the state space, as shown in Fig. (I and [1(b)
Exploration is a well-studied problem in the field of RL, and there many proven approaches with different benefits to improve the exploration (Colas et al. 2018 Campos et al. 2020). Note that these benefits orthogonal to those provided by our straightforward GPIM, and these approaches could be combined with GPIM for even greater effect. We leave combing our method with sophisticated exploration strategies to future work.
A.3 ABLATION STUDY
Here conduct the ablation study on 2D navi- gation task to analyze how the disentanglement of goals in GPIM affects the learned behaviors in terms of the generalization to new tasks?. For convenience,
2-D Navigation (x-y goal)
GPIM w/o KL w/o Pol.
remove certain component from GPIM and define the new method as fol- lows: w/o Rec removing the reconstruction loss (i.e. = 0); w/o KL removing the KL loss (i.e. 0); w/o Pol removing the policy loss updating 0g shown in Fig.
0.1M 0.3M 0.5M 0.7M 0.9M 1.1M 1.3M 1.5M
0.25 0.50 0.75 Normalized distance goals
Goals
w/o KL:
GPIM:
Figure 12: Ablation study and disentanglement.
The performance is reported in Fig. 12 observed that w/o. Pol performs worse than all other methods, which is consistent with the performance of RIG that trains VAE and policy separately. The main reason that the disentanglement fails to figure out the required latent factors on given tasks. Moreover, although GPIM has similar performance with the other three methods on 2D navigation task, GPIM has better interpretability to behaviors. considering series of goals from the first red to the last green (left) in counterclockwise order, GPIM can successfully disentangle them and learn effective latent (right), but w/o KL fails to keep the original order of goals (middle).
shown the bottom of Fig. [2
HalfCheetah
Swimmer
GPIM w/o Rec. w/o KL.
w/o Pol.
0.4M 1.2M 2M 2.8M 3.6M 4.4M 5.2M 0.4M 1.2M
2.8M 3.6M Steps
4.4M 5.2M
6M
Fetch
GPIM
Normalized Distance to Goal
w/o Rec.
w/o KL.
w/o Pol.
0.2
0.3
0.5
0.7
0.4M 1.2M 2M 2.8M 3.6M
4.4M 5.2M
6M
Steps
Figure 13: Ablation study on mujoco tasks.
Ablation study on mujoco tasks. We further study the impact of disentanglement part in our framework. The performance reported in Fig. [3 It observed that w/o Pol performs worse than all other methods, which is consistent with the performance in 2D navigation task And we can find that when we remove the reconstruction loss (a 0), the performance of w/o Rec degrades in these three environments. The main reason is that the process of learning generative factors become more difficult without supervised reconstruction loss. While in 2D navigation task, the reconstruction loss has little impact on the performance. Even though that w/o KL has similar performance with our full GPIM method, GPIM demonstrates better interpretability to behaviors as shown in Fig. T0(c).
Thhe results of 2D navigation task shown in the full paper.
16

---

Under review as a conference paper at ICLR 2021
A.4 GENERALIZATION THE GRIDWORLD TASK
Here we introduce an illustrative example of gridworld task and then show the generalization when the dynamics and goal conditions are missing in the gridworld task.
Illustrative example. We start with a simple gridworld example: the environment is shown on the left of Fig. [4]] where the goal for the agent is to navigate from the middle the given colored room. By abstract-level training, our method quickly acquire four skills to reach dif- ferent rooms. Each time the agent arrives in room induced by Ty. we train He conditioned on the room's 's color (e.g., green), allowing the agent to he guided to the same room by the cur- rent reward function. The learning process is shown the right of Fig. 14 It is concluded that the agent can automatically learn how to complete tasks given semantic goals in an unsupervised manner.
Abstract-level policy
Traning process
&
1000 v 0100
0010
0001
Goal-conditioned policy
goa goal-green"
"orange" 'magenta" 'blue'
Figure 14: Gridworld tasks. Y-axis is the percent- age of different rooms that the robot arrives in.
Training process
Abstract-level policy
0
1000
0010 Goal-condd policy
0100
0001
1
00
01
10
11
goal=
'blue'110
'green'/00
'rrange/011
'magenta'/11
Figure 15: Gridworld tasks. The Y-axis is the percentage of different rooms that agent arrives in.
Generalization on the gridworld task. We further show the generalization when the dynamics and goal conditions are missing in the gridworld task, where there are four rooms in different colors: blue, green, orange and magenta. We consider the situation where we train the goal-conditioned policy in the blue room, the green room and the orange room, and test in the magenta room.
As shown Fig. 15 we quickly acquire four skills for different rooms through the training at the abstract level. After the agent reaches at the colored rooms, we rendered the corresponding room as a two-bit representation: Render( 'blue") = 10, Render( green') ") = 00, and Render( 'orangee) = 01 do not render the magenta room. We take the magenta room corresponding to 11 the test goal to verify the generalization ability.
In lower part of Fig. 15| show the learning process of the goal-conditioned policy. We can find that the blue room tasks are learned quickly, and the green and magenta room tasks are learned relatively slowly, but the agent is still able complete the test task successfully (the magenta room). Compared to the task in Fig. 14 that renders all the rooms as goals, the whole learning process in Fig. [5] much slower. We hypothesize that the main reason is that the agent needs further infer the relationship between different goals. The lack of goal information (i.c., missing magenta 11) leads to a lower efficiency and unstable training.
B IMPLEMENTATION DETAILS
B. DERIVATION OF THE VARIATION BOUND
E [logp(wla) s)) + log p(w(3) - 2 log
E Em. [log (w)) 1) + log go (w]s) - 2logp(w))
+KI(p(w|s)] (w]s)) + [[(p(w]3]]loo[[133] Eass. [log (w))) (o (log((w))
17

---

Under review as a conference paper at ICLR 2021
B.2 ENVIRONMENT DETAILS
We introduce the details of environments and tasks here, including the environment setting of 2D navigation (x-y goal and color-shape goal), object manipulation, three atari games (seaquest, berzerk and montezuma revenge), and the mujoco tasks (swimmer, half cheetah and fetch).
2D navigation tasks: In 2D navigation tasks, the agent moves each of the four cardinal directions, where the states denote the 2D location of the agent. We consider the following two tasks: moving the agent to a specific coordinate named x-y goal and moving the agent to a specific object with certain color and shape named shlor-shape goal.
2D Navigation (x-y goal): The size of the environment is x 10 (continuous state space) or 7 x 7 (discrete state space). The state is the location of the agent, and the goal is the location the final location.
2D Navigation (color-shape goal): The size of the environment is 10 x 10. The state consists of the locations of the agent and three objects with different color-shape pairs (one real target and two distractors). The goal is described by the color and shape of the real target, encoded with one-hot.
More complex manipulation considers moving agent in 2D environment with one block for manipulation, and the other as a distractor. The agent first needs to reach the block and then move the block to the given location, where the block is described using color and shape. The size of the environment is 10 x 10. The state consists of the locations of the agent and two blocks with different color-shape pairs (one real target and one distractor). The goal consists of the one-hot encoding of the color-shape of the target block that needs to be moved, and the 2D coordinate of the final location of the movement.
Object manipulation:
Table 1: The repetition length of the action. Environments
k
Seaquest-ram-v0 Berzerk-ram-vo MontezumaRReveng-rran.
2, 4,
34, 36, 38, 40
2, 3, 4,
Atari games: We test the performance on three atari games: seaquest, berzerk, and montezuma re- venge. In order reduce the difficulty of training, we adopt the RAM-environment (i.e.. Seaquest- ram-v0, Berzerk-ram-vo, and MontezumaRevenge-n (eneruummaeeengg---m-O)))) where each state represents 128- dimensional vector, Each action repeatedly performs for duration of frames, where is uniformly sampled from Table
We consider to make diverse agents to fast imitate a given goal trajectory, including the imitation of behaviors of a swimmer, half cheetah, and a fetch, where the states in the trajectory denote the positions agents. Such experiments are conducted to demonstrate the effectiveness of proposed method in learning behaviors over continuous high nigh-dimensional action space, which is more complicated in physics than the 2D navigation.
Mujoco tasks:
Note that the goals in all the experiments are images 50 x 50 x 3 (3 channels, RGB) in size, except that the color-shape goal encoded with one-hot.
B.3 METRICS, NETWORK ARCHITECTURES AND HYPERPARAMETERS
Here give a clear definition of our evaluation metric "normalized distance to goal":
When the goal is to reach the final state of the trajectory induced by TT (Figs. 5(a) ((b) and 5(c) the distance to goal is the L2-distance between the final state 36 induced by πe and the goal state g randomly sampled from the goal (task) space:
N
Dis
L2(3
where is the number of testing samples. We set N = 50 for 2D navigation, object manipulation and atari games (seaquest, berzerk and montezuma revenge).
18

---

Under review as a conference paper at ICLR 2021
I: When the goal is to imitate the whole trajectory induced by TH (Figs.
5(d)
5(e) and 5(t)
the distance is the expectation of distance over the whole trajectory { 3f. ..... 57 Th y { ......................... and goal trajectory tgt oi - randomly sampled from the trajec-
induced by
tory (task) space:
1
Dis
Σ L2 gt)
where is the number of testing samples. We 50 for mujoco tasks (swimmer, half cheetah and fetch).
The term "normalized" means that the distance is divided by a scale factor.
Note that, for three atari games (seaquest, berzerk, and montezuma revenge), the L2-distance for evaluation is the difference between the position of controllable agent and the target's position, where the position is obtained by matching the pixel on the imaged state. See the Python code below.
def obtainxy _ea_uuue (img):
temp np. where (img 2, (ingl: 1 1, 1-11==177 (ingl , -=33)
global seaquestxy
1en (temp (temp101)==0) =0
temp = seaquestxy
else:
seaquestxy temp
xy np.array [np. mean (temp[(]] in. (temp 1) [1])1)/100 /100
return xy
def obddinnyy.ootceuua_eeeevee (img):
img [200]
temp np. where (img =00 & (imgo [:, ;,;1]]==722 (ing [;, :, :, 2-2==2222 ==72)
global mentezumaxy
if len (temp [0]]
temp montezumaxy
else:
montezumaxy = temp
= np. array [np. mean (temp (0)), mean (temp [11) /100 /1/00
return xy
def obtainxy wew.erk (img):
temp ar.arra (img)
temp temp[1:-1, 1:-1] temp [:-2, :-2] (21/2:1/3ttmmmp (1:1:ttmemp1:111111111:1
temp np. where ( (tempp 1, 1,0] ====040) & (temp [:, 1]==100)> (temp (1, : [,2]==) 2===103 )
global berzerkxy
if len (temp) [0] ==;
temp berzerkxy
else:
berzerkxy - temp
xy array ip mean (temp10) np. mean (temp (1111111///1 return xy
In implementation. use two independent SAC architectures (Haarnoja et al. 2018 for abstract-level policy тn and goal-conditioned policy πe. We find empirically that having two net- works share portion of the network structure will degrade the experimental performance. We adopt universal value function approximates (UVFAs) (Schaul et al.) 2015) for extra input (goals). For the abstract-level policy T,. pass latent variable to the Q function, value function, and policy, as in DIAYN, we simply concatenate w with the current state 8, (and action a.). For goal-conditioned
TThe reward function for baseline 1.2 Distance still calculates the I.2-distance directly the original space, instead of the distance of the agents' positions after pixel matching here.
19

---

Under review as a conference paper at ICLR 2021
policy иæ, also concatenate It with current state & (and action at). We update the dg using the gradients from both the Dis_loss and Q function's 's loss of the goal-conditioned policy ne-
The hyper-parameters presented in Table 2
C BROADER IMPACT
The main concern in the research area of goal-conditioned policy is how to find numerous diverse goals as well as obtain the reward function. Particularly, practical application. the goal and state often heterogeneous data with high variability of data types and formats, which further aggra- vates the difficulty of policy learning. Our model addresses these problems by allowing agents to in- teract with objects environment, and learn the behaviors in fully autonomous way the basis of intrinsic motivations, Bridging with the render function between abstract level and goal-conditioned policy, we could obtain diverse and versatile policies. By optimizing the intrinsic reward, our GPIM makes an agent capable of solving diverse tasks, which fits for wide-range applications such as quick imitation on behavior states, interactive navigation and object manipulation, and so on.
Moreover, our model provides approach to address the Robot general-purpose (REAL). This framework likely to speed up the progress of the general purpose robots that can achieve complex tasks given the corresponding goals, and drive the development of autonomous robot learning life-long learning mode.
Ended Autonomous Learning
By autonomous exploration of the environment, the agent is likely to generate some useful behaviors as well as the majority of useless skills. Of particular concern that the use of autonomous explo- ration is likely to generate a strategy that will induce dire consequences, such as collision skill in an autonomous driving environment. How to generate useful behaviors for user-specified tasks and how to use these induced skills also open problems. An alternative solution is reward to guide the exploration or the offline reinforcement learning (Levine et al. 2020). While another issue comes from the exploration-exploittiie to understand the limitations of REAL interacting with the environment autonomously. We would also encourage research understand the risks arising from autonomous robot learning.
extrinsic
trade-off. We would encourage further work
Another limitation is that our learning framework needs an extra abstract-level policy training for rendering goals and providing reward function. This requires twice much interaction time with the environment as learning a single policy network. For practical application, simulation platform preferred for pre-training. We also encourage researchers mitigate the difference between the simulation platform and the actual environment. We also pursue the effective models of transfer learning.
D MORE RESULTS
D.1 LEARNED BEHAVIORS ON TEMPORALLY-FXTENDER TASKS
More experimental results given in Fig. 16 to show the imitation several temporally-extent tasks.
D.2 LEARNED BEHAVIORS FROM GPIM
More experimental results are given in Fig. show the learned behaviors on 2D nav- igation, object manipulation. three atari games (seaquest, berzerk and montezuma revenge), and the mujoco tasks (swimmer, half cheetah and fetch). Videos are available under https//ssiesggoggee..ommeevgg
20

---

Under review as a conference paper at ICLR 2021
Table 2: Hyper-parameters
Slyper-paaamete
value
Batch Size
256
Discount Factor
0.99
Buffer Size
10000
Smooth coefficient
0.05
Temperature
0.2
X-y goal
0.001
2D Navigation
color-shape goal
0.001
Object Manipulation
0.001
Swimmer
0.0001
Learning Rate
Mujoco tasks
HalfCheetah
0.0001
Fetch
0.0001
Seaquest
0.0003
Atari games
Berzerk
0.0003
Montezuma Revenge
0.0003
x-y goal
20
2D Navigation
color-shape goal
20
Object Manipulation
20
Swimmer
50
Path Length
Mujoco tasks
HalfCheetah
50
Fetch
100
Seaquest
25
Atari games
Berzerk
25
Montezuma Revenge
25
x-y goal
128
2D Navigation
color-shape goal
128
Object Manipulation
128
Swimmer
256
Hidden Size
Mujoco tasks
HalfCheetah
256
Fetch
256
Seaquest
256
Atari games
Berzerk
256
Montezuma Revenge
256
x-y goal
2
2D Navigation
color-shape goal
2
Object Manipulation
4
Swimmer
3
Dimension of Generative Factor
Mujoco tasks
HalfCheetah
4
Fetch
4
Seaquest
16
Atari games
Berzerk
16
Montezuma Revenge
16
a
1
7
5
Spizel
255
21

---

Under review as a conference paper at ICLR 2021
Expert behavior
Expert behavior
Learned behavior
Learned behavior
Start
Start
Expert behavior
Learned behavior
Star
artt
tart
Start
Expert behavior
Learned behavior
Figure 16: Expert behaviors and learned behaviors. The four expert trajectories are described para-
metrically as: (Ir, y, =) = 1o + 1) + 1/50, sin(t)/5 1/5, 1/5), (x. y. 2) (1/5, cos (t) 1/5 sin,tin(t) (x. 2) = (cos(t)/5 + 1/50 1/1, 1/5, 1/5). and (z. v, = (sin(t) sin(2t)/2 /2, -t/5, cos(t)/2 22))
22

---

Under review as a conference paper at ICLR 2021
(a) navigation (discrete x-y goal; continous x-y goal; color-shape goal)
(b) Object manipulation
(c) Swimmer
(d) Half cheetah
FLFFFE
(c) Fetch
Seaquest
انا لن لنن
(g) Berzerk
(h) Montezuma revenge
Figure 17: Discovered goal-conditioned behaviors. (f-h): The left subfigure shows the expert be- haviors (goals); The middle subfigure shows the learned behaviors by GPIM; The right subfigure is the stacked view of goals and GPIM behaviors.
23


Reviews:
[{"review_id": "xhdFtfJ7IPK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThis paper proposes a new solution to the problem of learning goal-conditioned policies without hand-crafted rewards. Prior work in this domain learn an embedding space to compute reward between current state and goal. In contrast, this paper utilizes unsupervised skill discovery from [1] to obtain a discriminator that identifies which states belong to a particular skill. Then, the final state of a given skill's execution is used as a goal input to a goal-conditioned policy, which is rewarded if it generates states that the discriminator identifies with this skill. The paper aims to validate the benefit of such a reward over other embedding-distance based reward functions on a variety of environments.\n​\n​\n----------\n​\n# Strengths\n- The paper solves an important problem of unsupervised learning of goal-conditioned policies. To the best of my knowledge, the proposed way of training a goal-conditioned policy with skill-based discriminator is novel.\n- The training framework combines unsupervised exploration (skill-discovery) and learning of goal-conditioned policy elegantly through the discriminator. I find the insight very interesting that the learned skill's end-point can be treated as a target goal and the skill discriminator can be used as a reward to identify the states that are going towards the goal (since these states fall on the skill's induced trajectory).\n- The experiments are conducted on a large set of environments, which is good to justify the claims made by the paper.\n- I appreciate the video results and the visualizations provided on the project website.\n​\n​\n----------\n​\n# Weaknesses\n- The paper is missing the ablation study for the primary contributions. The paper argues that approaches taking reward as distance in embedding space can limit the repertoires of behaviors. One such example of unsupervised reward function is DISCERN [2]. To validate this point, and to make a fair comparison with DISCERN, the states used to train DISCERN's reward predictor should be generated from the same abstract-level policy (from [1]) as used by this paper. The current DISCERN baseline seems to be training the reward function on random exploration in the environment. I think a stronger version of this baseline is DIAYN+DISCERN.\n​\n- The section about disentanglement using information bottleneck seems orthogonal to the main contributions of the paper - about an unsupervised method to train goal-conditioned policies. While the information bottleneck can always be used to improve generalization, it is unclear why its use is emphasized. If I understand this correctly, then the ablation study in Section 5 can be moved to Appendix as it seems an extra detail. It is rather more important to ablate the role of the intrinsic reward function proposed in this paper (as discussed in the above point).\n​\n- The writing is understandable, but can be improved. It is not always clear what the main contributions of this paper are.\n(a) The \"abstract-level policy\" is precisely the unsupervised skill-discovery framework from [1], and it should be stated as such.\n(b) The paper emphasizes the view that the exploration and goal-conditioned policies share the same reward function, but a more relevant viewpoint is that the proposed training framework enables the discriminator learned for exploration to also be useful as a reward for the goal-conditioned policy.\n(c) Some parts of writing are incoherent and seem out-of-place at first. For instance, paragraph 3 of introduction should be written in a way that contrasts this paper's contributions to the prior work.\n(d) One clarification question I have is about where the goals for evaluation (such as in videos) come from? Are these goals just what were observed in training or are these manually-crafted goal-locations for evalution? It would be nice if a principled evaluation procedure is used, where a dataset of testing goals (maybe procedurally generated) are used to report the performance of the goal-conditioned policy.\n​\n​\n----------\n​\n# Reason for decision\nMy primary concern is the missing baseline where alternate unsupervised rewards are combined with better exploration methods (just like this paper uses [1]). In my understanding, the primary contribution is an alternate reward function, which does not depend on explicit embedding distances. To validate this contribution, this baselines is important. I would be happy to increase my score if this concern is addressed.\n​\n​\n----------\n​\n# Suggestions for improvement\n- It was a little difficult to understand the high-level intuition of the method. It can be more succinctly summarized in the text and/or Figure 1 caption, such as: \"Skill w is a way to reach the goal g, and the goal-conditioned policy with goal g is rewarded to imitate the trajectory induced by w.\"\n​\n​\n----------\n​\n# References\n[1] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.\n[2] David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through parametric discriminative rewards. arXiv preprint arXiv:1811.11359, 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "pCNcqICiHhO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes a novel method for learning goal-conditioned policy with images/text goals. \n\nQuality: The overall quality of the paper is good. \n    Strong side: Extensive evaluation on various environments and tasks showcases the advantage and generalizability of the method. The authors uses figures and algorithm boxes to make their method very clear.\n    Weak side: Several clarification on the motivation and details of the method is needed. See below.\n\nClarity: (1) The main argument is not strong: 1. Image/text goals is not intractable for current goal-conditioned policies [1] 2. What do you mean by ‘extrinsic’ reward? If you mean task-specific reward, many methods learn goal-conditioned policy without extrinsic reward, like hindsight experience replay.\n(2) Therefore, I guess the authors wanted to claim they use ‘intrinsic’ rewards, which is a mutual-information-based reward. Now I have two questions for Section 3.2: 1. Why do we use this loss function Eq. (1)? It comes out of nowhere without intuition. (2) Why do you decompose the optimization into ($\\mu, \\Phi$) and $\\theta$? I feel like decomposing into $\\Phi$ and ($\\mu, \\theta$) is more reasonable in that one optimizes the reward first then policies.\nAnother question on algorithm box: In step two, $g_t$ is changing with time steps. This is kind of strange because in typical goal-conditioned policy learning, in one epoch, people use a fixed goal to train their policy, then change the goal in the next epoch. What is the fundamental difference?\n(3) The whole disentanglement thing needs more clarification. I don’t understand the reason why you disentangle your policy. In computer vision, disentanglement has clear physical meanings like disentangling shape and color, but here I don’t have such intuition. In experiments, the effect of disentanglement is only demonstrated in a simple 2D-task, which seems not enough. \n(4) Details: 1. How is p(w) defined? It seems super important, but I was not able to find its details in the paper. Maybe I overlooked something.\n\nOriginality: As far as I know, the method is new.\n\nSignificance: Learning goal-conditioned policy with high-dimensional goals is an important problem. I think if the authors could clarify the above questions, the solid experiments will make the paper a good contribution. However, I don’t think its current version passes the bar of ICLR.\n\nReference: [1] https://papers.nips.cc/paper/9623-planning-with-goal-conditioned-policies.pdf", "extra_scores": {}, "extra_text": {}}, {"review_id": "htP0G5-KPo8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for combining intrinsic motivation on a state space with goal-conditioned reinforcement learning (GCRL), where goals are defined in some “perceptual space,” such as text or images, which describe the current state. The authors assume access to a renderer that maps states to perceptual goals, but do not assume that the renderer is differentiable. The authors propose to train an intrinsically motivated latent-conditioned policy, using similar techniques as past work in which a policy maximizes the mutual information between a latent variable and the current state. The goal-conditioned policy is then trained to effectively imitate the latent-conditioned policy by maximizing the same reward as the latent-conditioned policy, conditioned on only the rendered version of the final state reached by the latent-conditioned policy. The authors demonstrate that the overall method outperforms past GCRL methods on a variety of tasks (Atari, MuJoCo manipulation and locomotion, and toy tasks).\n\nThe idea of using intrinsic motivation rewards to train a GCRL policy is an interesting and novel idea, and the experiments indicate that this is a promising approach. The writing is a bit verbose and there are some details that rushed through, but overall I find the idea novel and the results compelling.\n\nI would recommend that the authors clarify that this method assumes access to a renderer. Currently, the writing does not make it clear that the comparison to RIG and DISCERN is not an apples-to-apples comparison: RIG and DISCERN do not assume access to the ground-truth state, but rather operate directly in the perceptual space. However, I think this comparison is reasonable since I do not know of any other method that makes the same assumption as this paper.\n\nThe authors state that, “the goals for πθ in mujoco tasks are the rendered trajectories induced by π...” In that case, how did the authors condition a policy on entire trajectories? This seems like an important detail, and it’s also unclear how the baselines could have been implemented in those cases.\n\nThe analysis of Figure 6 experiment seems to overstate the result. While it is true that, “during training the agent is required to imitate a large number of simple behaviors” it does not seem to be the case that it “has never seen such complex goals.” Unless I am mistaken, the policy at test time is not conditioned on a single complex goal at test time. Instead it’s simply conditioned sequentially on simple tasks, which is effectively the same as during training but just done sequentially.\n\n--- Post Rebuttal ---\n\nI've read the author response and do not intend to increase my score. Thank you for answering my questions. It would be good to clarify in the main paper that conditioning on the trajectory is implemented in the way described in the author response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DjT0Q-VIoet", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n--\nThis paper proposes an unsupervised learning objective for learning perceptual goal-conditioned policies. The goal is to enable unsupervised discovery of high-level behaviors in tandem with a perceptual-goal conditioned policy that can achieve these behaviors. The learning proceeds by training one policy to exhibit diverse behaviors; the states induced by these behaviors are then rendered and used as target goal states for a separate goal-conditioned policy.\n\nThe learning objective is to maximize a lower-bound on sum of two terms -- (1) the mutual information between the behavior variable and states with a behavior-conditioned policy; states from this behavior-conditioned policy are transformed into perceptual goals that serve as input to a separate policy that forms (2) the mutual information between perceptual goals and the states it induces.\n\nThe learning algorithm operates via alternating optimization, in which the skill-conditioned exploration policy is learned jointly with a skill 'discriminator' (inference network), and then the perceptual-goal conditioned policy is learned using the discriminator as a reward signal, which essentially estimates the extent to which a robot is achieving a skill given a perceptual goal along the skill-policy trajectory.\n\nA glut of experiments are used to investigate whether the method learns a meaningful skill reward function, whether it can achieve goals in various environments, how the method compares to related methods, and whether the specific 'disentanglement' inductive bias for constructing the policy is useful. The experiments demonstrate favorable performance over existing methods for these tasks.\n\nQuality\n--\nThe goal, method, and experiments are likely high quality, given my understanding. However, there are significant gaps in clarity that reduce my certainty in this assessment, and relatedly, there is some important missing discussion on the specific differences between the proposed method and prior work.\n\nClarity\n--\n- The learning algorithm is ambiguous: which goal(s), specifically, are used in the second learning stage? The current algorithm reused the 't' time index, which makes this unclear. I suspect it's just the goal corresponding to the last timestep from the first stage of the algorithm, but I'm not sure.\n- The state representation for the archery task is unclear\n- The 'fast imitation' procedure is unclear. The paper says the goals are the rendered states induced by the abstract policy, so how do the expert demonstrations get factored in? Is the learning algorithm different (is the abstract policy trained with the expert demonstrations somehow)? This is a significant ambiguity that makes it difficult to interpret the results of the imitation experiments. It's hard to guess at, because the conceptual difference between the standard imitation learning problem and a prototypical unsupervised \"RL\" algorithm is quite large.\n- The presence of two environments in Figure 1 is quite confusing. It's hard to tell from the context of the figure alone whether the policies are operating simultaneously in the same environment, simultaneously in separate equivalent environments, or at separate times in equivalent environments.\n- The \\tilde notation and relationship between the \\tilde an not-\\tilde variables needs to be discussed in 3.1\n\nOriginality\n--\nThe learning objective seems to be quite similar to the DIAYN and DISCERN objectives, and the task of learning to condition on general perceptual goal in unsupervised setting is shared by RIG and DISCERN. Discussion of specific algorithmic and assumption differences between the proposed method and these approaches is quite necessary, but unfortunately missing. \n\nSignificance\n--\nIn absence of knowing more about the specific differences between the proposed work and related works and the imitation learning experiments, the only thing that is clear is that the method seems relatively performant on an existing well-motivated task across a large range of settings, and compares favorably to existing methods on this task.\n\nOther points\n--\nDerivation is needed (e.g. in appendix) to show that (3) is a lower bound of the last line of (2). I don't think this aspect of the derivation follows from Jensen's inequality, as stated in the paper.\n- In the intro (para 3) \"an unsupervised method in RL\" is unclear to an RL reader unfamiliar with \"unsupervised RL\". I think this paper should explicitly define \"unsupervised RL\" to mean \"RL with an intrinsic reward function\" or something equivalent, and \"intrinsic reward function\" to mean a learning objective that can be applied in place of an alternative reward function across different MDPs.\n- Intro para 3 \"maximize an information theoretic objective\" is vague, because all learning objectives for probabilistic models are technically \"information theoretic\" -- all probabilistic models have a relationship to information.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work extends previous work on unsupervised learning of goal-conditioned policies: an abstract skill policy, which drives exploration of the state space, is used to propose goals as well as derive rewards for a goal conditioned policy. 

Reviewers agreed the approach was novel and interesting. All reviewers raised significant concerns about clarity and/or lack of details, as well as a lack of comparison to DIAYN/DISCERN, though these points were adequately addressed in revisions. One remaining issue raised by two reviewers are that the content related to the information bottleneck/disentangled representation learning seems out of place and ill-justified. Detailed discussion of this aspect of the work has been relegated to the appendix.

This is an important problem and a growing area of study, and while the submission has potential, improvements needed are not minor, and given the short process, we can only accept papers as is, rather than expecting certain changes. We urge the authors to further improve the focus of the work and perhaps plan to investigate the role and importance of disentanglement with IB in this setting in follow up work wherein they have the space to properly do justice to the topic in its own right.
Decision: Reject