Learning Subgoal Representations with Slow Dynamics
Authors: Siyuan Li
Lulu Zheng
Jianhao Wang
Chongjie Zhang
Venue: ICLR 2021 Poster (2021)
Keywords: Hierarchical Reinforcement Learning
Representation Learning
Exploration

Abstract: In goal-conditioned Hierarchical Reinforcement Learning (HRL), a high-level policy periodically sets subgoals for a low-level policy, and the low-level policy is trained to reach those subgoals. A proper subgoal representation function, which abstracts a state space to a latent subgoal space, is crucial for effective goal-conditioned HRL, since different low-level behaviors are induced by reaching subgoals in the compressed representation space. Observing that the high-level agent operates at an abstract temporal scale, we propose a slowness objective to effectively learn the subgoal representation (i.e., the high-level action space). We provide a theoretical grounding for the slowness objective. That is, selecting slow features as the subgoal space can achieve efficient hierarchical exploration. As a result of better exploration ability, our approach significantly outperforms state-of-the-art HRL and exploration methods on a number of benchmark continuous-control tasks. Thanks to the generality of the proposed subgoal representation learning method, empirical results also demonstrate that the learned representation and corresponding low-level policies can be transferred between distinct tasks.

Published as a conference paper at ICLR 2021
LEARNING SUBGOAL REPRESENTATIONS WITH SLOW DYNAMICS
Siyuan LiB Lulu Zheng; Jianhao Wang, Chongjie Zhang Institute for Interdisciplinary Information Sciences Tsinghua University, Beijing, China
fey-1i17,2111,, wjh1 @ma ils. ts chongjie@tsinghua. edu. en
ABSTRACT
In goal-conditioned Hierarchical Reinforcement Learning (HRL). a high-level policy periodically sets subgoals for low-level policy, and the low-level pol- icy is trained to reach those subgoals. A proper subgoal representation function, which abstracts state space to latent subgoal space, is crucial for effective goal-conditioned HRL, since different low-level behaviors are induced by reach- ing subgoals in the compressed representation space. Observing that the high-level agent operates at an abstract temporal scale, we propose a slowness objective to effectively learn the subgoal representation (i.e., the high-level action space). We provide a theoretical grounding for the slowness objective. That is, selecting slow features as the subgoal space can achieve efficient hicrarchical exploration. As a result of better exploration ability. our approach significantly outperforms state- of-the-art HRL and exploration methods on number of benchmark continuous- control tasks Thanks to the generality of the proposed subgoal representation learning method, empirical results also demonstrate that the learned representation and corresponding low-level policies can be transferred between distinct tasks.
INTRODUCTION
Deep Reinforcement Learning (RL) has demonstrated increasing capabilities in a wide range of do- mains, including playing games (Mnih al. 2015 Silver al. 2016), controlling robots (Schulman et al. 2015 Gu et al. 2017) and navigation in complex environments (Mirowski et al.) 2016 Zhu al. 2017). Solving temporally extended tasks with sparse or deceptive rewards one of the major challenges for RL. Hierarchical Reinforcement Learning (HRL), which enables control at multiple time scales via hierarchical structure, provides promising way solve those challenging tasks. Goal-conditioned methods have long been recognized as an effective paradigm in HRL (Dayan & Hinton 1993] (Schmidhuber Wahnsiedler, [ Nachum et al. 2019). In conl-conditioned HRL, higher-level policies set subgoals for lower-level ones periodically, and lower-level policies are in- centivized to reach these selected subgoals. A proper subgoal representation function, abstracting state space to latent subgoal space, is crucial for effective conl-conditioned HRL, because the abstract subgoal space, i... high-level action space, simplifies the high-level policy learning, and ex- plorative low-level behaviors can be induced by setting different subgoals in this compressed space as well.
Recent works in goal-conditioned HRL have been concentrated on implicitly learning the subgoal representation in an end-to-end manner with hierarchical policies (Vezhnevets et al.] [017]] 7 Dilok-
thanakul et al. 2019), e.g., using variational autoencoder (Péré al. 2018 Nair & Finn 2019
Nasiriany et al. 2019), directly utilizing the state space (Levy et al. 2019) or a handcrafted space
(Nachum al. 2018) as subgoal space. Sukhbaatar et al. (2018) proposed to learn subgoal em- beddings via self-play, and Ghosh et al. (2018) designed a representation learning objective using an actionable distance metric, but both of the methods need pretraining process. Near-Optimal
* Denotes equal contribution
Videos available a
Find open-source code at httpes///tthib.cco///yyaaeteeee

---

Published as a conference paper at ICLR 2021
Representation (NOR) for HRL (Nachum al. 2019) learns abstract space concurrently with hierarchical policies by bounding the sub-optimality. However, the NOR subgoal space could not support efficient exploration in challenging deceptive reward tasks.
In this paper, we develop a novel method, which LEarns the Subgoal representation with SIOw dyNamics (LESSON) along with the hierarchical policies. Subgoal representation in HRL is not only a state space abstraction, but also a form of high-level action abstraction. Since the high-level agent makes decisions namics from observations as the subgoal space to enable temporal coherence. LESSON minimizes feature changes between adjacent low-level timesteps, order for the learned feature representa- tion to have the slowness property. To capture dynamic features and prevent the collapse of the learned representation space, we also introduce an additional contrastive objective that maximizes feature changes between high-level temporal intervals. We provide a theoretical motivation for the slowness objective. That is, selecting slow features as the subgoal space can achieve the most effi cient hierarchical exploration when the subgoal space dimension is low and fixed. We illustrate on a didactic example that our method LESSON accomplishes the most efficient state coverage among all the compared subgoal representation functions. We also compare LESSON with state-of-the the- art HRL and exploration methods on complex MuJoCo tasks (Todorov et al. 2012). Experimental results demonstrate that (1) LESSON dramatically outperforms previous algorithms and learns hi- erarchical policies more efficiently; (2) our learned representation with slow dynamics can provide interpretability for the hierarchical policy; and (3) our subgoal representation and low-level policies can be transferred between different tasks.
low temporal resolution, our method extracts features with slow dy-
2 PRELIMINARIES
In reinforcement learning, an agent interacts with an environment modeled as an MDP M = (S, A, P, R. r), where S is a state space, A is action space. P S A x + [0, is an unknown dynamics model, which specifies the probability P(s'|s, of transitioning to next state s' from current state by taking action a. R S A - R is reward function, and 7 E [0, 1) is discount factor. We optimize a stochastic policy T(ass), which outputs distribution over the action space for a given state The objective to maximize the expected cumulative discounted reward E, D r're] under policy π.
3 METHOD
In this section, we present the proposed method for LEarning Subgoal representations with S1Ow dyNamics (LESSON). First, describe two-layered goal-conditioned HRL framework. We then introduce a core component of LESSON, the slowness objective for learning the subgoal represen- tation of HRL. Finally, summarize the whole learning procedure.
3.1 FRAMEWORK
Following previous work (Nachum et al. 2018 2019), we model policy <[ass) as two-level hierarchical policy composed of a high-level policy TH (gls i)s and low-level policy g). The high-level policy Tr (g]s) selects a subgoal g in state every e timesteps. The subgoal g is in low dimensional space abstracted by representation function Φ(s) : S + R2* The low-level policy (r(als.g) g) takes the high-level action g as input and interacts with the environment every timestep. Figure depicts the execution process of the hierarchical policy.
LESSON iteratively learns the subgoal representation function d(s) with the hierarchical policy. To encourage policy स, to reach the subgoal g. we train nı with an intrinsic reward function based on the negative Euclidean distance in the latent space, ri(St, at St+1. g) -lló(s++1) gll2. Policy Tr is trained to optimize the expected extrinsic rewards rf We use the off-policy algorithm SAC (Haarooj al. 2018) standard RL algorithm.
base RL optimizer. In fact, our framework is compatible with any
Apparently, proper subgoal representation d(s) is critical not only for learning an effective low- level goal-conditioned policy but also for efficiently learning an optimal high-level policy to solve a given task. As the feature dimension l is low, d6s) has compression property, which is necessary
2

---

Published as a conference paper at ICLR 2021
80
90 Ta (sp)
So
Sc
(-
r
r
(sx))
ro 2i=0 re
(sa)
(s,)
a
(5
High-level steps in the state space
((52)
distance
Low-level steps in the latent space (ss)
A schematic illustration of the hierarchical policy exccution. One high-level step corre- sponds to e low-level steps. The negative Euclidean distance the latent space provides rewards for the low-level policy.
Figure 1:
to make the hierarchical policy learning easier. If ((s) is exactly an identity function without any abstraction, the high-level policy иy still needs to explore in large space and the complicated subgoal for the low-level policy is hard to reach as well. In this circumstance, the hierarchical structure cannot simplify the MDP and has no advantage over flat structure.
3.2 LEARNING SUBGOAL REPRESENTATIONS
Inspired by physics-based priors, features with slow dynamics preserve higher temporal coherence and less noise (Wiskott & Sejnowski] 2002). As the high-level policy acts at a lower temporal reso- lution compared the low-level policy, it is sensible to learn subgoal representation function with a slowness objective. To solve large-scale problems, we parameterize the representation function ((s) with neural network to extract slow features. One natural way of learning ((s) is minimize the squared difference between feature values at times and t + 1,
min (((u))))==(ld(
((s++) 112),
where D is a replay buffer. This loss function eliminates fast features, but can be trivially optimized we allow lossy representation function & (e-g., if d(s) for V's E S). To avoid such trivial solutions and capture dynamic features, we propose contrastive loss to maximize the distance between high-level state transitions in the latent subgoal space, i.e. min, .=p==))))_--
adopt the technique of triplet loss (Chopra et al. 2003). i.e., imposing the latent distance between high-level transitions larger than margin parameter m, as shown by Eq. 2
((s++))]l2] To trade off these two loss functions,
we remove the margin parameter m and the mar operator, Eq will be dominated by the maximizing distance part. Margin m defines unit of distance in the latent space, which prevents trivial solutions as well.
min E (e [I]((s.) d(s + max m (1o(s.) - ((s++))l2)]
(2)
The above learning objective abstracts the state space to latent subgoal space with slow dynamics. As Eq. optimizes the squared difference between feature values, the learned representation can preserve the spatial locality property of the state space, so a subgoal g can selected in the neigh- borhood of (s)). In the next section, we give theoretical motivation for the slowness objective.
That is, selecting slow features as the subgoal space can promote efficient exploration. Algorithm I shows the learning procedure of our method. We update d(s) and æi at the same frequency so that the low-level reward function varies in a stationary way. The high-level policy updated less frequently, the high-level transitions are less.
4 EFFICIENT EXPLORATION WITH SLOW SUBGOAL REPRESENTATION
In this section, we provide theoretical motivation for subgoal representation learning with slow dynamics from statistical view. To support formal analysis, we consider selecting subset of features from the state space as a subgoal space. We prove that, given a fixed subgoal space dimension, selecting slow features the subgoal space can achieve the most efficient hierarchical exploration. We first define a measure for exploration and describe assumptions of our analysis. Then, present theorem about the optimality property and corresponding implications.
3

---

Published as a conference paper at ICLR 2021
Algorithm 1 LESSON algorithm
Input: Number training steps N, margin m. replay bufler D.
2: Initialize: Learnable parameters for na (g|s), m g) and o(s).
3: for t = 1..N do
4: Collect experience (st. St. az, $t+1, m under ny and Ti.
5: Compute low-level reward r -1((((++1)) - gel12.
6: Update the replay buffer D.
7: Optimize тH by maximizing cumulative task rewards with every e timesteps.
8: Optimize by maximizing cumulative low-level rewards with D every timestep.
9: Sample a batch of state transitions from and update & with Eq. 2 every timestep.
10: end for
11: Return: The Ty and Ø.
4.1 DEFINITIONS AND ASSUMPTIONS
To develop a theoretical analysis, we give a definition of slow features and measure of exploration. Then, we formulate the exploration process in goal-conditioned HRL as a random walk in the state space follows.
As our theoretical analysis is broadly applicable to arbitrary feature space, we denote a state S; = [s],...,s'f s/1" as vector containing / feature State S, can be factored into slow features Salow and fast features Sfast with a one-step feature change metric As; = |s s il (1 < S I). Without < E., [Ast+J, where т., is random policy. The
loss of generality, we assume that Ex, expected one-step feature change of slow features is relatively small. With limited slow feature dimension k Salow = [s-_ ...., s $ and the rest are fast features. For example, the movements of robot are slow, but the changes of noisy sensory observations are fast.
Definition 1 (Measure of Exploration). In erarchical exploration defined as the explored states q(x) to desired state distribution p(æ):
HRL, an effectiveness measure of hi- (KL) divergence from the distribution of
(((x)
(PL(pl1)) p(x) log
dr.
(3)
g(x)
In this definition, the desired state distribution p(x) is prior state distribution while g(x) is the distribution of the states explored by the agent. According to Definition when the state distribution of exploration ((z) is closer to the target state distribution p(x), the exploration is more effective. Definition 2 (Random Walk). In goal-conditioned HRL, the exploration process in the state space is an I T-dimensional random walk when there is no extrinsic reward for the high-level policy and the low-level policy is optimal. Define 8o as the origin of the state space: 8g = 0, and the unit step of the random walk is Xi 8t
St-e, I 2c, which is i.i.d. Denote sequence of random
variables Y, = - - then the asymptotic distribution of Y, is q(z): Y, B q(x).
We aim to solve sparse reward problems, where an agent needs to explore with little extrinsic wards, so we consider the circumstance with no extrinsic rewards and the optimal low-level policy to analyze the exploration problem. Thus the high-level policy selects subgoals randomly. The agent can move independently and identically the state space, leading to X; i.i.d. In fact, g(x) can be seen the steady state distribution of the Markov chain induced by the policy. To facilitate the analysis of different subgoal representations, we make the following assumptions throughout this section:
(a) The transition function P(s'|s, |s, a) is deterministic.
(b) The features are all independent.
(c) X is bounded in the state space: {|ra] S ri, = 1,..., , 7). where is the i-th element of Xú, and r; a fixed upper bound of [.s]]-
(d) The subgoal g selected by the high-level policy at time / is constrained in the neighbour-
hood of 8c: |اo =31 S T = 1,.... ))) where 9j and of are the j-th elements of corresponding vectors, and rg a fixed bound of subgoals in all dimensions.
'The here refers to Markovian
4

---

Published as a conference paper at ICLR 2021
Assumption (a) is a general technique to simplify theoretical analysis in RL (Kishhamurthy al. 2016 Boyan & Moore 1995). Assumption (b) makes it possible to analyze the exploration of each feature dimension separately. Assumption (c) means that every timesteps, the agent can move in dimension i with a step size |a:]| S ri. and slower features have a smaller bound: Vi < f',r; r; < ry- Taking advantage of the spatial continuity of the state space, subgoals are set in the neighhorhood of the current state in the selected subgoal feature dimensions, specified as Assumption (d).
4.2 OPTIMALITY AND IMPLICATIONS
Theorem 1. Assume p(æ) ) is multivariate Gaussian distribution: p(x) ~ (ax; O, R), where R is diagonal matrix diag(r") and is large enough. Given fixed subgoal space dimension k, selecting the slowest features for the subgoal space leads to the optimal hierarchical exploration. Denote the distribution of the explored states in this case as Islow, have:
yslow = q" = arg minDKL(Plla))
(4)
qEQ
where Q is the sets of all distributions of explored states brought by different subgoal space selection.
Without any prior knowledge, we assume p(x) is an isotropic Gaussian distribution with zero mean. When is large enough, g(x) approximates a uniform distribution.
Proof sketch. The exploration process is decided by the coverage area of the random walk, as shown in Definition | and a larger coverage area leads to better exploration (see Definition IT. We analyze the coverage scale in each dimension separately. The exploration ability varies in different dimen- sions since the slow-fcature dimension has smaller coverage scale. Notice that the exploration ability in dimension i changes if we select the i-th feature for the subgoal space. Concretely, if choose slow features for the subgoal space, the coverage area in these dimensions will expand. In contrast, selecting fast features decreases the ability of exploration. We prove that with a fixed subgoal space dimension k. if and only if we select the k slowest features as the subgoal space, (rll(lllg) is minimized, i.e., achieves the optimal hierarchical exploration defined in Definition 1 See the detailed proof in Appendix A
Selecting slow features as the subgoal space can achieve superior exploration shown in Theorem This property indicates that using the slowness objective to learn the subgoal representation can promote more efficient exploration. As real-world tasks are often on large scale, utilizing neural networks to extract slow features as the subgoal space more general. To conclude, Theorem i theoretical grounding for the learning objective of LESSON.
!
[
5 RELATED WORK
Learning subgoal representations is challenging problem in HRL (Dwiel et al. 2019). Nachum jet al. (2018) and Zhang et al. (2020) predefined a subspace of observations as a subgoal space with domain knowledge. Li et al. (2019) sought for an alternative way of setting advantage-based auxiliary rewards to the low level policy to avoid this difficult problem. Levy et al. (2019) directly used the whole observation space, which is unscalable to high-dimensional tasks. A variational autoencoder (VAE) (Kingma & Welling, 2013) can compress the high-dimensional observations in an unsupervised way, and it has been utilized to learn subgoal space in (Péré et al. 2018 Nair
& Finn 2019 Nasiriany et al. 2019). However, the features extracted by VAE can hardly capture the transitional relationship in MDPs. In Vezzneveet et al. (017) and Dilokthanakul et al. (2019). subgoal representation is learned in an end-to-end way with hierarchical policies. Since the resu Ting representation is under-defined, those methods often underperformed (see Nachum al. (2018) Ghosh et al. (2018) proposed to learn representations using an actionable distance metric, assuming that goal- coal-conditioned policies are given. [Sukhbattar et al. (2018) developed a method called HSP to learn subgoal representations via self-play, but HSP requires pretraining process, and thus it may be inefficient. Near-Optimal Representation (NOR) for HRL (Nachum et al. 2019) outperforms the previous methods by learning representations bounding the sub-optimality of hierarchical policies. However, the optimization of NOR is complicated, and the abstraction of the NOR space does not aim for efficient exploration. In contrast, develop simple subgoal space learning method with a slowness objective. Furthermore, we formally show that the slowness objective has a theoretical grounding for better exploration ability.
5

---

Published as a conference paper at ICLR 2021
Slowness or temporal coherence has been important prior for learning state representations in continuous control tasks (Bengio et al.
2013 Jonschkowski & Brock, 2015 Lesort et al. 2018
Standard Slow Feature Analysis (SFA)
methods learn slow features by solving optimization
problem with constraints (Wiskott 1999 Wiskott & Sejnowski 2002). However, their expressivity
tends to scale unfavorably in sigh--iinnnional problems. To increase the expressivity, hierarchical
SFA (Franzius et al. 2007; 2011 Escallnte-B & Wiskott, 2013) composes multiple SFA modules in layer-wise way. More recent works use neural networks extract slow features using slow- ness loss function. To avoid trivial solutions, another term, such reconstruction loss (Goroshin al., 2015a| Finn al., 2016) or prediction error (Goroshin et al. 2015b), is also included in the loss function. In similarity metric learning, contrastive or triplet loss is investigated to capture slow features in video and audio datasets as well (Jayaraman Grauman 2016 Jansen et al. 2018). In reinforcement learning, several approaches exploit the slowly changing bias to extract useful fea- tures so that policy learning can be accelerated (Zhang et al.] 2009, Legenstein et al. 2010 Oord et al. 2018). To the best of our knowledge, are the first utilize the slowness objective in HRL, and our proposed method significantly outperforms state-of-the-arr HRL. methods on benchmark environments.
The inductive bias of slowness has largely been investigated in the skill discovery methods as well. Continual Curiosity driven Skill Acquisition (CCSA) learns latent space with SFA, and utilizes curiosity-driven rewards in this latent space to train skills (Kompella et al. 2017). Similarly, Machado al. (2017a). Jinnai al. (2019) and Bar et al. (2020) proposed learn options reach local maxima or minima of the Proto-value functions (PVFs) (Mahadevan & Maggioni 2007). As
pointed out by Sprekeler (20TT), the objective functions of SFA and PVFs are equivalent, when the adjacent function of PVFs is the transition function in MDP. But obtaining the full transition func- tion in large scale tasks is nearly infeasible. To solve large scale problems, Machado et al.) (2017b) and |aaeesh et al. (2019) proposed to replace PVFs with eigenvectors of the deep successor rep- resentation (Kulkarni et al., 2016), which equal to scaled PVFs. Jiinnii et al. (2020) approximated the computation of PVFs with the objective introduced by Wu et al. (2018). Our method and those skill discovery methods share some similarities in learning low-level policies a smooth or slow latent space. However, the skill discovery methods be regarded bottom-up HRL, where set of task-agnostic low-level skills are firstly learned with some intrinsic reward functions and then composed to solve downstream tasks. In contrast, our goal-conditioned method can be regarded as top-down HRL, where the high-level policy sets subgoals to the low level during learning a task, and the level-level policy is incentivized to reach those subgoals.
EXPERIMENTS
Start
COLOC
We conduct experiments to compare our approach to existing state-of-the-aa methods in HRL and in efficient exploration. Firstly, we show on a didactic example that LESSON most efficient state coverage among all the com- pared subgoal representations. To demonstrate our strengths in high-dimensional tasks, we then com- pare with several baselines on number of bench- mark continuous-contro tasks. After that, we an- alyze the dynamic property of the learned subgoal representation and provide an interpretation by vi- sualization. Lastly, we show that both the subgoal representation and the low-level policies learned by our method are transferable.
N-2
N-1
achieve the
Explored States (%6
0.6
0.4
Ours
NOR
Fast
60
80
Episodes (b)
6.1 DIDACTIC EXAMPLE: NCHAIN
(a) The NChain environment. (b) Re- sults on the 64-link chain environment. Each line is the mean of 20 runs with shaded regions corresponding confidential intervals of 95%.
Figure 2:
The NChain environment was designed hard to ex- plore by Osband et al. (2016), shown in Figure 2(a). Starting from state 0, an agent can move for- ward (blue arrow) to the next state in the chain

---

Published as a conference paper at ICLR 2021
backward (green arrow) to the previous state. The state representation is encoded in binary, so the low bits are features with fast dynamics. To make the problem harder, the effect of each action is randomly swapped with probability of 0.1. In this determeerriiiittic environment, the exploration ability of our method to HRI. methods using other subgoal spaces with a dimension k 1. Baselines include the NOR subgoal space (Nachum al. 2019). randomly selected bit of the state representation and the lowest bit (fast features). For pure exploration comparison, we consider the circumstance of no external rewards, the same the setting of our theoretical analysis in Section 4 Figure b) illustrates that using slow features as the subgoal space can achieve the most efficient exploration with goal-conditioned HRL. As expected, the performance of fast features as the subgoal space is the worst. Randomly selected features perform better than fast features. Al- though NOR aims to bound the sub-optimality of the value function, our method outperforms NOR in terms of exploration.
compare
6.2 MUJOCO TASKS
We evaluate our proposed subgoal representation learning objective on a set of challenging MuJoCo tasks that require a combination of locomotion and object manipulation. The details of our full implementation and environments are available in Appendix B We conduct experiments comparing to the following methods in hierarchical learning and exploration, and all the learning curves in this section are averaged over 10 runs.
Ours
NOR
DCO
EMI
SAC
Oracle
1.0
0.8
Success Rate
Ay
4.3
0.2
6.1
0.0
0.50
0.25
1.00 3
1.50
1775 2.50
0.3
6.5
1.5
1.5
2,0
2.5
3.0
3.5
4,0
0.0
05
1.0
1.5
2.0
2.5
3.0
355
4.0
(a) Point Maze
(b) Ant Maze
An)Ant Push
0.6
1.4
0.8
0.7
SUccess Rata
0.8
0.8
0.4
0.4
0.3
0.4
0.5
0.2
0.1
0.0
0.0
0.5
1.0
15
2.5 Sleps (millon)
2,3
2.5
3.0
4.0
(d) Ant Fall
(e) Ant Maze (Images)
Stops Ins
(t) Ant Push "Imaal (Images)
Steps Instion)
Figure 3: Performance of each method on a suite of MuJoCo environments.
NOR: HRL with a learned subgoal space, which is optimized to bound the sub-optimality of the hierarchical policy (Nachum et al. 2019).
Oracle: HRL with the oracle subgoal space (æ, y position of the agent) in navigation tasks.
DCO: A hierarchical exploration method with deep covering options (Jinnai et al. 2020
EMI: A flat exploration method by predicting dynamics in a latent space (Kim et al. 2019
SAC: The base RL. algorithm used in our method (Haarnoja al., 2018).
Benefiting from better exploration ability, our method with cemporally-cohereee subgoal space significantly outperforms baseline methods in terms of speed and quality of convergence. Even when the raw observation is given by using top-down images, our method can achieve high success rate, presented in Figure 3(e) c). (f). In the Ant Maze task, our method reaches a success rate of 100% only 1.5 million training steps, which is more than two times faster than the NOR algorithm.
FFrr fair comparison, use the online version of DCO, as the offline version needs pretraining process.

---

Published as a conference paper at ICLR 2021
In the Point Maze task, the flat exploration method EMI shows equal performance with approach. However, when the dynamic model is more complex (c.g., for the Ant robot), predicting dynamics becomes much harder, and the performance of EMI degrades dramatically. We evaluate NOR with its published code and results show its ineffectiveness tasks. The online DCO method can hardly learn successful policies in those tasks, partly because the pretraining of the second eigenfunction in their method is necessary.
exploration in challenging
6.3 ANALYSIS OF LEARNED REPRESENTATIONS
We visualize the subgoal representation and learned hierarchical policies of our method the Ant Push task in Figure 4 The learned subgoal space highly resembles the oracle (a, y) position space. By setting subgoals in the learned latent space, the high-level policy guides the agent to jump out of the local optimum of moving towards the goal. The Ant robot under the hierarchical policy firstly moves to the left, then pushes the block to the right, and finally reaches the goal. In contrast, the SAC agent without hierarchical structure easily gets stuck into the local optimum of moving directly to the goal, since the immediate extrinsic reward is given as the negative L2 distance to the environment goal.
high-level action
Goal
C...b 8
Movable block
Start
Ant Push environment
(b) Trajectory y space
(c) Trajectory representation space (k
Figure 4: The color gradient of the trajectory is based on episode timestep (red for the beginning of an episode, blue for the end). Black arrows denoting high-level actions point to the subgoals from the decision- decision-making states. If the Ant robot moves directly towards the goal, it will fail to reach it, as it will push the movable block into the path to the goal.
6.4 PARALLEL LEARNING OF THE REPRESENTATION FUNCTION AND POLICIES
We show the subgoal representation learning process in the Ant Push (Images) task in this section. Figure 5 (a)~(h) visualize trajectories to a hard goal the representation spaces and the visited areas in the space at different learning stages. Along with the representation visualization, we evaluate an easy goal as the midpoint the trajectory to the hard goal.
The learning of the hierarchical policy and the subgoal representation could promote each other. At about 0.2 million timesteps, with the distance-to-goa, dense rewards, our method approximately learns inaccurate subgoal representation and the policy reach the easy goal. Since the learned representation is generalizable to the neighborhood of the explored areas to some extent, which facilitates the exploration of the hierarchical policy, the explored areas expanded little by lit- tle. The newly collected samples in the expanded region could be utilized to improve the subgoal representation further.
6.5 TRANSFFRABILITY OF REPRESENTATIONS
Because of the generality of representation learning objective with slow dynamics, the learned subgoal space is transferable between different tasks of the same robot. The low-level policy in- duced by the learned subgoal representation is transferable well. To verify this transferability, we initialize the representation network and the low-level policy network in target task with those
Cooee https: http://pptppppc..aaaaoooooaaa/ta
ent-
8

---

Published as a conference paper at ICLR 2021
Hard Goal
Easy
X
Hard
Hand Goal
Easy
Start
Easy-goal trajoctory
Hard-goal trajectory
10
Supp
13
Easy
Steps (million)
(a)
0.IM
(c) 0.15M
(d) 0.2M
(c) 0.35M
(1) 0.5M
(
(h)
Trajectories in the representation spaces
Explored areas
Figure 5: Subgoal representations at different learning stages in the Ant Push (Images) task. The red transparent arrows denote the trajectories from the start to the midpoint (easy goal), while the blue denote the trajectories the hard goal. The explored areas visualized by 1000 experiences sampled from the replay buffer.
weights learned in source task and further finetune them in the target task. The high-level policy for the target task is randomly initialized. From Figure we can see that transfer learning helps the agent learn more efficiently and achieve better asymptotic performance.
9.2
3.0
Stops Oneilend
Onalkant
(a) Ani Push > Ant Fall
(b) Ant Muze + Ant FourRooms
(c) Ant Push Ant Fourrooms
Figure 6: Ant FourRooms is a navigation task four-room maze, The source models are ran- domly picked from the runs shown in Figure 3
CONCLUSION
In this work, we propose self-supervised subgoal representation learning method, LESSON. Our approach is motivated by the slowness prior and supports iterative learning of the representation function and hierarchical policies. In addition, we provide theoretical grounding for the slowness prior in hierarchical exploration. We test our method a suite of high-dimensional, continuous control tasks, and significantly outperforms state-of-the-an HRL and exploration methods. Fur- thermore, the subgoal representation and low-level policies learned by LESSON transferable be- tween different tasks. Since the low-level policy learning may result a non-stationary high-level transition function, combining LESSON with off-policy correction methods to reduce the variance of off- off-policy learning might be a promising future direction. Furthermore, as the rewards for the continuous control tasks are deceptive and dense, another challenging problem is learning good subgoal representation and hierarchical policies with extremely sparse rewards.
ACKNOWLEDGEMENTS
The authors would like to thank the anonymous reviewers for their valuable comments and helpful suggestions. This work is supported in part by Science and Technology Innovation 2030 "New Generation Artificial Intelligence" Major Project (No. 2018AAA0100904), and grant from the Institute of Guo Qiang, Tsinghua University.

---

Published as a conference paper at ICLR 2021
REFERENCES
Amitay Bar, Ronen Talmon, and Ron Meir. Option discovery in the absence of rewards with mani- fold analysis. arXiv preprint arXiv:2003. 05878, 2020.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1825 2013.
Justin A Boyan and Andrew W Moore. Generalization in reinforcement learning: Safely approxi- mating the value function. In Advances in neural information processing systems, pp. 369-376, 1995.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR'05). volume 1. pp. 539-546. IEEE, 2005.
Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. In Advances in neural infor- mation processing systems, pp. 271-278 1993.
Nat Dilokthanakul, Christos Kaplanis, Nick Pawlowski, and Murray Shanahan. Feature control as intrinsic motivation for hierarchical reinforcement learning. IEEE transactions on neural net- works and learning systems, 30(1 (0(11))4409-4418 2019.
Chuong Do. The multivariate gaussian distribution. Section Notes, Lecture Machine Learning, CS, 229, 2008.
John Duchi. Derivations for linear algebra and optimization. Berkeley, California, 3:2325-5870 2007.
Zach Dwiel, Madhavun Candadai, Mariano Phielipp. and Arjun Bansal. Hierarchical policy learning is sensitive to goal space design. arXiv preprint arXiv: 2019.
Alberto N Fscalante-B and Laurenz Wiskott. How to solve classification and regression problems on high-dimensional data with a supervised extension of slow feature analysis. The Journal of Machhne Learning Research, (4L):3683-3711,2000
Chelsea Finn, Xin Yu Tan, Yan Duan. Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 512-519. IEEE, 2016.
Mathias Franzius, Henning Sprekeler, and Laurenz Wiskott. Slowness and sparseness lead to place, head-direction, and spatial-view cells. PLoS Comput Biol, 3(8) 166, 2007.
Mathias Franzius, Niko Wilbert, and Laurenz Wiskott. Invariant object recognition and pose esti- mation with slow feature analysis. Neural computation, 2289):228--2233, 2011.
Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal- conditioned policies. arXiv preprint arXiv:1811.07811 2018
Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised feature learning from temporal data. arXiv preprint arXiv: 1504:5504..225. 02518, 2015a.
Ross Goroshin, Michael F Mathieu, and Yann LeCun. Learning linearize under uncertainty. In Advances in Neural Information Processing Systems, pp. 1242- 2015b.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer- ence on robotics and automation (ICRA), pp. 3389-3396. IEEE, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv: 180100000../290. 2018.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
10

---

Published as a conference paper at ICLR 2021
Aren Jansen, Manoj Plakal, Ratheet Pandya, Daniel PW Ellis, Shawn Hershey, Jiayang Liu, R Chan- ning Moore, and Rif A Saurous. Unsupervised learning of semantic audio representations. In 2018 IEEE international conference acoustics, speech and signal processing (ICASSP), pp- 126-130. IEFE, 2018.
Dinesh Jayaraman and Kristen Grauman. Slow and steady feature analysis: higher order temporal coherence in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3852-3861, 2016.
Yuu Jinnai, Jee Won Park, David Abel, and George Konidaris. Discovering options for exploration by minimizing cover time. arXiv preprint
2019.
Yuu Jinnai, Jee Won Park, Marlos C Machado, and George Konidaris. Exploration in reinforcement learning with deep covering options. In International Conference on Learning Representations, 2020.
Rico Jonschkowski and Oliver Brock. Learning state representations with robotic priors. Au- tonomous Robots, 39(3):407-428 2015.
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Ex- ploration with mutual information. In International Conference on Machine Learning, pp. 3360- 3369, 2019.
Diederik P Kingma and Max Welling. 1rXiv:1312.6114,2000 2013.
Auto-encoding variational bayes.
arXiv preprint
Varun Raj Kompella, Marijn Stollenga, Matthew Luciw, and Juergen Schmidhuber. Continual suriosity-drive skill acquisition from high-dimensional video inputs for humanoid robots. Arti- ficial Intelligence, 247:313-335, 2017.
Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Pac reinforcement learning with rich observations. In Advances in Neural Information Processing Systems, pp. 1840-1848, 2016.
Tejas D Kulkarni, Ardavan Sacedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv: 02396, 2016.
Robert Legenstein, Niko Wilbert, and Laurenz. Wiskott. Reinforcement learning on slow features of high-dimensional input streams. PLoS Comput Biol, 6(8)=e100889) 1000994, 2010.
Timothée Lesort, Natalia Dodz-Rodríg,ez, Jean-Franois Goudou, and David Filliat. State represen- tation learning for control: An overview. Neural Networks, 108:379-392. 2018.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In International Conference on Learning Representations, 2019.
Siyuan Li, Rui Wang. Minxue Tang, and Chongjie Zhang, Hierarchical reinforcement learning with advantage-based auxiliary rewards. In Advances in Neural Information Processing Systems, pp- 1409-1419, 2019.
Marlos Machado, Mare Bellemare, and Michael Bowling. laplacian framework for option discovery in reinforcement learning. arXiv preprint 0rXiv:1703.00P 2017a.
Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv: 1710. 1089, 2017b.
Sridhar Mahadevan and Mauro Maggioni. Proto-value functions: A laplacian framework for learn- ing representation and control in markov decision processes. Journal of Machine Learning Re- search, 8(0ct):2169-223) 2007.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv: 16661103675 2016.

---

Published as a conference paper at ICLR 2021
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, nature, 51 518(7540))529-5333 00152299-33,31100
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3303-3313, 2018.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning, In International Conference on Learning Representa- tions, 2019.
Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning long-horizon tasks via visual subgoal generation. arXiv preprint arXiv: 1909.05829 2019.
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal- roal-conditioned policies. In Advances Neural Information Processing Systems, pp. 14843-14854, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: 1rXXiv:807..074555 2018.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Advances neural information processing systems, pp. 4026-4034, 2016.
Alexandre Péré, Sébastien Forestier, Olivier Sigaud, and Pierre-Yves Oudeyer. Unsupervised learn- ing of goal spaces for intrinsically motivated goal exploration. arXiv preprint wrXXiv1803.0077 2018.
Rahul Ramesh, Manan Tomar, and Balaraman Ravindran. Successor options: An option discovery framework for reinforcement learning. arXiv preprint arXiv:1905.05731, 2019.
Jürgen Schmidhuber and Reiner Wahnsiedler. Planning simple trajectories using neural subgoal generators. In From Animals to Animats 2: Proceedings of the Second International Conference on Simulation of Adaptive Behavior, volume 2. pp. 196. MIT Press, 1993.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittweeser, Ioannis Antonoglou, Veda Pannecrshelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489) 2016.
Henning Sprekeler. On the relation of slow feature analysis and laplacian eigenmaps. Neural com- putation, 23(123332773330 2011.
Sainbayar Sukhbaatar, Emily Denton, Arthur Szlam, and Rob Fergus. Learning goal embeddings via self-play for hierarchical reinforcement learning. arXiv preprini arXiv: 1.09083, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. arXiv preprint 1rX3v:1703.01 2017.
Laurenz Wiskott. Learning invariance manifolds. Neurocomputing, 26:925-9333 1999.
Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invari- ances. Neural computation, 14(4):715-770, 2002.
Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in rl: Learning representations with efficient approximations. arXiv preprint arXiv: 1810.04586, 2018.
12

---

Published as a conference paper at ICLR 2021
Chongjie Zhang, Sherief Abdallah, and Victor Lesser. Integrating organizational control into multi- agent learning. In Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Dystems-Volume 2. pp. 757-764, 2009.
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency- constrained subgoals in hierarchical reinforcement learning. arXiv preprint arXiv:2006. / 1485, 2020.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3357-3364. IEEE, 2017.
13

---

Published as a conference paper at ICLR 2021
A PROOF
Theorem 1. Assume p(æ) is multivariate Gaussian distribution: p(æ) ~ 0, R), where R is diagonal matrix diag(r?) and is large enough. Given fixed subgoal space dimension k, selecting the k slowest features for the subgoal space leads to the optimal hierarchical exploration. Denote the distribution of the explored states in this case as qalows we have:
qstow = qp = arg minDkr,(pll4).
(4)
qeQ
where Q is the sets all distributions of explored states brought by different subgoal space selection.
First, we prove that 4(x) is multivariate Gaussian distribution regardless of the distribution of X Since there is no extrinsic reward, the high-level policy will set subgoals to the low-level randomly every steps, thus X Xeo..... ne are independent and identically distributed with the same mean vector μ = E [ tc E IR' and the same covariance matrix Σ1 Denote the average Yn as
Y,
X,
(5)
i=1
By Multidimensional Central limit theorem (Van der Vaart 2000), we have
Vn 1.) 2, (z;
(6)
Plug Eq. 5 into Eq. 6 consider finite samples, we have
Y, L No (w;nu,
When n + oc, the distribution of Y, converges to ((x),, which means g(x)) is a multivariate Gaus- sian distribution. Without loss of generality, we consider the case when 1, i.e., N1 (æ: μu. E),,, to compare different KL divergence induced by different subgoal representations. The exploration process can be formulated as random walk in the state space with continuous action space (Defi- nition 2). The selection of the features for the subgoal space only changes the variance of the unit action X in the random walk, furthermore, deciding the covariance matrix of q(x).
Next, analyze the statistical characteristics of X Since all features are independent, the joint distribution is the product of all the marginal distributions: (x) = I fi f(()), where fx; (1) is the Probability density function (PDF) of X; and fi (x) is the marginal PDF of X; in dimension i. As Assumption (c) indicates that I; E [-r;, ra], if not selecting the i-th feature for the subgoal space, fi()) is a continuous uniform distribution U[-r;, rz], so the variance dimension can be denoted o? 1
However, if we select the i-th feature for the subgoal space, the distribution is modified since the low-level policy is optimal (i.e., the agent moves the subgoal as close as possible during steps). Besides, notice that by Assumption (d), the i-th element of subgoal
is uniformly distributed in [-rg. rgl. Therefore, when gi lies within the interval [x], the agent can reach the subgoal
within steps. When > Ti (g. < -T;), the agent can only reach as far as (-r:). Denote the changed Cumulative Distribution Function (CDF) of X in dimension i (x)), if rg > rs
I I<-
(x)x)
=
Tu-Te 1 (x 2rg
2r,
- S x <
(8)
r4 S
In contrast, if rg I ry. the agent can reach any subgoal within the interval [-rg: so we have
0
-rg -Tg x r.
F((x)
xtr,
2ru
Tg <
In both cases, the mean vector 4 is still 0, but the variance of will increase r?
2r when rg Fs
3r,
and of will decrease of when Vg S ri- Denote the selection operation operator S, and
14

---

Published as a conference paper at ICLR 2021
have
0f
2r
(10)
Ti S rg
8(o
3r
Ti > 1
Finally, want prove if and only if q(x) Istow (z). ( can reach the minimum with the constraint fixed subgoal space dimension k. Consider a distribution g(x) brought by randomly selecting k features from the state space the subgoal space, since p(x) and q(x)) are
both multivariate Gaussian distribution (Assumption (b)), the KI. divergence from g(x) to p(x) is
1 log det (Etp)
det (2ig) I ur (Ey²Ep) + (1g up)² Eg (Ug - -ppp
DRI(Plla) -
(11)
where I is the dimension of g(z), and tr stands for the trace of the matrix (Duchi 2007). Now we want to prove the KL. divergence reaches the minimum if and only if ((x) = qslow (x). Since the covariance matrix Σ is symmetric positive definite, there exists a full rank orthogonal matrix U containing of the eigenvectors of E as its columns and diagonal matrix A such that Σ UAUT (Horn & Johnson 2012). Then NG can be transformed into standard multivariate Gaussian distribution through rotation and stretching.
= B (Y p)) Z NY(
(12)
where Y ~ NG pt (z) -A 1/2 and A 1/2 is a diagonal matrix whose entries are the square roots of the corresponding entries from A (Do 2008). Since g(x)) is symmetrical, thus rotation won't 't change the KI. divergence, so we only need to consider the case where Σ is diagonal matrix, which can be denoted as below.
0
0
Σ
(13)
where of is the variance in dimension i. Notice b4 0, thus Eq.
can be rewritten as 11
1
pia3
of
(Kl())
log
2
2121
2
log
1
(14)
2
22
a²
log
+
2
p
Consider function:
o 22 2 o²
f(a?) log
(15)
It's easy to find that f(a?) is monotonically decreasing when o² E 22) (since r is large enough, the condition is easily met), which means increasing the variance in dimension i will decrease DxL (pllg). Consider feature dimension and feature dimension j. and r, S Ty.. Then we prove
S(S(a?)) + f(a)) < f(o?) + f(S(o))).
(16)
where of ===
- 33.00 o3
We consider three cases:
(1) r; Sr S r,. Recall that selecting the i-th feature for the subgoal space can increase of when rg > r3 while decrease of when rg ri. Therefore, we have:
S(S(o?)) + {(a)) < f(0)) ((3)) +
(17)
15

---

Published as a conference paper at ICLR 2021
(2) Tg S S r3. Since f(S(o?)) + f(oz) f( + f(oz). and f(o?) + f(S(o)))
f(o]) + f. Notice that f(o}) f then Eq. 16 holds.
(3) 14 13 - Tg. Rewrite Eq. 16 as
S(S(o))) - f(o)) E [(S(o))) - [(o})
2r
f f(f) f r3 2r}
=
13
-((7))
3rg
3rg
(18)
2r:
6r 6r 6r
2rg
6r² 6r rg
log
3
+
< log
3
+
rg
(3rgg 2r;)
Tg
(3 2r,)
To prove Eq.
18
we consider another function:
g(x) = log (3 2x)) + 6t - 6t
(19)
x² 2x)
where 1= -- >> It's easy to find that g g(xx) > when x E (0, 1), which means g(x)
Fg
monotonically increasing when r E (0, 1). Therefore, Eq. 18 holds.
Now we have proved that Eq. I6 holds under all possible conditions, which means selecting the slower features the subgoal space will lead to smaller KL divergence, i.e., better exploration. Thus Theorem follows immediately.
B EXPERIMENTAL DETAILS
B. ENVIRONMENTS
The environments of Point Maze, Ant Maze, Ant Push, and Ant Fall are as described in Nachum jtt al. (2019), shown in Figure 7 In each navigation task, we create an environment composed of x 4 x 4 blocks, some movable and some with fixed position. During training, the target locations (I, y) are randomly selected by the environment from all possible points. Final results are evaluated single challenging goal denoted by small green block. For the 'Images' versions of these environments, we zero-out the x, y coordinates in the observation and append a low-resolution 5 x 5 top-down view of the environment, equal to that used in Nachum al. (0099)
Goall
Goal
Goul
Movable
block
Start
Movable block
Start
Start
Point) Maoc
Ani Push
Ant
Ant FourRoums
Figure 7: A collection of environments that use.
The Ant FourRooms task has a much larger maze structure, which is four times as large the Ant Maze task. So the maximal episode length for Ant FourRooms is also larger, which equals 1000. The maximal episode lengths the other tasks are 500.
B.2 NETWORK STRUCTURE
The actor network for each level is Multi-Layer Perceptron (MLP) with two hidden layers of di- mension 256 using ReLU activations. The critic network structure for each level identical to that of the actor network. We scale the outputs of the actor networks of both levels to the range of cor- responding action space with tanh nonlincarities. The representation function o(s) is parameterized by an ML. with one hidden layer of dimension 100 using ReL.U U activations.
16

---

Published as a conference paper at ICLR 2021
B.3 TRAINING PARAMETERS
Discount factor = 0.99 for both levels.
Adam optimizer, learning rate 0.0002.
Soft update targets 0.005 for both levels.
Replay buffer of size 1e6 for both levels.
Reward scaling of 0.1 for both levels.
Entropy coefficient of SAC a = 0.2 for both levels.
Low-level policy length c = 10 for the Point robot and c = 20 for the Ant robot except for the Ant Push task. In the Ant Push task, 50.
Subgoal dimension of size 2. We train the high-level policy to output actions in [-10, 10) when c 10 or c 20 ([-20, 20] when e 50). These actions correspond to desired deltas in state representation.
We did not perform a grid search on hyper-parameters, therefore better performances might be possible for these experiments.
B.4 EVALUATION
Learned hierarchical policies are evaluated every 25000 timesteps by averaging performance over 10 random episodes.
C ADDITIONAL EXPERIMENTAL RESULTS
Table I demonstrates that the dynamics of the features learned by our method are slow. NOR has relatively good performance in Ant Maze and Ant Maze (Images), since the NOR features these two tasks are slower than those in other tasks. The state space of the Point robot is low-dimensional and contains little information other than the (z, y) position, so the slow features (positions) are easy to be selected by random strategy. But NOR projects the state space of the Point robot to latent space with fast dynamics, which results unsatisfactory performance.
Point Maze
Maze
Ant Push
Ant
Ant Maze (Img)
Ant Push (Img)
LESSON
0.05 1 0.01
0.17
0.1410.02
0.1410.02
0.18.10.01
0.14 0.03
NOR
0.316
0.3710.01
0.50.10.05
0.46:10.06
0.3710007
0.40=00.4
Random
0.03
0.94
0.8611021
0.90
0.38
0.46=1.00
Table Slowness of different representations under the policy learned by LESSON, averaged over 5 randomly seeded trials with standard error. The slowness is estimated by one-step feature change [s:)); ((*++1) ]||2 of 100 randomly sampled transitions.
17

---

Published as a conference paper at ICLR 2021
(a) 0.05M
(.)
(c) 0.15M
(d) 0.2M
0.35M
(9 0.5M
(g) IM
(h) 2M
#2
#3
#4
Figure 8: The representation learning process of four runs in the Ant Push (Images) task. Those visualizations demonstrate the gradually learned subgoal representations.
1.0
Early Stopping
0.2M
0.8
0.35M
Success Rate
0.5M
0.6
1M
Ours
0.4
0.2
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Steps (million)
Figure 9; Early stopping the subgoal representation learning at early stages (0.2M, 0.35M, 0.5M and 1M timesteps) in the Ant Push task with visual observations. The early stopping hurts the learning performance, verifying that the subgoal representation function is gradually learned.
18


Reviews:
[{"review_id": "xjWACvS6aIN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main idea of this paper is very nice:  that we want the features in the representation space for subgoals in HRL to be \"slow\" in the sense that they don't change much over primitive steps of the policy, but do change significantly over \"macro\" steps.  The margin-based criterion in the loss (that says we want them to change by at least m at the high level) seems well justified.   \n\nIt seems like, additionally, one might want to incentivize this representation to cover the state space s well (that is, not map big chunks of s onto the same latent representation phi) and I didn't exactly see how this would be encouraged---I guess the drive for significant change at the high level accomplishes this implicitly.\n\nUsing this loss function leads to a sensible algorithm, although I was curious about why the representation learning happens at the same rate as the low-level policy learning;  I could imagine that you might want to do that at a slower time scale (more like the time scale for the high-level policy learning).\n\nThe experimental results seem to make a compelling case for the effectiveness of this algorithm.  The results for transfer learning are particularly strong---in fact transfer seems to be one of the best motivations for learning hierarchical representations, which might be harder to get to pay off in the case of learning to control a single problem instance.  Some of the explanation in this section didn't completely make sense to me:\n- Why do you attribute the success of your method to improved exploration?  It seems like another very plausible explanation is that you are providing a good inductive bias for learning a policy that generalizes well (though 2 million is a lot of steps!)\n- (Related) What is the purpose of the experiments on state coverage?  I don't completely see why that should be, in itself, a goal.  In fact, one attribute of good RL methods that generalize well is that they do early exploration but then focus on parts of the state space that are profitable given the particular high-level reward function they are optimizing.\n\nAll this being good, the theoretical part of the paper was quite weak.    The definitions and theorem were not rigorous and possibly incorrect.   First, from the expositional perspective, it would have helped to begin with crisp definitions and then the assumptions.  Here are many points that came up for me as I read through the text:\n- What is the role of the target distribution?  Where would you get it?\n- In the application of KL divergence It seems more like you'd want the p to be the target distribution.\n- What is a \"fact\"?    Break it down into definitions and then a crisp formal assertion with a proof (if there's something in there beyond definitions).\n- In Fact 1 the term \"exploration process\" is not defined.\n- Importantly, I don't think this is well-formed unless we assume that there is no learning happening at the low level. Are we assuming that?  (If not, then the process is non-stationary).  \n- The high-level policy is assumed to select subgoals \"randomly\" :  Does this mean uniformly at random from S?   Is S bounded?\n- What does it mean for the agent to be able to \"move independently and identically in the state space\"?     \n- What the X^c_t are like depends completely on the low-level policy!  It could always move left, in which case these variables would not be anything like IID.  Is this analysis supposed to be assuming that the low level is perfectly able to achieve the targets?  Or at least always successfully move c steps in the target \"direction\" (not sure what that means, exactly, though).\n- What, precisely, do you mean by \"the features are all independent\"?\n- Brushing assumption (a) off as a general technique to simplify analysis is not so good.  It's okay to say you're making it for now, but there's no reason to think results obtained with this assumption will generalize!\n- Assumption d would require changes to the algorithm, no?\n- In Theorem 1:  \"r is large enough\"  for what?\n- I don't really understand what it means to \"assume q is an isotropic Gaussian ...\"  ;  that is, what exactly is q intended to be.   I see that later you say that if r is big enough it approximates a uniform, and so maybe what you're trying to show is that our exploration is uniform.  But a uniform needs to be over a bounded space.  And even with large r, q is only sort of uniform in a region bounded around the mean.  \n- What makes \"optimal hierarchical exploration\"?   I guess maybe this means:  with a fixed set of features and low-level control policy we're deciding which k features to use for subgoals, and showing that among those choices we get the best exploration (in the sense of matching a Gaussian around the origin) if we use the k slowest features.\n\n(This is just a related thought, not really a review of this paper.  One other principle for selecting features for a subgoal representation is that they should be, in some sense, locally achievable.  Hierarchy works well when the subproblems are, in a sense, \"serializable\":  that we can achieve subgoal the first any way we want to, without thinking about (and/or making it harder to achieve) the next subgoal we're going to be asked to do.    One way to do this might be to encourage some \"disentanglement\" in the latent features, so that the policy for changing one dimension of the latent space tends not to change the other dimensions.)\n\nIn the end, I'm positively inclined toward this paper because it made me think about the concepts of what we really want in a hierarchical representation, but the mathematical exposition really needs substantial cleaning up.  But I'd rather have an interesting paper than a perfectly-executed boring one.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MjyNc1WWcpk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:**\nThis paper proposes a new method for learning subgoal representations in HRL. The method learns a representation that emphasises features that change slowly, through a “slowness objective”. The slowness objective minimises changes in the subgoal representation between low level time steps, while maximising feature changes between the high-level temporal intervals. This objective allows for efficient exploration, which the paper justifies theoretically, and supports with some empirical experiments on challenging control domains.\n\n**Strengths:**\nThe issue of subgoal selection is a critical issue for HRL, and constructing or learning a good subgoal representation on which to create subgoals is important and interrelated. Thus, any significant progress in this area is useful. The paper presents an apparently novel method that would be of interest to HRL researchers and deep RL practitioners more generally.\n\nThe paper is clearly written and the main ideas are generally very well explained (except for maybe the use of the term “state”; see weaknesses). \n\nThe paper provides some theoretical justification for the slowness objective, which is useful to support the intuition behind it. The empirical results are conducted on challenging domains and they appear strong. However, including more independent runs in each domain would strengthen the conclusions significantly (see weaknesses).\n\n**Weaknesses:**\nIn sections 3.1 and 4.1 the paper uses the term \"state\" when describing the algorithm and other definitions. It is not clear if it is actually referring to a proper state in the Markov sense, or some approximation of a state, or even just an observation that might not be Markov at all. This is confusing, and could be clarified. How does the slowness objective interact with approximation when you do not have access to the true state (which would be most of the time)? Does the theoretical result still hold?\n\nI think 10 runs for the NChain environment, and 5 run for Mujoco are too few. More runs would give a much better sense of the variability of the runs, greatly strengthen the conclusions about the relative performance between the algorithms. 20 runs for the NChain environment, and 10 runs for Mujoco would be a significant improvement, but even more is better.\n\nAlso, in the low run regime (~10 runs or less) sometimes it can be more informative just to plot each of the learning curves for all the runs on the same plot, along with the mean. This gives a really good sense of the variability between runs. Presenting the results this way assumes less about the distribution that the performance samples are coming from.\n\n**Recommendation:**\nOverall, I recommend to accept this paper. A clarification about the use of the term “state” and adding more runs to the experiments would increase my score.\n\n**Questions:**\nClarify the use of the term \"state\" (see weaknesses).\n\n**After Author Response and Discussion:** Thanks to the authors for their responses. After reading the other reviews and the author responses, I am raising my score to 7 (accept).", "extra_scores": {}, "extra_text": {}}, {"review_id": "GtGvqFMIca", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper proposes to use slow features as the subgoal representation space in the HRL setting. The proposed approach adopts a slowness objective to learn a representation (high-level action space) and use it to train goal-conditioned policies (low-level).\n\n### Main contributions\n- Slowness is an interesting property for representation learning. This work's approach to slowness is simple and easy to integrate (in practice) to the challenging setting of HRL.\n- This work also makes a theoretical effort (section 4) to motivate the slowness property for subgoal space.\n- The proposed method is evaluated on continuous control tasks against other references approaches to HRL. \n\n### Main comments/concerns\n- In the definition (def.1) of the measure of exploration, what justifies the choice of the reverse KL ? Is it still a good choice if q is multimodal ? Does this definition really align with your claim \"a larger coverage area leads to better exploration\" ?\n- The algorithm implementation is not clear:\n    - How is the goal sampled ? Randomly in the large domain as mentioned in the Appendix or in the neighbourhood of s as mentioned in sections 3 and 4 ?\n    - What is the extrinsic reward used to train the high level policy exactly ?\n- There is a chicken-and-egg problem regarding the parallel learning of the representation and its usage to train the low policies. The representation should first be trained on some area before trusting the induced training of the low policy, which requires exploring that area beforehand. It is not clear how the algorithm deals with this challenge. Moreover, the evaluation on Mujoco tasks could be misleading regarding this point (see the following comment).\n- The method is evaluated on Mujoco environments that provide the xy-coordinates. The representation can learn to extract the coordinates by only training on a limited area (e.g. neighbourhood of initial state) -- being able to early stop the representation training seems to support this hypothesis. Was this phenomenon studied is your experiments ? How is it limiting the proposed method beyond this type of environments ? How does the proposed method work when this inductive bias can't be leveraged, and when exploring an area of the state space is required to build a useful representation of it ?\n- Figure 4 shows the learned representation. Was it learned from the states or the images setting ? If learned from states, how does it look like when learned from images ? And, related to the previous comment, how does it look like along the training ? \n- How was the oracle trained ? It would be interesting to see the oracle training curves along with the compared methods on Figure 3.\n- How reliable/relevant is the slowness evaluation (section 6.3) in comparing the different methods ? It seems that the slowness objective can learn arbitrarily slow features (according to your measure) by weighting the attraction term more than the repulsive one (tuning m and c can also influence the slowness measure).\n- \"As the state space of the Point robot is simple, the dynamics of the randomly selected features are slow as well\": What do you mean by simple state space ?\n- \"the high-level policy guides the agent to jump out of the local optimum of moving towards the goal\" : local optimum w.r.t which reward function ?\n- \"The transfer effect [...] is more significant [...] since the target task, Ant FourRooms, is of a larger size and more difficult\": Is the transfer effect confirmed when transferring from AntPush and AntFall to the larger Ant FourRooms ? \n\n### Minor comments\n- The AntPush performance curves are limited to 2.5 and 2 million steps. How was this horizon decided for this environment ? How do these curves look like for a longer training (4 million steps) ?\n-  Equation 7 seems confusing: p(x) is the asymptotic distribution of Y_n, meaning that it should not depend on n as shown in equation 7.\n- Why does Figure 5(a) has a 4 millions step while 5(b) has 8 million steps ?\n\nClarity: The paper reads well, but the experiments and algorithm presentation is sometimes unclear if not confusing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8R5fsGTqEtT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a method for learning a slow-changing (\"low-frequency\") embedding function, which can be used as a state-abstraction function in the context of Hierarchical RL. A high-level policy, trained to solve the environments task, acts by selecting abstracted states as targets for the low-level policy which is trained as in a goal-conditioned fashion (acting in the environment itself attempting to get to the set goal).\n\nThe paper provides theoretical motivation for the focus on \"slow\" features in a simplified synthetic example. The method itself is evaluated on high-dim control tasks (As well as on an illustrative toy-example).\n\nThere is a lot to like about this paper. The method is elegant, building on and extending previous approaches to HRL and particularly learning subgoal representations. The use of a low-frequency (slowness) criterion for this purpose is, to the best of my knowledge, a novel contribution. The work presents strong empirical evidence for the usefulness of the approach. The paper itself is well-structured and overall well-written, easy to follow, and with good references.\n\nI do have a few questions for which I would appreciate the authors comments:\n* In principle, the loss for the embedding function (highly) depends on the behavioral policy. This is because the embedding is trained to represent \"far away\" states as different from each other. And, generally, the distribution of $s_t$ and $s_{t+c}$ can be rather different for, say, a random policy compared to a more purposeful/trained one. Since everything is trained jointly here (embedding is used to train the policies which in turn can change the embedding and so on), this could in principle leads to instability of the training. I think this point deserve at least some discussion.\n\n* I find section 4 a bit confusing. \"Fact 1\" seems more like a definition for me, because the properties of the \"random walk\" itself will be *determined* by $p$ (as $p$ obviously depend on the behavioral policy. In fact from \"Fact 1\" we can simply say that $p$ is the steady-state/limiting/stationary distribution of the markov chain induced by the policy and the MDP). In general the relation between $p$ and the policy is not entirely clear. Note that in general, the change metric $\\Delta s^i_t$ is also policy dependent. I also don't see the direct relation between $p$ in this discussion, which ultimately is an expression of the (low-level) policy, to the embedding or representation principle used throughout the paper.\n\n* It seems the underlying assumption here is something like: the agent can independently control/change the state-features in an unconstrained way (i.e an \"action\" is an offset vector s.t the next state is $s_{t+1}=s_t+a_t$, but in such a way that the environment somehow 'rescales' $a_t$ so that earlier entries are smaller). If this is indeed the case, I find this section not too motivating. If there is completely no structure in the environment (state/action spaces) and everything reduced down to a uniform random-walk, we cannot learn much about even simple \"realistic\" RL settings. \n\n* Having said that I think the paper presents other good reasons and motivations for the focus on slow-features. In particular I think that the example discussed in section 6.1 is important, as well as the more qualitative explanations and the relation to the unsupervised-learning literature around similar ideas.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an HRL method that uses slow features to define subgoals (or abstract states), which can then be used by goal-conditioned policies. It is said that such an approach allows for efficient exploration. Most reviewers are recommending the acceptance of this paper, they found the method interesting and they think it introduces interesting ideas that are not that common to the HRL literature. Thus, I’m recommending the acceptance of this paper.

I’d still encourage the authors to take the reviewers comments into consideration when preparing the final version of the paper. Specifically, it would be useful to explicitly discuss the “chicken and egg problem” and the fact that the agent has access to a function defining the distance to the goal before the goal was observed for the first time. Some baselines have the same assumption, but it is somewhat weird to discuss exploration in this setting without further clarifications.


Decision: Accept (Poster)