Active Hierarchical Exploration with Stable Subgoal Representation Learning
Authors: Siyuan Li
Jin Zhang
Jianhao Wang
Yang Yu
Chongjie Zhang
Venue: ICLR 2022 Poster (2022)
Keywords: Hierarchical Reinforcement Learning
Exploration
Representation Learning

Abstract: Goal-conditioned hierarchical reinforcement learning (GCHRL) provides a promising approach to solving long-horizon tasks. Recently, its success has been extended to more general settings by concurrently learning hierarchical policies and subgoal representations. Although GCHRL possesses superior exploration ability by decomposing tasks via subgoals, existing GCHRL methods struggle in temporally extended tasks with sparse external rewards, since the high-level policy learning relies on external rewards. As the high-level policy selects subgoals in an online learned representation space, the dynamic change of the subgoal space severely hinders effective high-level exploration. In this paper, we propose a novel regularization that contributes to both stable and efficient subgoal representation learning. Building upon the stable representation, we design measures of novelty and potential for subgoals, and develop an active hierarchical exploration strategy that seeks out new promising subgoals and states without intrinsic rewards. Experimental results show that our approach significantly outperforms state-of-the-art baselines in continuous control tasks with sparse rewards. 

Published as a conference paper at ICLR 2022
ACTIVE HIERARCHICAL EXPLORATION WITH STABLE SUBGOAL REPRESENTATION LEARNING
Siyuan Lif, Jin Zhang?, Jianhao Wang?, Yang Yuţ, Chongjie Zhang? +Tsinghua University, &Nanjing University
(sy-1i17) din-zhan20, wjh1 (eth11996mn @mmat s. tsinghua. edu. cn 1yuy]e1. amda. elada... nju.. edu.en, chongj i el singgghua edu. cn
ABSTRACT
Goal-conditioned hierarchical reinforcement learning (GCHRL) provides a promising approach to solving long-horizon tasks. Recently, its success has been extended to more general settings by concurrently learning hierarchical policies and subgoal representations. Although GCHRL possesses superior exploration ability by decomposing tasks via subgoals, existing GCHRL methods struggle temporally extended tasks with sparse external rewards, since the high-level pol- icy learning relies on external rewards. As the high-level policy selects subgoals online learned representation space, the dynamic change of the subgoal space severely hinders effective high-level exploration. this paper, we propose novel regularization that contributes to both stable and efficient subgoal representation learning. Building upon the stable representation, we design measures of novelty and potential for subgoals, and develop an active hierarchical exploration strategy that seeks out new promising subgoals and states without intrinsic rewards. Exper- imental results show that our approach significantly outperforms state-of-the-as baselines in continuous control tasks with sparse rewards.
INTRODUCTION
Goal-conditioned hierarchical reinforcement learning (GCHRL) has long demonstrated great poten-
tial to solve temporally extended tasks (Dayan & Hinton 1993 Schmidhuber & Wahnsiedler 1993]
Vezhnevets et al., 2017 Péré et al. 2018: Nair Finn 2019), where a high-level policy periodically
sets subgoals to a low-level policy, and the low-level policy is intrinsically rewarded for reaching those subgoals. Early GCHRL studies used a hand-designed subgoal space, such as positions of
robots (Nachum et al. 2018 Levy et al.) 2019) objects in images (Kulkarni et al., 201 To alleviate the dependency on domain-specific knowledge, recent works investigate learning subgoal representations along with hierarchical policies (Nachum et al., 2019a| Dilokthanakul et al. 2019 Li ct al., 2021). which have shown promise in a more general setting. Although the exploration ability of HRI. is boosted via decomposing tasks and reusing low-level policies, existing GCHRL. methods struggle in long-horizon tasks with sparse external rewards. The high-level policy learn- ing in GCHRL relies on external rewards, and those tasks are hard solve with naive high-level exploration strategies. Furthermore, as the high-level policy makes decisions in a latent subgoal space learned simultaneously with the hierarchical policy, the instability of the online-learned sub- goal space results in severe non-stationarity of the high-level state transition function and hinders effective high-level exploration.
To address these challenges, we develop an active Hicrarchical Exploration approach with Stable Subgoal representation learning (HESS). Our approach adopts a contrastive objective for subgoal representation learning inspired by Li et al. 2021). To learn subgoal space that could effectively guide hierarchical exploration, we propose a novel state-specific regularization that contributes to both stable and efficient subgoal representation learning. The proposed regularization constrains the changes for those embeddings that have already well satisfied the representation learning objective. As the hierarchical agent gradually expands its exploration to new state regions, the regularization allows for updates of the embeddings that underfit the learning objective. Benefiting from the pro- posed regularization, we could improve the representation learning efficiency without hurting its

---

Published as a conference paper at ICLR 2022
stability via the prioritized sampling technique (Hinton 2007 which prioritizes training samples with larger losses.
Building upon stable subgoal representation learning, design active exploration strategy for high-level policy learning. As shown in previous work (Strehl & Littman 2008 Bellemare et al. 2016 Ostrovski et al.] 2017), the visit count provides simple and effective novelty measure for exploration. However, with online subgoal representation learning, the visit count for subgoals defined in changing space. Therefore, such a novelty measure alone is not sufficient for efficient high-level exploration, although the stability regularization could improve its accuracy. Our insight that desirable novel subgoals should be reachable and effectively guide the agent to unexplored areas. Thus we design novel potential measure to regularize the novelty measure, which indicates the reachability to the neighbor regions of the visited state embeddings. With the regularized nov- elty measure, our proposed active exploration strategy directly selects state embeddings with high potential and novelty as subgoals without introducing intrinsic rewards. It more efficient than the reactive exploration methods that need to learn how to maximize the intrinsic rewards before performing exploratory behaviors (Tang et al. 2017; Pathak et al. 2017; Burda et al. 2018), since in the active exploration approach, the subgoal measures have direct effects the behavior policy. Furthermore, the active exploration strategy naturally avoid introducing additional non-stationarity (i.e., induced by dynamically changing intrinsic rewards) into HRL.
We compare the proposed method HESS with state-of-the-art baselines in a number of difficult control tasks with sparse rewards. Note that the environment setting in this paper is much more challenging to exploration than the multi-task and deceptive dense-reward ones in the baselines. Ex- perimental results demonstrate that HESS significantly outperforms existing baselines. In addition, we perform multiple ablations illustrating the importance of the various components of HESS.
2 BACKGROUND
We consider a Markov Decision Process (MDP) defined as a tuple (S, P, y), where is a state space, A is action space, P(s'ls, is unknown transition function, r: S A A I is reward function, and 7 E [0, 1) is a discount factor. Let ı (a]s) denote a stochastic policy over actions given states. The objective of reinforcement learning (RL) is to learn a policy that maximizes the expected cumulative discounted rewards: max, Ep.- / rrr(Se, az)], where T denotes the horizon length.
9o
9c
50
Sc
12
ı
TT1
so (sp)
sz (s,)
Sc-1 (a-1lo((sc-)))
sc sello(se)
ag
a
ac
Environment
Figure 1: The hierarchical framework of GCHRL.
In GCHRL, two-level hierarchical policy Theer is composed of a high-level policy Tr (gels:) and a low-level policy ne mr(urls/s9)) 9t), as illustrated in Figure [ A latent subgoal space G is abstracted by representation function (bs)= S + Rd. The high-level policy m.s samples a subgoal ge from a neighborhood (()(s))) of the current latent state Φ(s;) every e steps, i.e., when = (mod c): (d((s)) (gt E D(t,, o(s))) S rg), where is a fixed constant, D distance function, and rg is a radius of the neighborhood. The sampled subgoal is repeated, Jr -t-1, when # 0 (mod c). Jy is trained to optimize the expected extrinsic rewards. The low-level controller takes primitive action a₂ A every step, and is intrinsically rewarded to reach subgoals set by the high level, r/ = -D ()). To provide dense non-zero rewards for low-level policy learning, employ L2 distance D. Furthermore, to keep the low-level reward function r' stationary while learning é, we concatenate states and state embeddings together as the low-level states, s' s|lø(s).
Previous work (Li et al. 2021) ) minimizes contrastive triplet loss Tari to learn Φ(s), where positive
2

---

Published as a conference paper at ICLR 2022
pairs adjacent states in trajectories, and negative pairs states steps apart.
(5: St+1, S(+c) = ||o(s;) ((s++1)l22 + maz(0, ||Ø(,,) - o(s)+))l2))
(1)
where 6 margin parameter. In this work, we also adopt the contrastive loss in Equation
METHOD
In GCHRL, the low-level policy is optimized with intrinsic rewards generated by subgoals, but external rewards for the high level are still sparse and hard to explore. Furthermore, the changing subgoal space makes it even more challenging for high-level exploration. this section, we first present novel regularization that contributes to stable and efficient subgoal representation learning, and then introduce two measures for subgoals, novelty and potential. Finally, we design an active hierarchical exploration strategy based these two measures.
3.1 STABLE SUBGOAL REPRESENTATION LEARNING
Both stability and efficiency are crucial to subgoal representation learning. The stability of sub- goal representation contributes the stationarity both the high-level transition function and the low-level reward function. Meanwhile, a fast learned subgoal representation could provide effective guidance to exploration. However, stability seems odds with efficiency, c.g., training neural net- works with smaller learning rates leads to better stability but is slower (Goodfellow et al. 2016). In this subsection, we develop a novel state-specific regularization that resolves the Stailitt--effiiiee dilemma in subgoal representation learning.
To achieve stable representation learning, we propose a regularization L, which restricts the repre- sentation change during each update. I, works by anchoring the current representation (0s) to the old subgoal representation (sta(s) before the update as follows:
L (s) @ota())lla).
where B is a replay buffer, and A(s) is a function that controls the weight of regularization for dif- ferent states. States with smaller representation losses fit the learning objective well, so X(s) should be larger for those states. In practice, before each representation update, we rank the triplets in the buffer with Leri Sc+1, St+e). For the top 6% of the triplets with the minimum representation losses, we set A(s) = do 0 for the anchor states (5x) in these triplets, and for the other states, X(s) =
The stability regularization enables us to use prioritized sampling (Hinton 2007 to improve repre- sentation learning efficiency without hurting its stability. The overall loss for subgoal representation learning is:
T (e..)))111))))) [Tere 8141, St+c)] + Tes:
where B, a prioritized replay buffer, and states with larger Tari have higher probabilities of being sampled for training.
3.2 MEASURES FOR SUBGOALS
2008 Belle- mare et al. 2016 Ostrovski et al.) 2017 Tang al. (017), formulate the novelty of subgoals with visit counts. As desirable subgoals should incentivize the agent to explore faraway novel states, we consider both immediate counts n (()(s:) and expected cumulative counts of future states, and the novelty measure N (o(s))) is defined as follows:
Novelty Measure: Inspired by count-based exploration methods (Strehl & Littman
(((ss)) E ETALOL
Σ *n((((54)()))
In practice, we partition the www-dimessional continuous latent space into discrete cells, i.e., the state embeddings are mapped into cells containing them. By maintaining how many times each cell is visited, we could estimate the visit count n()(s)). In practice, the novelty measure is approximated with the data from the replay buffer, and the implementation detail is described in Appendix
3

---

Published as a conference paper at ICLR 2022
With online representation learning, the novelty measure is mixture of counts in the past and current representation spaces, so it might mislead the exploration, as demonstrated in Figure B Our insight is that desirable novel subgoals should be reachable and effectively guide the agent to unexplored areas. Therefore, we design a potential measure for subgoals to regularize the novelty measure. In the following, first introduce subgoal generation mechanism, which is involved in the definition of the potential measure.
Potential Measure:
To guide the low-level controller to reach unexplored states, the subgoals pursued by the low level had better be in unexplored areas as well. Therefore, we propose to add some perturbations to the subgoal Ir selected from the replay buffer and obtain an imagined subgoal Je. and then pass ge to the low-level policy. To enable ge in an unexplored or less explored area, the perturbation is conducted as extending 9: in the direction of gt ((x)), as illustrated in Figure
High Level
Extend
Select
(5s))
gi
Replay Buffer
ge
& (St+c)
Explored arca in the latent space
Low Level
Figure 2: schematic illustration of the subgoal selection and perturbation.
The imagined subgoal is ge 9t + de: - ((st))/llge d(s))ll2. where de denotes an extended distance. As ge imagined, and may have not been visited before, it could be inherently unreach- able due to the transition function of the MDP or online representation learning, e.g., there may be obstacles in navigation tasks. To encourage the agent to explore in promising and reachable direc- tions, we define a measure of potential U (g:) for the selected subgoal ge as the expected negative distance between the ending state &6 (8t+e) and 9e:
U (g:) ) Es, -e.. (ttc)) Ie)], where
(5)
8, Ph; at Tr( 6. ge), S1+1 P(st+1|st,a)) at).
Ph is the high-level state distribution under the hierarchical policy Thier. Building on (Li et al. 202T), the representation learned with the contrastive objective preserves temporal coherence, and the distances between nearby features approximately represent the number of transitions between them (Oord et al., 2018). Hence, with higher potential (g:)). is more reachable, and thus ex- ploring the direction of Sr is more promising to expand the explored areas. The potential U(ae) in Equation 5 is estimated from the data in buffer as well. To calculate the novelty, we partition the continuous representation space into discrete cells. Similarly, we maintain the potential of a cell by averaging the potential of features in that cell, and use the potential of the cell to represent that of the states inside it.
(a) XY density
(b) Latent density
(c) Feature changes
(d) Potential
(e) Combining novelty & potential
features at 0.2M steps at 0.15M
Figure 3: Visualization of visitation density and potential in the Ant Maze task. (a) Visitation density
in the x, y coordinate space of the Ant robot. (b) Visitation density in the subgoal representation
space. (c) Feature changes between 0.15M and 0.2M steps for the same batch of states. (d) Potential
for the sampled state embeddings. (e) Combination of novelty and potential in Section 3.3 Our
method selects state embeddings with darker colors as subgoals.
Illustrative example: In Figure B a) and b)), we visualize the visitation density at 0.2M steps in the Ant Maze task in Figure 5 The visitation density is the counts normalized by the total number of
transitions in the buffer. Comparing Figure 3 a) to ()),) we found a mismatch between the density in the v. y coordinate space and the representation space, especially in the red box, since the updated
4

---

Published as a conference paper at ICLR 2022
representation at 0.2M steps has projected the state embeddings to somewhere new, and the feature changes are noted by the black arrows in Figure B(c). Furthermore, the counts frontiers of the latent explored areas would not increase with more exploration, the online learned representation keeps changing. With the potential measure, we could distinguish the promising novel areas from the unpromising ones, illustrated in Figure3]d)
3.3 ACTIVE EXPLORATION STRATEGY
In the reactive exploration methods with intrinsic rewards, the agent needs to learn how to maximize cumulative intrinsic rewards before performing exploratory behaviors. Therefore, the effects of intrinsic rewards on behavior policy is indirect. Furthermore, the changing intrinsic rewards would introduce additional non-stationarity into HRL. To make the proposed measures directly influence the behavior policy and avoid the non-stationary issue, we develop without intrinsic rewards for high-level policy learning. Based on the two measures in Section 3.2 the proposed exploration strategy selects and explores novel subgoal Ir with high potential, specified as follows:
active exploration strategy
= aragiin min (ó(s)) aU(((s)) 6(a)
subject D(o(s), o(s)) S B
where 9t is the current state and is scaling factor. To balance these measures more easily, we normalize future count (ő(s)) by the total number of transitions the buffer, and denote with ((((s)) We visualize the proposed exploration incentive in Figure Bíe.. Taking the state
embeddings with darker colors as subgoals could lead the agent expand the explored areas.
Algorithm [ provides the full procedure of the proposed approach, HESS. To balance exploration and exploitation, HESS probabilistically utilizes the exploration strategy Equation [or explores with the learned policy тr (Line:5-6), and the probability is decayed over the course of learning. The representation é is updated every I episodes (Line: so that stable during the update interval, and the proposed regularization is maintain the stability before and after the update.
Algorithm HESS algorithm
1: Initialize: Tr. s). mi (als', and d()).
2: for = _eprsuseppisoes do
3: for 0..I' do
4: if = 0 (mod e) then
5:
With probability of p. explore with the strategy in Eq. 6
6:
With probability of 1 p, sample subgoal ge with learned policy mh. Update Tr with an off-policy RL method.
7:
8:
else
9:
9: 9t-1 end if
11:
Tt. 81+1 <- execute o; m1(-1o6, 9t), and update with an off-policy RL method.
end for
13:
if i 0 (mod I) then
14:
Update @ with Eq. using m minibatches.
15:
end if
16:
end for
17:
Return: The Ty and ó.
4 RELATED WORK
Learning effective subgoal representations has been an important and challenging problem in GCHRL (Dwiel et al. 2019). Previous works have proposed to learn the subgoal representation in an end-to-end manner with policies (Vezhnevets et al. 2017: Dilokthanakul et al. 2019). or by bounding the sub-optimality of the hierarchical policy (Nachum al., 2019a). with a learning objective slow dynamics (Li et al. 202T). Nevertheless, those methods have not considered the stability of the subgoal representation learning, which results in non-stationary high-level learning
5

---

Published as a conference paper at ICLR 2022
environment. Other prior methods utilize predefined pretrained subgoal space (Péré al. 2018 Nair & Finn 2019 Zhang et al. 2020, Ghosh et al., 2018) to keep the stability of the subgoal repre- sentation. However, those methods require task-specific human knowledge extra training data. In this work, we propose a novel regularization to stabilize the subgoal representation learning. Specif- ically, the subgoal representation is learned with contrastive triplet loss. The contrastive objective is used as an auxiliary representation loss in other RL literature as well (Oord et al., 2018 Laskin al. 2020 Fortunato et al. 2019).
Benefiting from temporally extended exploration, HRL methods have shown better exploration abil- ities (Nachum et al., 2019b). Bottom-up HRL works learn a set of diverse skills or options a self-supervised manner, and use those semantically meaningful low-level skills to explore in down- stream tasks (Jinnai et al. 2020 Co-Reyes et al. 2018, Eysenbach et al. 2018 Sharma et al. 2019, Li et al., 2019). Nevertheless, the skills produced by those methods may not be required by the downstream tasks, since when learning the skills, the agent knows nothing about the downstream task rewards. In GCHRL, Zhang et al. (2020) restricted the high-level action space to k-step ad- jacent region of the current state to achieve better sample efficiency. |Röder et al. (2020) aimed at improving high-level exploration via auriosity-driver intrinsic rewards. However, those methods require oracle subgoal spaces designed with prior knowledge. In contrast, HESS learns subgoal rep- resentations online. [Li et al. (202T) ) proposed HRL approach, LESSON, with online subgoal rep- resentation learning, and provided theoretical analysis that the learned representation could support efficient exploration. However, without intrinsic rewards at the high level, LESSON could hardly solve long-hone Long-horizon tasks with extremely sparse rewards. This paper improves exploration from an orthogonal perspective from LESSON, and develops an active exploration strategy for high-level policy learning.
We propose two measures for subgoals: novelty and potential. The cumulative count in the novelty measure is related to successor features (Kulkarni et al. (2066)). Previous exploration methods use successor features to propagate the uncertainty value function (Janz et al. 2019) or whether the features occur (Machado et al. (020). In contrast, our method propagates explicit estimation of the number of feature occurrence through trajectories, which is conducted without function approx- imation. The potential measure of reachability shares some similarities with empowerment (Salge) eet al. 2014), which is defined as the agent's ' control over its environment. By maximizing empower- ment (Campos al., 2020) Gregor al. 2016 Mohamed & Rezende, 2015)) an agent could reach more diverse states. Unlike empowerment measured by mutual information, the potential measure is formalized with the distances between desired subgoals and achieved latent states. In Appendix we discuss how HESS is related to goal selection strategies in the multi-goal RL domain (Schaul
al. 2015).
5 EXPERIMENTS
We evaluate HESS on a set of challenging searse-reward environments that require a combination of locomotion and object manipulation skills, aiming at answering the following questions: (1)
Can HESS outperform state-of-the-art exploration strategies in sample efficiency and overall perfor-
mance? (2) Can HESS learn stable subgoal representation space efficiently? (3) How important
are the various components of the HESS agent?
(4) What do the subgoals selected by HESS look like? (5) How much would the choice of hyper-paun paremmerrssssss influence the experimental performance? For better sample efficiency, we utilize an off-policy algorithm. Soft Actor- (SAC) (Haarnoja al., 2018), to learn both-level policies. To mitigate the influence of the entropy term of SAC on HESS, we use SAC with automatic entropy tuning.
5.1 ENVIRONMENT SETUP
We evaluate on suite of MuloCo (Todorov et al. 2012) tasks that are widely used in the HRL community, including Ant (or Point) Maze, Ant Push, Ant FourRooms, Cheetah Hurdle, Cheetah Ascending, and two variants with low-resolution image observations. The experiments with image input are labeled 'Images'. To demonstrate the exploration ability of HESS, we adapt those tasks to make them more challenging. Different from the settings of random start random goal with dense external rewards (Nachum et al., 2018: 2019a] Li et al. [202T] Zhang et al. 2020), in the tasks used in this work, simulated robot starts from fixed position and needs to reach faraway target position

---

Published as a conference paper at ICLR 2022
HESS
H-IM
LESSON
Maze
[c] Maze
Chestah Hurdle
1.0
1.3
1.4
Success
0.8
2
(e) Ant (mages)]
(images)
(g) Fourfooms
Ascending
1.3
Success
Stato
Stcm
RASD
Figure 4: Learning curves of the proposed method and baselines on all the tasks. The y axis shows the average success rate in 10 episodes. Each line is the mean of 5 runs with shaded regions cor- responding to a confidence interval of 95%. All the curves have been smoothed equally for visual clarity. Code is available at atthtps::/ / github. com/ SiyuanLee/HES
with sparse external rewards. Furthermore, there is no predefined subgoal space provided. More details about the environments and implementtion are available in Appendix A and respectively.
5.2 COMPARATIVE ANALYSIS
We conduct experiments comparing the proposed hierarchical exploration approach to the state-of- the-art baselines: (1) H-ICM (Pathak al., 2017): prediction error in the latent subgoal space as intrinsic rewards to the high level. (2) H-SR Machado et al., 2020):: count-based exploration bonus in the high level, and the counts are estimated by the norm of the successor representation. (3) DADS (Sharma et al. (2019): : an unsupervised skill discovery method via predicting dynamics. (4) LESSON (Li al., (021): an GCHRL method learning the subgoal representation online using triplet loss, and no intrinsic rewards in the high level. (5) SAC (Haarnoja et al. (018):: the non-hierarchical base RL algorithm used this work.
As shown in Figure the proposed method substantially outperforms the baselines in all the tasks. The outperformance in the Ant FourRooms task is more significant since the maze scale of this task is larger, and the exploration problem is harder. Both H-ICM and H-SR underperform our method. The dynamic model of the simulated physics engine is relatively easy to fit by neural networks, so the intrinsic rewards of H-ICM may vanish before the policies have been learned well. The H- SR method estimates visit counts using the & fn-nrrm of the successor representation, which is the expected discounted future state occupancies in trajectories. This idea shares the cumulative counts in the novelty measure. Nevertheless, the successor representation estimates the expected future state occupancy starting from given state (Dayan) 1993), but not the visitation number of the given state, which is less helpful to promote exploration.
similarities with
When predicting dynamics in the whole observation space, the intrinsic rewards of DADS could hardly help the agent learn gait skills that make the robots move. The original paper (Sharma et al., 2019) has also demonstrated that without 2, y prior, the trajectories generated by primitive skill have large variance. We provide a visualization of those skills in Appendix [ As result, the success rates of DADS on all the tasks are zero. Hence, we omit them from Figure LESSON uses no intrinsic reward to promote exploration in the high level. Even with a good subgoal representation space, still could not solve those tasks with sparse rewards. The hon-hierarchical method SAC performs poorly in all the tasks, demonstrating the strength of the hierarchical structures in solving long-horizon tasks with sparse rewards.
Goal
5.3 VISUALIZATION OF REPRESENTATION LEARNING
We visualize state embeddings of 5 trajectories from the beginning to the end of the 3-shape corridor in the Ant Maze (Images) task in Figure comparing the representations learned by HESS and HESS without stability regulariza- tion. Those representations are all learned from top-down image observations.
Figure 5: Ant Maze
HESS is able to learn an effective subgoal representation at an early stage. By optimizing the contrastive objective, the Fuclidean distances in the latent space approximately cor-

---

Published as a conference paper at ICLR 2022
HESS
(b) HESS wi stability regularization
0 0.075M
0.25M
0.45M
0.65M
0.85M
L.5M
1.25M
1.45M Steps
Figure 6: Subgoal representation learning process in the Ant Maze (Images) task. Each subfigure contains 2D state embeddings of 5 trajectories in the D-shape maze (red for the start, blue for the end). For larger axis labels, see videos of the representation learning process at uttttps://siteeee
google ccom/vieww weeiee/heae wwwee//whsiiclii
respond the number of transitions between states in local areas, which provides subgoal-reaching rewards for low-level policies. But globally, the distance could not represent the number of transi- tions. For example, it requires lots of transitions to move from the start the goal in the D-shaped maze, but they have not been pushed much away in the latent space, since there is no explicit con- straint in the triplet loss for the distance between features more than timesteps apart. Comparing Figure ((a) to Figure b()). we find that with the stability regularization, when the features have already fitted the learning objective well, their changes are minor afterward. There is almost no change for the red features from 0.25M steps to the end in Figure aaa.
Without the stability regularization, all the features are changing dramatically. The non-stationary issue makes high-level policy learning very hard. In Figure (]b), at 0.25M steps (the second sub- figure), the high-level agent at the red start should select a subgoal to move upwards. However, at 0.65M steps (the fourth subfigure), selecting subgoal in the right of the start state is better.
5.4 ABLATIVE ANALYSIS OF VARTOUS COMPONENTS
To understand the importance of various design choices of HESS, we conduct ablation studies on the subgoal representation learning and the proposed exploration strategy separately. In the ablation studies of representation learning, we compare HESS, HESS without stability regularization, and HESS without both stability regularization and prioritized sampling, shown by the curves without markers in Figure 7 In the tasks with image input, the advantage of stability regularization and prioritized sampling more substantial, since the representation of image observations is harder to leam. In the tasks with vector input, the subgoal representation has a good generalization ability, so the effect of stability regularization and prioritized sampling becomes less significant.
(8) Ant Maze (Images)
(b) Ant Push (Images)
(c) Ant FourRooms
HESS
stability
Success
3.6
Success
Success
w/o stability and prioritized sampling
wie potential
replace ((e(st) with n(a(s))
0-
0.2
reactive exploration
0.0
0.0
0.0
(millon)
Oetilionn
Imilliont
Figure 7: Ablation studies of representation learning and the exploration strategy.
In the ablation studies of the exploration strategy, we compare HESS, HESS without the potential measure, replacing cumulative counts N (((s)) with immediate counts n(((s)), and reactive explo- ration with high-level intrinsic rewards (s) m/ /N(()(((s))
Tz/V-U(b(ss). shown by the curves with markers in Figure The potential measure is extremely important, since without it, the subgoals are concentrated on the areas with low counts, but not extendable, like the areas near the corners in Figure b). Therefore, the success rates without the potential measure much lower.
With immediate counts n(((s)) as the novelty measure, the agent loses a long-term vision to seek out novel subgoals, and only focuses on novel states in its neighborhood. Hence, the states near the walls are selected as subgoals more frequently in Figure ((c) than in Figure M(.). but those subgoals still better for exploration than the subgoals selected without the potential measure. In the ablation study of active exploration, we can see that by actively selecting favorable subgoals, HESS has achieved better sample efficiency than the reactive exploration method with intrinsic
8

---

Published as a conference paper at ICLR 2022
(a) HESS
(b) w/o potentual meusure
(6) Replace N (d)))) with (s))
explored areas
selected subgoals
Repessoration Space
Repesstratio Space
Reprssemtion Space
Figure 8: Visualization of the selected subgoals during 10 episodes and their corresponding positions in the x, space in the Ant Maze task at 0.25 million timesteps.
rewards, the reactive method could not immediately seek the novel and promising subgoals even when it has detected them.
5.5 ABLATION STUDIES ON
SELECTION
In this section, we set up of ablation tests several hyper-parameters of HESS in the Ant Push (Images) task. From Figure | we use single suite of parameters in all the tasks this paper, which also indicates that the proposed approach is robust to pppam--r selection.
we can see that most parameters work in a large range. Furthermore,
(a) Scaling factor
(b) Extended di distance d.
(c) Stable ratio 9%
(d) Low-level policy langth c
2- 0.01
d
k5k=0.2
c=10
0.63
d,-5
R54-0.3
C-20
k56-0.5
C-50
Succes
SuccEs
0.2
Siea Iniiiet
Stape tevilann
Supp mn)
1nd Morl
Figure 9: Ablation studies of hyper-parameter selection in the Ant Push (Images) task.
Scaling factor a balances novelty and potential. Smaller a encourages HESS to choose more novel state embeddings as subgoals. Results in Figure 9 (a) indicate that HESS is robust against since proposed method could work well, when the potential measure mitigates the negative influence of novelty inaccuracy exploration. For all tasks in Section 5.2
we set 0.03.
Extended distance de is the distance between the subgoal selected from the replay buffer and the imagined subgoal ge- To keep 9e still near the current latent state, de should be no larger than the radius rg of the neighborhood for selecting subgoals. The performance long de is not too large, level too far away from the current latent state. For all tasks in Section 5.22 we set de 5.0.
HESS is reasonable as excessively large de may cause the subgoal Je pursued by the low
Stable ratio k%: The stability regularization constrains the changes of the embeddings for k% of the states with the minimum triplet loss in the buffer. When 1% is too large, the representation learning efficiency may be slightly affected, which harmful to the hierarchical policy learning, as indicated in Figure 9 (c). For all tasks in Section 5.2 we set k% = 0.3, i.c., the regularization is applied to 30% of the data in buffer.
Low-level policy length e is an important and common hyper-parameter in HRL. With a larger e, the burden of high-level decision- decision-making lighter, but low-level policy learning becomes harder. The learning performance is better when is 20 or 50 with an episode length of 500. Among all the hyper-parameters e seems influence the performance most. For all tasks and baselines in Section we set 50.
5.2
CONCLUSION
To solve long-horizon sparse-reward tasks with GCHRL, we design novelty and potential measures for subgoals upon stable subgoal representation learning, and develop hierarchical exploration strategy that actively seeks out new promising subgoals and states. As the dimension of the subgoal space in this work is low, we employ naive count estimation method in the representation space. When the dimension of the subgoal space is higher, it would be better to learn a density model (Papamakarios et al. 2019). or utilize more advanced count estimation methods, such as kemel- based methods (Davis et al.) 2011). For the future work, believe that the idea of stability is general and could be used beyond HRL. For example, maybe many existing techniques in non-deep RI. could be applied to representations learned with stability well.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work is supported by Science and Technology Innovation 2030 "New Generation Artificial Intelligence" Major Project (No. 2018AAA0100904) and National Natural Science Foundation of China (No. 2021 2021 1300509).
REPRODUCIBILITY STATEMENT
For reproducibility, we include anonymous downloadable source code in the supplementary ma- terial. Beyond that, we describe the details about the environments and the implementation in Ap- pendix A and B.
REFERENCES
Mare G Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. arXiv preprint 15t::6606.0868 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei Efros. Large-scale study of crrrosity-driven learning. arXiv preprint 1FXX::18888043555 2018.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Interna- tional Conference on Machine Learning, pp. 1317-1327. PMLR, 2020.
John Co-Reyes, YuXuan Liu, Abhishek Gupta. Benjamin Eysenbach, Picter Abbeel, and Sergey Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajec- tory embeddings. In International Conference on Machine Learning, pp. 1009-1018. PMLR, 2018.
Richard Davis, Keh-Shin Lii, and Dimitris Politis. Remarks on some nonparametric estimates of density function. In Selected Works of Murray Rosenblatr, pp. 95-100. Springer, 2011.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Computation, 5(4):613-624, 624, 1993.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in neural infor- mation processing systems, pp. 271-278, 1993.
Nat Dilokthanakul, Christos Kaplanis, Nick Pawlowski, and Murray Shanahan. Feature control as intrinsic motivation for hierarchical reinforcement learning. IEEE transactions on neural net- works and learning systems, 30(1 11:3409-3418, 2019.
Zach Dwiel, Madhavun Candadai, Mariano Phielipp, and Arjun K Bansal. Hierarchical policy learning is sensitive to goal space design. arXiv preprint arXiv: 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: 1022:1800.00000, 2018
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528. PMLR, 2018.
Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomènech Badia. Gavin Buttimore, Charlie Deck, Joel Z Leibo, and Charles Blundell. Generalization of reinforcement leamerrs with working and episodic memory. arXiv preprint 1rXiv:1910.13406 2019.
Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal- conditioned policies. arXiv preprint arXiv: 1.108//9 2018
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
10

---

Published as a conference paper at ICLR 2022
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv: 2016.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint FrXiv:/8/2.05000 2018.
Geoffrey E Hinton. To recognize shapes, first learn generate images. Progress in brain research, 165:535-547, 2007.
David Janz, Jiri Hron, Przemyslaw Mazur, Katja Hofmann, José Miguel Hernández-Lobato and Se- bastian Tschiatschek. Successor uncertainties: exploration and uncertainty in temporal difference learning. Advances in Neural Information Processing Systems, 32:4507-4516, 2019.
Yuu Jinmai, Jee Won Park, Marlos C Machado, and George Konidaris. Exploration in reinforcement learning with deep covering options. In International Conference on Learning Representations, 2020.
Tejas D Kulkarni, Karthik R Narasimhan, Ardavan Saeedi, and Joshua B Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. arXiv preprint (644v:666440005555 2016a.
Tejas Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel Gershman. Deep successor reinforcement learning. arXiv preprint arXiv: 12060066, 2016b.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, pp. 5639- 5650. PMLR, 2020.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In International Conference Learning Representations, 2019.
Siyuan Li, Rui Wang. Minxue Tang, and Chongjie Zhang. Hierarchical reinforcement learning with advantage-based auxiliary rewards. In Advances in Neural Information Processing Systems, pp. 1409 1419, 2019.
Siyuan Li, Lulu Zheng, Jianhao Wang, and Chongjie Zhang. Learning subgoal representations with slow dynamics. In International Conference Learning Representations, 2021. URL
https://copenreview net http:/enneeeettieeeeeeeeoeoe
Marlos C Machado, Marc G Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pp. 3125-5133, 2020.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsi- cally motivated reinforcement learning, arXiv preprint /rXiv:/509.08733 2015.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. arXiv preprint arXiv:
2018.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. In International Conference on Learning Representa- fions, 2019a.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv:/909 10618, 2019b.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. arXiv preprint arXiv:1807.0477 2018.
Suraj Nair and Chelsea Finn. Hierarchical foresight: supe-supe learning of long-horizon tasks via visual subgoal generation. arXiv preprint arXiv: 1909.0929909 2019.
11

---

Published as a conference paper at ICLR 2022
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018.
Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02266. 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pp. 2778-2787. PMLR, 2017.
Alexandre Péré, Sébastien Forestier, Olivier Sigaud, and Pierre-Yves Oudeyer. Unsupervised learn- ing of goal spaces for intrinsically motivated goal exploration. arXiv preprint 2018.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pp. 7750-7761. PMLR, 2020.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. arXiv preprint FrXiv:1903003690 2019.
Frank Röder, Manfred Eppe, Phuong DH Nguyen, and Stefan Wermter. Curious hierarchical actor- critic reinforeement learning. In International Conference on Artificial Neural Networks, pp- 408-419. Springer, 2020.
Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment-an introduction. In Guided Self-Organization Inception, pp. 67-114. Springer, 2014.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
Jürgen Schmidhuber and Reiner Wahnsiedler. Planning simple trajectories using neural subgoal generators. In From Animals to Animats 2: Proceedings of the Second International Conference on Simulation of Adaptive Behavior, volume 2, pp. 196. MIT Press, 1993.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: 1XX.:1907.01657 2019.
Alexander L Strehl and Michael L Littman. An analysis of model based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8): (1(8):1309-111 1309-1331, 2008.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In 31st Conference on Neural Information Processing Systems (NIPS), volume 30, pp. 1-18, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEFF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. arXiv preprint arXiv:1703.0116) 2017.
David Warde-Farley. Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non- con-paramttric discriminative rewards. arXiv preprint arxiv: 10.e:1//1.11333 2018
12

---

Published as a conference paper at ICLR 2022
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency- constrained subgoals in hierarchical reinforcement learning. arXiv preprint arXiv:2006. 1485, 2020.
13

---

Published as a conference paper at ICLR 2022
A ENVIRONMENT DETAILS
The environments of Point Maze, Ant Maze, Ant Push, Ant FourRooms, Cheetah Hurdle, and Chee- tah Ascending are shown in Figure The environment rewards are sparse and binary. We define "success" being within an L2 distance of 1.5 from the goal. If the agent successfully reaches the goal, the reward I, and else the reward sition and velocity of the simulated robot and the environment goal. For the 'Images' versions of these environments, we zero-out the 2, coordinates in the observation and append a low-resolution 5 x top-down view of the environment, described in (Nachum al. 2019a Li al.. 2021). Each episode ends at 500 timesteps for the Ant Maze, Point Maze, and Ant Push task and ends at 1000 timesteps for the Ant FourRooms, Cheetah Hurdle, and Cheetah Ascending task.
The observation space includes the current po-
Goal
Goal
Goal
Goal
(a)
(b) Ant
Push
(d)
Cheetah Hurdle
Cheetah Ascending
Figure 10: A collection of benchmark sparse-reward environments.
Ant Maze. The size of the D-shape maze is 12 x 12. For both training and evaluation, the start position is fixed to (0, 0), and the environment goal is fixed to (0, 8).
Point Maze. The maze structure is the same as the Ant Maze task. The simulated robot a point mass, which is controlled by actions of dimension two, rotation on the pivot of the point mass and movement in the direction
pivot.
Ant Push. In this task, movable block is placed at (0, 4). The agent is initialized position (0, 0), and the goal position is fixed (0, 8). Therefore, agent must first move to the left, then push the movable block to the right, and then navigate the goal unimpeded.
Ant FourRooms. this environment, the maze structure is larger, 18 x 18. The start position is (0, 0), and the goal is (14, 14).
Cheetah Hurdle. The simulated Half-Cheetah robot is required to stride over a hurdle with a height of 0.35 and walk to the goal, which is 3.2 meters from the start position.
Cheetah Ascending. The simulated Half-Cheetah robot has to climb up to a hill to reach the goal. The height of the goal position is I.
B IMPLEMENTATION DETAILS
we list other ranges considered in the experiments in Table The probability of actively selecting subgoals with the proposed exploration strategy initialized as 0.7, and linearly annealed at half the total training steps. For subgoal selection, method firstly samples M states satisfying the constraint
Except for the
discussed in Section 5.5
and their
14

---

Published as a conference paper at ICLR 2022
from the replay buffer B, {st
((s))) < Ng -- B, and then selects a favorable subgoal
with Equation 6 from the candidate subgoal set {o($:));= When calculating the expectation in the novelty and potential measures (Eq. 5) to deal with the offlpolic of the data in the buffer, decrease the effects of the older rollouts with a discount factor = 0.995, taking inspiration of the Exponential Moving Average (EMA) method.
For the neural network structures, the actor network for each level is Multi- Perceptron (MLP) with two hidden layers of dimension 256 using ReLU activations. The critic network struc- ture for each level is identical to that of the actor network. We scale the outputs of the actor networks of both levels to the range of corresponding action space with tanh nonlinearities. The representa- tion function ((s) is parameterized by an MLP with one hidden layer of dimension 100 using ReLU activations. We use Py Porch and Adam optimizer to train all the networks. Experiments are carried out on NVIDIA GTX 2080 Ti GPU. The learned hierarchical policies are evaluated every 25000 timesteps by averaging performance over 10 episodes.
Table 1: Hyper-parameters for experiments
spper-parameters
Values
Ranges
Subgoal dimension d
2
Radius rg of subgoal selecting neighborhood
20
{10, 20)
Grid size for count estimation
3
(2, 8}
Number of candidate subgoals M
1000
Learning rate for both level policies
0.0002
Learning rate for the subgoal representation
0.0001
Discount factor ? for both levels and cumulative counts
0.99
Soft update rate for both levels
0.005
Replay buffer size for both levels Reward scaling for high level Reward scaling for low level High-level action frequency Batch size for both level policies Batch size for representation learning Intrinsic reward coefficient 171 Intrinsic reward coefficient 172
1e6
0.1
(1,
1
{1, 0.13
50
(10, 20, 50, 100)
128
100
1000
{10, 100. 1000]
10
{10, 100, 1000]
Stability regularization coefficient Ào
0.1
Number of episodes I between representation updates
100
Number of minibatches m in one representation update
50000
{ (10000, 50000}
C RELATED WORK
With the proposed novelty and potential measures, we develop a subgoal selection strategy to im- prove exploration in HRL. There are numerous goal selection strategies in the multi-goal domain (Schaul et al. 2015) as well, including sampling diverse goals uniformly from a buffer (Warde- Farley et al.] 2018 Pong et al. 2019), sampling goals from the achieved goal distribution (Nair ct al. 2018), sampling goals of intermediate difficulty from a generative model (Florensa et al. 2018), and selecting goals in sparsely explored areas (Pitis et al. (020). However, those meth- ods use predefined or pretrained goal spaces. In addition, unlike HESS, those flat goal-conditioned methods pursue only one goal in episode and thus may be less able to decompose complex tasks.
D BASELINES
H-SR is the baseline method using count-based intrinsic rewards in the high level, where the counts are estimated with the & no-mr of the successor representation (Machado et al.) 2020). Since the l fo-nrrm of the successor representation only counts for whether the features occur in trajectories, but not the times of feature occurrence. Therefore, the intrinsic rewards of H-SR vanish at an early learning stage. We show how the variance of the intrinsic rewards changes with the environment
15

---

Published as a conference paper at ICLR 2022
timesteps in the Ant Maze task in Figure 11 Since there is little difference in the intrinsic rewards among different states, the H-SR intrinsic rewards could hardly facilitate efficient exploration in late learning stage.
Variance of intrinsic rewards
60
0
(million)
Figure 11:
The variance of the intrinsic rewards of the H-SR method in the Ant Maze task, with
x-axis as millions of environment timesteps.
DADS learns set of primitive skills by predicting the skill dynamics and then uses the learned skills in downstream tasks (Sharma et al. 2019). The intrinsic rewards of skill learning encourage skill produce transitions (a) predictable under the dynamic model and (b) different from the transitions produced under other skills. However, when the dynamic model is defined on the whole observation space without the F, y prior, the trajectories of the learned skills suffer from large variance, as shown in Figure 12 Those high variance skills offer limited utility for hierarchical control.
2-
Figure 12: X-Y traces of skills learned by DADS in the Ant Maze task, where the same color represents trajectories resulting from the execution of the same skill.
FeUdal Networks (FuN) Vezhnevets et al. (017) also utilizes directional subgoal vectors. We have run this method in the Ant Maze task for 70 million steps, and the success rate FuN is still zero. We analyze the reason for the underperformance of FuN, and find that the subgoal representation learned by FuN is much worse than ours, as shown Figure 13 since the representation learning the contrastive objective to learn the
in FuN lacks strong inductive bias. In our approach, subgoal representation.
XY
FuN representation
Figure 13: Trajectories in the XY space and the FuN representation space. The dimension of the FuN representation space is 50, and we use t-SNE to perform dimension reduction and visualize the latent trajectories.
16

---

Published as a conference paper at ICLR 2022
E ADDITIONAL EXPERIMENTAL RESULTS
E. ABLATION STUDIES ON ADDITIONAL HYPER-PARAMETER
We conduct ablation studies on additional hyper--paameeeee in the Ant Maze and Ant Maze (Images) task, and find that the representation update interval which influences the representation stability as well, is also important to learning performance.
interval
Pa
1-350
bemtet
P-007
1e500
bow2e5
Pe-10
(Images)
Figure 14: Ablation studies on additional hyper-parameters.
Representation update interval / (episodes) balances the stability and efficiency of the represen- tation learning. The task performance is robust when / is in proper range (50 I / S 150). With short update interval (T = 50), the learning performance is slightly hurt since the subgoal represen- tation is less stable. When I is too large (I 500), successful policy could hardly be learned in the Ant Maze (Images) task, as the large update interval hinders the representation learning efficiency.
Buffer size bs for both levels. Figure [4 demonstrates that HESS is indeed robust against this hyper-parameter when 2c5 < bs < 2c6.
Initialization po for probability p. When Pn is set relatively large value (Po 0.5), HESS achieves stable performance, as shown in Figure [4] Small Po (0.3) makes the proposed exploration strategy seldom used, which leads poor sample efficiency.
E.2 ABLATION STUDY ON IMAGINED SUBGOALS
To guide the low-level controller to reach unexplored areas, we firstly sample state embedding ge from the neighborhood of the current latent state with radius of rg. and then extend 9e with an extended distance d, to obtain an imagined subgoal ge- In this section, we compare the performance of directly increasing rg and perturbing gr in subgoal selection.
1.0
0.8
0.8
Success
Success
0.6
0.4
0.4
1g 20=
0.2
20, d. =5
= 25, =
= 25, d. =
0.0
0.0
4 (b) Ant Push (Images)
Steps (million)
Steps (millian)
(a) Ant Push
Figure 15; Ablation studies the imagined subgoals in Ant Push and Ant Push (Images) task.
Results in Figure 15 demonstrate that when de > 0, the sample efficiency is better, ge is obtained via perturbing an achieved subgoal, which might be in the unexplored areas, and thus more helpful to exploration. Only increasing Ty the sampled subgoal 9; is still be in explored areas, which is less useful to expand the explored areas.
17

---

Published as a conference paper at ICLR 2022
E.3 NOVELTY ESTIMATION
We estimate the novelty measure (((s)) with the data in the replay buffer. To make the nov- elty estimation more accurate, need to deal with the off-policyness of the data in the buffer, so we adopt the EMA method to decrease the effect of old rollouts, as described in Appendix [ An alternative way deal with the offlpolicysess is the importance sampling (IS) technique, i.c., (o(s))) E& | T is the current high-level policy, Th is the old high-level sampling policy at state Sec. B is the replay buffer. We compare these two implementations in the Ant Maze task, and those two methods empirically demonstrate comparable performance, shown in Figure [6
where paa = I [(itee)/ < (g)stc)/ (ot (g (tc),,
1.0
0.8
Success
0.6
0.4
EMA
0.2
1S
IS(high&low)
0.0
2
3
4
Steps (million)
Figure 16: Different novelty estimation in the Ant Maze task.
E.4 IMPLEMENTATION OF THE STABILITY REGULARIZATION
In this section, we ablate different implementations of the weighting function X(s) in the stability regularization:
Binary A(s): For the top k% of the triplets with the minimum representation losses,
A(s) = o for the anchor states (s₂) in these triplets, and for the other states, A(x) = O.
Continuous A(s): Weighting A(s) continuously based on the representation loss, i.c., A(s,) = >o/ /1 Leri(Ses ', where the triplet (St, y is uniformly sampled from the replay buffer.
Compared the continuous X(s) implementation, binary X(s) sharply distinguishes the well- optimized embeddings from the underfitting ones, and thus has slightly better performance, as demonstrated in Figure 17
(a) Ant Maze (Images)
(b) Ant Push (Images)
0.8
Success
Success
0.6
0.4
Nm
0.2
binary Als)
continuous Als)
3
5
Steps (million)
Steps (mmiliia))
Figure 17: Different implementations of weighting function A(s).
18


Reviews:
[{"review_id": "xsrdRPbFvMf", "reviewer": "Reviewer_XmcD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- I appreciate that this paper studies the subgoal learning instability and high-level exploration which are of importance to GCHRL research.\n- The regularization method for representation stability is reasonable, simple but empirically effective. Meanwhile, the representation instability is a problem encountered in many other scenarios and the proposed regularization method is general and of potentials to be leveraged in other representation learning problem.\n- This paper proposes effective active exploration for high-level exploration of GCHRL. To my knowledge, the subgoal perturbation along with the definition of potential is new in GCHRL. I appreciate the combination of novelty and potential which properly takes novelty and reachability into consideration for an effective exploration selection.\n- The experiments are extensive, well evaluating and demonstrating the characteristics of HESS across multiple perspectives.\n\n\n&nbsp;\n\n\nWeaknesses:\n- I think the methods proposed in this paper are relatively simple and somewhat incremental, however, thanks to the solid experiments, the effectiveness of these methods are demonstrated. At a first glance, the representation regularization seems to be disconnected to the active exploration method. Later, I found that the stable representation learned is important to the effectiveness of novelty calculation. I recommend the authors to make the connection more obvious for a better convey of the story.\n- Although I think the methods are reasonable in an overall view, I have a few concerns on concrete implementations. I list my questions and concerns below.\n\n\n&nbsp;\n\n\nMy first concern is the calculation of novelty (Equation 4). I have no question on the maintenance of $n(\\phi(s))$; but for the calculation of accumulated visit count of high-level state trajectory, I wonder given a state $s_i$, how the trajectory of policy $\\pi_{hier}$ is obtained exactly? \n\n&nbsp;\n\nSecond, for Equation 5, since the potential is defined over the expectation of high-level transition obtained by $\\pi_{hier}$ with the perturbed subgoal $g_e$, how are such transitions obtained?\n\n&nbsp;\n\n\nFor both above two concerns, one possible way is to simulate the rollouts with a world model, but this seems not to be the way used in this paper. Alternatively, are these approximated with the trajectories in the replay buffer? If so, how should consider the off-policyness and suboptimality?\n\n&nbsp;\n\nThe third question is on the computation complexity. The top $k%$ selection in representation regularization and the calculation in Equation 6 (the selection of candidates according to the constraints, the calculation of novelty and potentials). It seems the computation is heavy for these. What are the practical implementations?\n\n&nbsp;\n\nBesides, I have a few questions on the experiments.\n- How to understand that some baseline algorithms work better in image-version environments? E.g., H-SR/H-ICM on Ant-Maze and LESSON on Ant Push.\n- Is the sentence ‘so the intrinsic rewards of H-ICM may vanish before the policies have been learned well’ checked in the experiments?\n- In Figure 6, are the same 5 trajectories used for the upper and lower panels at each time point? And what are the trajectories exactly, since at the beginning of learning, the agent fails to reach the final goals (according to the results in Figure 4)?\n\n\n&nbsp;\n\n\nMinors:\n- Can the authors explain more on the sentence ‘to keep the low-level reward function $r^l$ stationary while learning $\\phi$, we concatenate states and state embeddings together as the low-level states’ (above Equation 1)?\n\n&nbsp;\n\n\nI will raise my score if my questions and concerns mentioned above can be well addressed.\n\n================================\n\nPost-rebuttal comments:\n\nSome of my concerns and questions are well addressed. I raised my evaluation to a borderline acceptation although my main concern on the relatively complex mechanisms involved in the implementation, e.g., hash table, iterative sampling and fitering, table look-up and so on (a few of these are sample-wise), remains. And I think these computation implementation should be noted and described in detail in the revision later. However, I recognize the authors' efforts in pushing the boundary of HRL.\n", "summary_of_the_paper": "The paper studies Goal-conditioned Hierarchical RL (GCHRL) and proposes a new algorithm called Hierarchical Exploration approach with Stable Subgoal representation learning (HESS) to improve the stability of subgoal representation learning and strengthen the exploration at high level. HESS is built on previous method LESSON. The instability of subgoal representation learning is alleviated by a representation regularization which is utilized to encourage the representation to be stable for the states with relatively lower triplet losses (originated from LESSON). Further, this paper proposes an active exploration method for the high-level learning. The method is built on the definitions of Novelty and Potential of states, which corresponds to accumulated visit counts of high-level state trajectory and a negative distance to the perturbed subgoals. Extensive experiments are conducted in a few MuJoCo environments with sparse reward, demonstrating the superiority of proposed algorithm and the effectiveness of different ingredients.", "summary_of_the_review": "I vote for a borderline acceptation after discussing with the authors. I recognize the authors' efforts in pushing the boundary of HRL although I still have some concerns on the complexity of the proposed methods and the practical computation cost.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QzCifjiL97A", "reviewer": "Reviewer_yhF9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. \n\nI think this paper is interesting and explores a novel set of ideas. The baselines also seem reasonable. The closest baseline in terms of using directional goal vectors is Feudal Networks. I would have expected to see a head to head comparison with this approach, even though this proposed method goes beyond it. However, the core idea of having a meta controller output goal vectors and then sub-controllers learning to execute them was explored in Feudal networks. \n\n2. \n\nWhat are the effects of changing the option termination condition? Currently it is hard coded to be c. What are the implications of this? Do the authors observe any deviations or improvements if this hyper parameter varies. It seems like the potential function, novelty measure and option termination are deeply interlinked. It would have been good to more clearly understand the relationship between these measures.\n\n3. \n\nFigure 4 is the main quantitative figure. It seems important to test the effects of stability regularization. This is highlighted qualitatively in Figure 6 but not shown in Figure 4. \n\n4.\n\nThe qualitative analysis on the effects of the interaction between potential and novelty measure is quite sparse. It is not clear how it fails and where it works. Figure 6 is helpful but it needs improvements in terms of clarity and scope (other environments)\n\n5. \n\nFigure 5 is truncated at 5 million steps. How does the asymptotic performance look like for this method? Does it plateau sooner than baselines? What is the maximum achievable reward for these tasks? ", "summary_of_the_paper": "1. This paper investigates learning stable subgoals within a deep hierarchical reinforcement learning setup. \n\n2. Two controllers are learned from the same experience replay buffer. The high level controller serves as a meta controller and the low level controller serves as a goal-achieving agent. The high level controller communicates abstract goals to the low level controllers. \n\n3. The high level controller is optimized using an extrinsically specified reward function. The low level controller optimizes the intrinsic goal communicated by the high level controller. The subgoals are changed after a deterministic time length (known option termination).\n\n4. The subgoals are designed with the key insight that \"desirable novel subgoals should be reachable and effectively guide\nthe agent to unexplored areas.\". Typically count-based, predicted or successor feature based rewards have been used as novelty measures. However, these fall short in terms of reasoning about reachability of states. To handle this, a potential measure for subgoals is proposed which regularizes the novelty measure.\n\n5. To go to unexplored states, a directional goal is synthesized/imagined using the current state and a directional vector. The potential function makes sure that this is approximately reachable by formulating reward as the expected negative distance between the ending state and imagined goal. This is similar to Feudal networks (Vezhnevets et al.) but goes beyond it to handle diversity and reachability. \n\n6. The approach is validated on a set of hard to explore continuous environments with reasonably strong and relevant baselines.", "summary_of_the_review": "This paper presents an interesting and novel idea at the intersection of deep HRL, novelty based exploration and reachability. The experiments are sound but could require further clarification and expansion of scope. The clarity of the paper can also be improved to more directly address the need and important of stability regularization. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PBzO4_83_0", "reviewer": "Reviewer_EgA9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is really well executed. It builds on top of an already complicated architecture adding more than one new component to that architecture, but it does so while providing proper intuitions for each one of these new components and, more importantly, actually doing ablation studies that quantify the impact of each component. To me, Section 5.4 is the highlight of the paper. I also appreciated Section 5.5, which shows how the paper is also concerned with stability over different parameters introduced by the proposed metric. I think the paper would benefit from further clarifying some parts of the text, but otherwise this is a good paper. Specifically:\n\n* In the Introduction, for example, it is said that methods based on visit count are \"_reactive_ exploration methods that need to learn how to maximize the intrinsic rewards before performing exploratory behaviors\". I don't necessarily disagree with that, although the whole idea of visit counts is to incentivize these exploratory behaviors. My question though, is: isn't this exactly the same with the proposed idea? It does use counts and not only that, but also expected state visitation counts for the trajectory, which is even more demanding in terms of having to visit the state first.\n\n* In Section 2, when defining $U(\\phi(s_t))$, the distance is defined to be between $g_t$ and $\\phi(s_t)$. For the proposed algorithm, should it be $g_e$ instead of $g_t$? Still on Section 2, it is said \"we concatenate states and state embeddings together as the low-level states\". What does this actually mean? What are the states here? For images, for example, would it literally be all pixels on the screen?\n\n* In Section 3.1, I don't think $\\lambda_0$.\n\n* In Section 3.2, it is said \"low-dimensional continuous latent space into discrete cells, i.e., the state embeddings are mapped into cells containing them.\". What are these cells? How were they defined? I can imagine this is somewhat straightforward to do if you assume you have access to x,y positions, but how is this done in higher dimensional settings? How are these cells defined for Images, for example?\n\n* In Section 3.2, when discussing the potential measure, it is said that Figure 3 demonstrates that \"with online representation learning, the novelty measure is a mixture of counts in the past and current representation spaces, so it might mislead the exploration\". How is that? I couldn't understand what I should be looking at in Figure 3 to reach this conclusion.\n\n* In Section 3.3, it is said that the active exploration strategy \"avoids the non-stationary issue\". How? Aren't these reward signals changing constantly based on counts and the representation being learned? How does the active exploration strategy actually avoids the non-stationarity issue?\n\n* In Section 4, it is said that \"Bottom-up HRL works learn a set of diverse skills or options in a self-supervised manner, and use those semantically meaningful low-level skills to explore in downstream tasks\", but \"those methods may produce some redundant and useless skills\". This claim is not backed up by any reference or experiment. Why is this true when some of these methods explicitly ask for diverse skills that are not supposed to overlap to each other?\n\n* In Figure 4, how were the confidence intervals computed if only 5 samples were available?\n\n* In Section 5.2 it is said \"the successor representation estimates the expected future state occupancy starting from a given state (Kulkarni et al., 2016b), but not the visitation number of the given state, which is less helpful to promote exploration.\" However, isn't this exactly what H-SR shows? That the $\\ell_1$ norm of the SR captures the visitation number of a given state? Moreover, the reference to the SR should be \"Peter Dayan: Improving Generalization for Temporal Difference Learning: The Successor Representation. Neural Comput. 5(4): 613-624 (1993)\". \n\n* No details were given on how Figure 6 was generated. I don't know how to reproduce it.\n\n* *Importantly, in the ablations, were the parameters of the ablated methods tuned?*\n ", "summary_of_the_paper": "This paper proposes a new algorithm for goal-conditioned hierarchical reinforcement learning that is able to succeed in tasks with sparse rewards (differently from most other methods in the field). It does so through two innovations: (1) a representation learning procedure that is more stable, and (2) a exploration strategy that takes into consideration not only novelty but also reachability.\n\nSpecifically, the representation learning procedure is based on what is now a standard a contrastive loss, but it is augmented by a regularization term that make the learning procedure stable where the representation is already satisfactory, allowing goal sampling to be more effective. Figure 6 is a particularly nice visualization of the impact of this regularization term.\n\nThe exploration strategy to sample goals to be visited is also novel. Instead of using goal visitation counts, this paper proposes the idea of using expected sum of state visitation counts from that state onwards, capturing some notion of long term novelty. Moreover, the exploration bonus also has a potential term that captures how promising each goal state is in terms of how far from the goal state the agent is expected to end up. Quantitative impact is reported in Figure 7, but I particularly liked the intuitions/visualizations provided in Figure 8.", "summary_of_the_review": "This paper is really well executed. It builds on top of an already complicated architecture adding more than one new component to that architecture, but it does so while providing proper intuitions for each one of these new components and, more importantly, actually doing ablation studies that quantify the impact of each component. To me, Section 5.4 is the highlight of the paper. I also appreciated Section 5.5, which shows how the paper is also concerned with stability over different parameters introduced by the proposed metric. I think the paper would benefit from further clarifying some parts of the text, but otherwise this is a good paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IEk6WFp9Ahu", "reviewer": "Reviewer_GV2u", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n1. Comparisons between the proposed method and other hierarchical methods demonstrate that the algorithm results in better performance.\n2. The authors performed thorough ablations demonstrating the impact of each proposed component of their algorithm.\n\n**Cons**\n1. The authors do not explain how the counts and potential measures are estimated from data. In particular:\n    1. How are the cumulative counts $N(s)$ in (4) estimated, given $\\pi_{hier}$ is changing over the course of training?\n    2. How is $U(g_t)$ estimated from buffer data, given that the expectation is calculated with $g_e$ being set as a subgoal for the policy, and thus would not have been observed in the actual environment rollouts?\n2. Why is prioritized sampling used in Equation 3? The motivation on this point was not really explained in detail.\n3. For the ablative analysis, it seems like it would be better to evaluate reactive exploration using cumulative counts instead of immediate counts to better isolate the impact of reactive exploration versus learning a policy to maximize the same intrinsic rewards.\n\n**Clarification Questions**\n1. Why does choosing $\\lambda(s)$ as a continuous function of the representation loss impose heavy computational demands? It seems like the losses are already being computed in the process of obtaining the triplets with minimal representation losses.\n2. How is the latent space partitioned into cells if there are no knowns bounds on $\\lVert \\phi(s) \\rVert$ a priori?\n3. In motivating the potential measure, the authors claimed that the “novelty measure is a mixture of counts in the past and current representation spaces”, but it is unclear why this is the case if one can easily recompute $n$ when $\\phi$ changes.\n4. How is the low-level policy training done? Is hindsight experience replay used?", "summary_of_the_paper": "The authors propose a hierarchical RL algorithm which augments an existing contrastive learning-based subgoal representation objective with heuristics for exploration. The proposed algorithm seeks to reduce representation drift over the course of learning by penalizing the learner for modifying $\\phi(s)$ for states $s$ with low contrastive loss. Furthermore, the authors propose exploration heuristics that encourages the learner to explore in promising areas of latent space by combining count-based novelty and potential measures. The proposed algorithm is demonstrated to have the desirable properties, and outperforms existing methods. The analysis is complemented by an ablative analysis that disentangles the effects of each proposed mechanism.", "summary_of_the_review": "Overall, I vote for a weak accept. The ideas in the paper are interesting, and the experimental evaluation is thorough and demonstrates the benefits of the proposed algorithm. However, the work could benefit from a more detailed description of how the relevant measures are estimated, as well minor changes to the experimental procedure.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new goal-conditioned hierarchical RL method aimed at improving performance on sparse reward tasks. Compared to prior work the novelty lies in a new way of improving the stability of goal representation learning and in an improved exploration strategy for proposing goals while taking reachability into account.

The paper does a good job of motivating the main ideas around stability and combining novelty with reachability. Reviewers found the quantitative evaluation and the choice of baselines to be good with the exception of not including Feudal Networks which the authors explained was due to poor performance on the hard exploration tasks (something that has been observed in prior work). Reviewers also found the thoroughness of the ablations and insightful visualizations to be highlights. Overall, reviewers were unanimous in recommending acceptance, which I support.
Decision: Accept (Poster)