Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies Goal-conditioned Hierarchical RL (GCHRL) and proposes a new algorithm called Hierarchical Exploration approach with Stable Subgoal representation learning (HESS) to improve the stability of subgoal representation learning and strengthen the exploration at high level. HESS is built on previous method LESSON. The instability of subgoal representation learning is alleviated by a representation regularization which is utilized to encourage the representation to be stable for the states with relatively lower triplet losses (originated from LESSON). Further, this paper proposes an active exploration method for the high-level learning. The method is built on the definitions of Novelty and Potential of states, which corresponds to accumulated visit counts of high-level state trajectory and a negative distance to the perturbed subgoals. Extensive experiments are conducted in a few MuJoCo environments with sparse reward, demonstrating the superiority of proposed algorithm and the effectiveness of different ingredients.
main_review: Strengths:
- I appreciate that this paper studies the subgoal learning instability and high-level exploration which are of importance to GCHRL research.
- The regularization method for representation stability is reasonable, simple but empirically effective. Meanwhile, the representation instability is a problem encountered in many other scenarios and the proposed regularization method is general and of potentials to be leveraged in other representation learning problem.
- This paper proposes effective active exploration for high-level exploration of GCHRL. To my knowledge, the subgoal perturbation along with the definition of potential is new in GCHRL. I appreciate the combination of novelty and potential which properly takes novelty and reachability into consideration for an effective exploration selection.
- The experiments are extensive, well evaluating and demonstrating the characteristics of HESS across multiple perspectives.


&nbsp;


Weaknesses:
- I think the methods proposed in this paper are relatively simple and somewhat incremental, however, thanks to the solid experiments, the effectiveness of these methods are demonstrated. At a first glance, the representation regularization seems to be disconnected to the active exploration method. Later, I found that the stable representation learned is important to the effectiveness of novelty calculation. I recommend the authors to make the connection more obvious for a better convey of the story.
- Although I think the methods are reasonable in an overall view, I have a few concerns on concrete implementations. I list my questions and concerns below.


&nbsp;


My first concern is the calculation of novelty (Equation 4). I have no question on the maintenance of $n(\phi(s))$; but for the calculation of accumulated visit count of high-level state trajectory, I wonder given a state $s_i$, how the trajectory of policy $\pi_{hier}$ is obtained exactly? 

&nbsp;

Second, for Equation 5, since the potential is defined over the expectation of high-level transition obtained by $\pi_{hier}$ with the perturbed subgoal $g_e$, how are such transitions obtained?

&nbsp;


For both above two concerns, one possible way is to simulate the rollouts with a world model, but this seems not to be the way used in this paper. Alternatively, are these approximated with the trajectories in the replay buffer? If so, how should consider the off-policyness and suboptimality?

&nbsp;

The third question is on the computation complexity. The top $k%$ selection in representation regularization and the calculation in Equation 6 (the selection of candidates according to the constraints, the calculation of novelty and potentials). It seems the computation is heavy for these. What are the practical implementations?

&nbsp;

Besides, I have a few questions on the experiments.
- How to understand that some baseline algorithms work better in image-version environments? E.g., H-SR/H-ICM on Ant-Maze and LESSON on Ant Push.
- Is the sentence ‘so the intrinsic rewards of H-ICM may vanish before the policies have been learned well’ checked in the experiments?
- In Figure 6, are the same 5 trajectories used for the upper and lower panels at each time point? And what are the trajectories exactly, since at the beginning of learning, the agent fails to reach the final goals (according to the results in Figure 4)?


&nbsp;


Minors:
- Can the authors explain more on the sentence ‘to keep the low-level reward function $r^l$ stationary while learning $\phi$, we concatenate states and state embeddings together as the low-level states’ (above Equation 1)?

&nbsp;


I will raise my score if my questions and concerns mentioned above can be well addressed.

================================

Post-rebuttal comments:

Some of my concerns and questions are well addressed. I raised my evaluation to a borderline acceptation although my main concern on the relatively complex mechanisms involved in the implementation, e.g., hash table, iterative sampling and fitering, table look-up and so on (a few of these are sample-wise), remains. And I think these computation implementation should be noted and described in detail in the revision later. However, I recognize the authors' efforts in pushing the boundary of HRL.

summary_of_the_review: I vote for a borderline acceptation after discussing with the authors. I recognize the authors' efforts in pushing the boundary of HRL although I still have some concerns on the complexity of the proposed methods and the practical computation cost.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 1. This paper investigates learning stable subgoals within a deep hierarchical reinforcement learning setup. 

2. Two controllers are learned from the same experience replay buffer. The high level controller serves as a meta controller and the low level controller serves as a goal-achieving agent. The high level controller communicates abstract goals to the low level controllers. 

3. The high level controller is optimized using an extrinsically specified reward function. The low level controller optimizes the intrinsic goal communicated by the high level controller. The subgoals are changed after a deterministic time length (known option termination).

4. The subgoals are designed with the key insight that "desirable novel subgoals should be reachable and effectively guide
the agent to unexplored areas.". Typically count-based, predicted or successor feature based rewards have been used as novelty measures. However, these fall short in terms of reasoning about reachability of states. To handle this, a potential measure for subgoals is proposed which regularizes the novelty measure.

5. To go to unexplored states, a directional goal is synthesized/imagined using the current state and a directional vector. The potential function makes sure that this is approximately reachable by formulating reward as the expected negative distance between the ending state and imagined goal. This is similar to Feudal networks (Vezhnevets et al.) but goes beyond it to handle diversity and reachability. 

6. The approach is validated on a set of hard to explore continuous environments with reasonably strong and relevant baselines.
main_review: 1. 

I think this paper is interesting and explores a novel set of ideas. The baselines also seem reasonable. The closest baseline in terms of using directional goal vectors is Feudal Networks. I would have expected to see a head to head comparison with this approach, even though this proposed method goes beyond it. However, the core idea of having a meta controller output goal vectors and then sub-controllers learning to execute them was explored in Feudal networks. 

2. 

What are the effects of changing the option termination condition? Currently it is hard coded to be c. What are the implications of this? Do the authors observe any deviations or improvements if this hyper parameter varies. It seems like the potential function, novelty measure and option termination are deeply interlinked. It would have been good to more clearly understand the relationship between these measures.

3. 

Figure 4 is the main quantitative figure. It seems important to test the effects of stability regularization. This is highlighted qualitatively in Figure 6 but not shown in Figure 4. 

4.

The qualitative analysis on the effects of the interaction between potential and novelty measure is quite sparse. It is not clear how it fails and where it works. Figure 6 is helpful but it needs improvements in terms of clarity and scope (other environments)

5. 

Figure 5 is truncated at 5 million steps. How does the asymptotic performance look like for this method? Does it plateau sooner than baselines? What is the maximum achievable reward for these tasks? 
summary_of_the_review: This paper presents an interesting and novel idea at the intersection of deep HRL, novelty based exploration and reachability. The experiments are sound but could require further clarification and expansion of scope. The clarity of the paper can also be improved to more directly address the need and important of stability regularization.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a hierarchical RL algorithm which augments an existing contrastive learning-based subgoal representation objective with heuristics for exploration. The proposed algorithm seeks to reduce representation drift over the course of learning by penalizing the learner for modifying $\phi(s)$ for states $s$ with low contrastive loss. Furthermore, the authors propose exploration heuristics that encourages the learner to explore in promising areas of latent space by combining count-based novelty and potential measures. The proposed algorithm is demonstrated to have the desirable properties, and outperforms existing methods. The analysis is complemented by an ablative analysis that disentangles the effects of each proposed mechanism.
main_review: **Pros**
1. Comparisons between the proposed method and other hierarchical methods demonstrate that the algorithm results in better performance.
2. The authors performed thorough ablations demonstrating the impact of each proposed component of their algorithm.

**Cons**
1. The authors do not explain how the counts and potential measures are estimated from data. In particular:
    1. How are the cumulative counts $N(s)$ in (4) estimated, given $\pi_{hier}$ is changing over the course of training?
    2. How is $U(g_t)$ estimated from buffer data, given that the expectation is calculated with $g_e$ being set as a subgoal for the policy, and thus would not have been observed in the actual environment rollouts?
2. Why is prioritized sampling used in Equation 3? The motivation on this point was not really explained in detail.
3. For the ablative analysis, it seems like it would be better to evaluate reactive exploration using cumulative counts instead of immediate counts to better isolate the impact of reactive exploration versus learning a policy to maximize the same intrinsic rewards.

**Clarification Questions**
1. Why does choosing $\lambda(s)$ as a continuous function of the representation loss impose heavy computational demands? It seems like the losses are already being computed in the process of obtaining the triplets with minimal representation losses.
2. How is the latent space partitioned into cells if there are no knowns bounds on $\lVert \phi(s) \rVert$ a priori?
3. In motivating the potential measure, the authors claimed that the “novelty measure is a mixture of counts in the past and current representation spaces”, but it is unclear why this is the case if one can easily recompute $n$ when $\phi$ changes.
4. How is the low-level policy training done? Is hindsight experience replay used?
summary_of_the_review: Overall, I vote for a weak accept. The ideas in the paper are interesting, and the experimental evaluation is thorough and demonstrates the benefits of the proposed algorithm. However, the work could benefit from a more detailed description of how the relevant measures are estimated, as well minor changes to the experimental procedure.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new algorithm for goal-conditioned hierarchical reinforcement learning that is able to succeed in tasks with sparse rewards (differently from most other methods in the field). It does so through two innovations: (1) a representation learning procedure that is more stable, and (2) a exploration strategy that takes into consideration not only novelty but also reachability.

Specifically, the representation learning procedure is based on what is now a standard a contrastive loss, but it is augmented by a regularization term that make the learning procedure stable where the representation is already satisfactory, allowing goal sampling to be more effective. Figure 6 is a particularly nice visualization of the impact of this regularization term.

The exploration strategy to sample goals to be visited is also novel. Instead of using goal visitation counts, this paper proposes the idea of using expected sum of state visitation counts from that state onwards, capturing some notion of long term novelty. Moreover, the exploration bonus also has a potential term that captures how promising each goal state is in terms of how far from the goal state the agent is expected to end up. Quantitative impact is reported in Figure 7, but I particularly liked the intuitions/visualizations provided in Figure 8.
main_review: This paper is really well executed. It builds on top of an already complicated architecture adding more than one new component to that architecture, but it does so while providing proper intuitions for each one of these new components and, more importantly, actually doing ablation studies that quantify the impact of each component. To me, Section 5.4 is the highlight of the paper. I also appreciated Section 5.5, which shows how the paper is also concerned with stability over different parameters introduced by the proposed metric. I think the paper would benefit from further clarifying some parts of the text, but otherwise this is a good paper. Specifically:

* In the Introduction, for example, it is said that methods based on visit count are "_reactive_ exploration methods that need to learn how to maximize the intrinsic rewards before performing exploratory behaviors". I don't necessarily disagree with that, although the whole idea of visit counts is to incentivize these exploratory behaviors. My question though, is: isn't this exactly the same with the proposed idea? It does use counts and not only that, but also expected state visitation counts for the trajectory, which is even more demanding in terms of having to visit the state first.

* In Section 2, when defining $U(\phi(s_t))$, the distance is defined to be between $g_t$ and $\phi(s_t)$. For the proposed algorithm, should it be $g_e$ instead of $g_t$? Still on Section 2, it is said "we concatenate states and state embeddings together as the low-level states". What does this actually mean? What are the states here? For images, for example, would it literally be all pixels on the screen?

* In Section 3.1, I don't think $\lambda_0$.

* In Section 3.2, it is said "low-dimensional continuous latent space into discrete cells, i.e., the state embeddings are mapped into cells containing them.". What are these cells? How were they defined? I can imagine this is somewhat straightforward to do if you assume you have access to x,y positions, but how is this done in higher dimensional settings? How are these cells defined for Images, for example?

* In Section 3.2, when discussing the potential measure, it is said that Figure 3 demonstrates that "with online representation learning, the novelty measure is a mixture of counts in the past and current representation spaces, so it might mislead the exploration". How is that? I couldn't understand what I should be looking at in Figure 3 to reach this conclusion.

* In Section 3.3, it is said that the active exploration strategy "avoids the non-stationary issue". How? Aren't these reward signals changing constantly based on counts and the representation being learned? How does the active exploration strategy actually avoids the non-stationarity issue?

* In Section 4, it is said that "Bottom-up HRL works learn a set of diverse skills or options in a self-supervised manner, and use those semantically meaningful low-level skills to explore in downstream tasks", but "those methods may produce some redundant and useless skills". This claim is not backed up by any reference or experiment. Why is this true when some of these methods explicitly ask for diverse skills that are not supposed to overlap to each other?

* In Figure 4, how were the confidence intervals computed if only 5 samples were available?

* In Section 5.2 it is said "the successor representation estimates the expected future state occupancy starting from a given state (Kulkarni et al., 2016b), but not the visitation number of the given state, which is less helpful to promote exploration." However, isn't this exactly what H-SR shows? That the $\ell_1$ norm of the SR captures the visitation number of a given state? Moreover, the reference to the SR should be "Peter Dayan: Improving Generalization for Temporal Difference Learning: The Successor Representation. Neural Comput. 5(4): 613-624 (1993)". 

* No details were given on how Figure 6 was generated. I don't know how to reproduce it.

* *Importantly, in the ablations, were the parameters of the ablated methods tuned?*
 
summary_of_the_review: This paper is really well executed. It builds on top of an already complicated architecture adding more than one new component to that architecture, but it does so while providing proper intuitions for each one of these new components and, more importantly, actually doing ablation studies that quantify the impact of each component. To me, Section 5.4 is the highlight of the paper. I also appreciated Section 5.5, which shows how the paper is also concerned with stability over different parameters introduced by the proposed metric. I think the paper would benefit from further clarifying some parts of the text, but otherwise this is a good paper.
review_title: null

Meta-review: The paper proposes a new goal-conditioned hierarchical RL method aimed at improving performance on sparse reward tasks. Compared to prior work the novelty lies in a new way of improving the stability of goal representation learning and in an improved exploration strategy for proposing goals while taking reachability into account.

The paper does a good job of motivating the main ideas around stability and combining novelty with reachability. Reviewers found the quantitative evaluation and the choice of baselines to be good with the exception of not including Feudal Networks which the authors explained was due to poor performance on the hard exploration tasks (something that has been observed in prior work). Reviewers also found the thoroughness of the ablations and insightful visualizations to be highlights. Overall, reviewers were unanimous in recommending acceptance, which I support.
Decision: Accept (Poster)