Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### Summary
This paper proposes to use slow features as the subgoal representation space in the HRL setting. The proposed approach adopts a slowness objective to learn a representation (high-level action space) and use it to train goal-conditioned policies (low-level).

### Main contributions
- Slowness is an interesting property for representation learning. This work's approach to slowness is simple and easy to integrate (in practice) to the challenging setting of HRL.
- This work also makes a theoretical effort (section 4) to motivate the slowness property for subgoal space.
- The proposed method is evaluated on continuous control tasks against other references approaches to HRL. 

### Main comments/concerns
- In the definition (def.1) of the measure of exploration, what justifies the choice of the reverse KL ? Is it still a good choice if q is multimodal ? Does this definition really align with your claim "a larger coverage area leads to better exploration" ?
- The algorithm implementation is not clear:
    - How is the goal sampled ? Randomly in the large domain as mentioned in the Appendix or in the neighbourhood of s as mentioned in sections 3 and 4 ?
    - What is the extrinsic reward used to train the high level policy exactly ?
- There is a chicken-and-egg problem regarding the parallel learning of the representation and its usage to train the low policies. The representation should first be trained on some area before trusting the induced training of the low policy, which requires exploring that area beforehand. It is not clear how the algorithm deals with this challenge. Moreover, the evaluation on Mujoco tasks could be misleading regarding this point (see the following comment).
- The method is evaluated on Mujoco environments that provide the xy-coordinates. The representation can learn to extract the coordinates by only training on a limited area (e.g. neighbourhood of initial state) -- being able to early stop the representation training seems to support this hypothesis. Was this phenomenon studied is your experiments ? How is it limiting the proposed method beyond this type of environments ? How does the proposed method work when this inductive bias can't be leveraged, and when exploring an area of the state space is required to build a useful representation of it ?
- Figure 4 shows the learned representation. Was it learned from the states or the images setting ? If learned from states, how does it look like when learned from images ? And, related to the previous comment, how does it look like along the training ? 
- How was the oracle trained ? It would be interesting to see the oracle training curves along with the compared methods on Figure 3.
- How reliable/relevant is the slowness evaluation (section 6.3) in comparing the different methods ? It seems that the slowness objective can learn arbitrarily slow features (according to your measure) by weighting the attraction term more than the repulsive one (tuning m and c can also influence the slowness measure).
- "As the state space of the Point robot is simple, the dynamics of the randomly selected features are slow as well": What do you mean by simple state space ?
- "the high-level policy guides the agent to jump out of the local optimum of moving towards the goal" : local optimum w.r.t which reward function ?
- "The transfer effect [...] is more significant [...] since the target task, Ant FourRooms, is of a larger size and more difficult": Is the transfer effect confirmed when transferring from AntPush and AntFall to the larger Ant FourRooms ? 

### Minor comments
- The AntPush performance curves are limited to 2.5 and 2 million steps. How was this horizon decided for this environment ? How do these curves look like for a longer training (4 million steps) ?
-  Equation 7 seems confusing: p(x) is the asymptotic distribution of Y_n, meaning that it should not depend on n as shown in equation 7.
- Why does Figure 5(a) has a 4 millions step while 5(b) has 8 million steps ?

Clarity: The paper reads well, but the experiments and algorithm presentation is sometimes unclear if not confusing.
review_title: This work would benefit from a more rigorous investigation and clearer discussion of the main claims. Interesting theoretical grounding.

final_score: rating: 7: Good paper, accept
review_content: This work presents a method for learning a slow-changing ("low-frequency") embedding function, which can be used as a state-abstraction function in the context of Hierarchical RL. A high-level policy, trained to solve the environments task, acts by selecting abstracted states as targets for the low-level policy which is trained as in a goal-conditioned fashion (acting in the environment itself attempting to get to the set goal).

The paper provides theoretical motivation for the focus on "slow" features in a simplified synthetic example. The method itself is evaluated on high-dim control tasks (As well as on an illustrative toy-example).

There is a lot to like about this paper. The method is elegant, building on and extending previous approaches to HRL and particularly learning subgoal representations. The use of a low-frequency (slowness) criterion for this purpose is, to the best of my knowledge, a novel contribution. The work presents strong empirical evidence for the usefulness of the approach. The paper itself is well-structured and overall well-written, easy to follow, and with good references.

I do have a few questions for which I would appreciate the authors comments:
* In principle, the loss for the embedding function (highly) depends on the behavioral policy. This is because the embedding is trained to represent "far away" states as different from each other. And, generally, the distribution of $s_t$ and $s_{t+c}$ can be rather different for, say, a random policy compared to a more purposeful/trained one. Since everything is trained jointly here (embedding is used to train the policies which in turn can change the embedding and so on), this could in principle leads to instability of the training. I think this point deserve at least some discussion.

* I find section 4 a bit confusing. "Fact 1" seems more like a definition for me, because the properties of the "random walk" itself will be *determined* by $p$ (as $p$ obviously depend on the behavioral policy. In fact from "Fact 1" we can simply say that $p$ is the steady-state/limiting/stationary distribution of the markov chain induced by the policy and the MDP). In general the relation between $p$ and the policy is not entirely clear. Note that in general, the change metric $\Delta s^i_t$ is also policy dependent. I also don't see the direct relation between $p$ in this discussion, which ultimately is an expression of the (low-level) policy, to the embedding or representation principle used throughout the paper.

* It seems the underlying assumption here is something like: the agent can independently control/change the state-features in an unconstrained way (i.e an "action" is an offset vector s.t the next state is $s_{t+1}=s_t+a_t$, but in such a way that the environment somehow 'rescales' $a_t$ so that earlier entries are smaller). If this is indeed the case, I find this section not too motivating. If there is completely no structure in the environment (state/action spaces) and everything reduced down to a uniform random-walk, we cannot learn much about even simple "realistic" RL settings. 

* Having said that I think the paper presents other good reasons and motivations for the focus on slow-features. In particular I think that the example discussed in section 6.1 is important, as well as the more qualitative explanations and the relation to the unsupervised-learning literature around similar ideas.

review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The main idea of this paper is very nice:  that we want the features in the representation space for subgoals in HRL to be "slow" in the sense that they don't change much over primitive steps of the policy, but do change significantly over "macro" steps.  The margin-based criterion in the loss (that says we want them to change by at least m at the high level) seems well justified.   

It seems like, additionally, one might want to incentivize this representation to cover the state space s well (that is, not map big chunks of s onto the same latent representation phi) and I didn't exactly see how this would be encouraged---I guess the drive for significant change at the high level accomplishes this implicitly.

Using this loss function leads to a sensible algorithm, although I was curious about why the representation learning happens at the same rate as the low-level policy learning;  I could imagine that you might want to do that at a slower time scale (more like the time scale for the high-level policy learning).

The experimental results seem to make a compelling case for the effectiveness of this algorithm.  The results for transfer learning are particularly strong---in fact transfer seems to be one of the best motivations for learning hierarchical representations, which might be harder to get to pay off in the case of learning to control a single problem instance.  Some of the explanation in this section didn't completely make sense to me:
- Why do you attribute the success of your method to improved exploration?  It seems like another very plausible explanation is that you are providing a good inductive bias for learning a policy that generalizes well (though 2 million is a lot of steps!)
- (Related) What is the purpose of the experiments on state coverage?  I don't completely see why that should be, in itself, a goal.  In fact, one attribute of good RL methods that generalize well is that they do early exploration but then focus on parts of the state space that are profitable given the particular high-level reward function they are optimizing.

All this being good, the theoretical part of the paper was quite weak.    The definitions and theorem were not rigorous and possibly incorrect.   First, from the expositional perspective, it would have helped to begin with crisp definitions and then the assumptions.  Here are many points that came up for me as I read through the text:
- What is the role of the target distribution?  Where would you get it?
- In the application of KL divergence It seems more like you'd want the p to be the target distribution.
- What is a "fact"?    Break it down into definitions and then a crisp formal assertion with a proof (if there's something in there beyond definitions).
- In Fact 1 the term "exploration process" is not defined.
- Importantly, I don't think this is well-formed unless we assume that there is no learning happening at the low level. Are we assuming that?  (If not, then the process is non-stationary).  
- The high-level policy is assumed to select subgoals "randomly" :  Does this mean uniformly at random from S?   Is S bounded?
- What does it mean for the agent to be able to "move independently and identically in the state space"?     
- What the X^c_t are like depends completely on the low-level policy!  It could always move left, in which case these variables would not be anything like IID.  Is this analysis supposed to be assuming that the low level is perfectly able to achieve the targets?  Or at least always successfully move c steps in the target "direction" (not sure what that means, exactly, though).
- What, precisely, do you mean by "the features are all independent"?
- Brushing assumption (a) off as a general technique to simplify analysis is not so good.  It's okay to say you're making it for now, but there's no reason to think results obtained with this assumption will generalize!
- Assumption d would require changes to the algorithm, no?
- In Theorem 1:  "r is large enough"  for what?
- I don't really understand what it means to "assume q is an isotropic Gaussian ..."  ;  that is, what exactly is q intended to be.   I see that later you say that if r is big enough it approximates a uniform, and so maybe what you're trying to show is that our exploration is uniform.  But a uniform needs to be over a bounded space.  And even with large r, q is only sort of uniform in a region bounded around the mean.  
- What makes "optimal hierarchical exploration"?   I guess maybe this means:  with a fixed set of features and low-level control policy we're deciding which k features to use for subgoals, and showing that among those choices we get the best exploration (in the sense of matching a Gaussian around the origin) if we use the k slowest features.

(This is just a related thought, not really a review of this paper.  One other principle for selecting features for a subgoal representation is that they should be, in some sense, locally achievable.  Hierarchy works well when the subproblems are, in a sense, "serializable":  that we can achieve subgoal the first any way we want to, without thinking about (and/or making it harder to achieve) the next subgoal we're going to be asked to do.    One way to do this might be to encourage some "disentanglement" in the latent features, so that the policy for changing one dimension of the latent space tends not to change the other dimensions.)

In the end, I'm positively inclined toward this paper because it made me think about the concepts of what we really want in a hierarchical representation, but the mathematical exposition really needs substantial cleaning up.  But I'd rather have an interesting paper than a perfectly-executed boring one.

review_title: An interesting and apparently useful objective function for subgoal representation learning in hierarchical RL

final_score: rating: 7: Good paper, accept
review_content: **Summary:**
This paper proposes a new method for learning subgoal representations in HRL. The method learns a representation that emphasises features that change slowly, through a “slowness objective”. The slowness objective minimises changes in the subgoal representation between low level time steps, while maximising feature changes between the high-level temporal intervals. This objective allows for efficient exploration, which the paper justifies theoretically, and supports with some empirical experiments on challenging control domains.

**Strengths:**
The issue of subgoal selection is a critical issue for HRL, and constructing or learning a good subgoal representation on which to create subgoals is important and interrelated. Thus, any significant progress in this area is useful. The paper presents an apparently novel method that would be of interest to HRL researchers and deep RL practitioners more generally.

The paper is clearly written and the main ideas are generally very well explained (except for maybe the use of the term “state”; see weaknesses). 

The paper provides some theoretical justification for the slowness objective, which is useful to support the intuition behind it. The empirical results are conducted on challenging domains and they appear strong. However, including more independent runs in each domain would strengthen the conclusions significantly (see weaknesses).

**Weaknesses:**
In sections 3.1 and 4.1 the paper uses the term "state" when describing the algorithm and other definitions. It is not clear if it is actually referring to a proper state in the Markov sense, or some approximation of a state, or even just an observation that might not be Markov at all. This is confusing, and could be clarified. How does the slowness objective interact with approximation when you do not have access to the true state (which would be most of the time)? Does the theoretical result still hold?

I think 10 runs for the NChain environment, and 5 run for Mujoco are too few. More runs would give a much better sense of the variability of the runs, greatly strengthen the conclusions about the relative performance between the algorithms. 20 runs for the NChain environment, and 10 runs for Mujoco would be a significant improvement, but even more is better.

Also, in the low run regime (~10 runs or less) sometimes it can be more informative just to plot each of the learning curves for all the runs on the same plot, along with the mean. This gives a really good sense of the variability between runs. Presenting the results this way assumes less about the distribution that the performance samples are coming from.

**Recommendation:**
Overall, I recommend to accept this paper. A clarification about the use of the term “state” and adding more runs to the experiments would increase my score.

**Questions:**
Clarify the use of the term "state" (see weaknesses).

**After Author Response and Discussion:** Thanks to the authors for their responses. After reading the other reviews and the author responses, I am raising my score to 7 (accept).
review_title: A new objective for learning subgoal representations

Meta-review: This paper introduces an HRL method that uses slow features to define subgoals (or abstract states), which can then be used by goal-conditioned policies. It is said that such an approach allows for efficient exploration. Most reviewers are recommending the acceptance of this paper, they found the method interesting and they think it introduces interesting ideas that are not that common to the HRL literature. Thus, I’m recommending the acceptance of this paper.

I’d still encourage the authors to take the reviewers comments into consideration when preparing the final version of the paper. Specifically, it would be useful to explicitly discuss the “chicken and egg problem” and the fact that the agent has access to a function defining the distance to the goal before the goal was observed for the first time. Some baselines have the same assumption, but it is somewhat weird to discuss exploration in this setting without further clarifications.


Decision: Accept (Poster)