Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary: This paper studies the hierarchical reinforcement learning (HRL) problem. It proposes a framework TAIC that learns temporal abstraction from past experience or expert demons without task-specific knowledge. The method is to formulate the problem by a temporal abstraction problem. That is, they assume that the action sequence is generated by a latent variable o. By regularizing the latent space by adding information-theoretic constraints, they are able to learn the representation. The paper later uses visualization to demonstrate the effectiveness of the learning. 

I would think this paper is slightly below the borderline. It is an interesting method of encoding the option sequence by a continuous variable. Therefore, the action space becomes continuous rather than discrete. However, I found it not convincing why continuous option space is better than discrete ones. It appears to me that the experiment section does not provide a comparison with previous discrete option based methods as well. 

Comments:
* 4th line of related work: Parr --> "Parr & Russel"
* Page 2, problem formulation: in beta(s,o), s is not defined. Maybe you can denote it as beta_o(.).
* It appears to be that the"option" is a sequence of actions? This can only happen in the deterministic environment. What will you do if applying pi does not give the same sequence of actions? For instance, from (s1,a1) -> (s2, a2), where s2 is generated from a random distribution, and a2 is based on s2.
* the paper is overlength 
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Summary:

This paper develops a method for learning a latent action representation based on prior experiences (and specifically, prior action sequences). Additionally, the paper proposes to regularize the learning of this representation using an information-theoretic constraint, yielding Temporal Abstraction with Information-theoretic Constraints (TIAC). Indeed, one promise of HRL is to allow for learning and decision making algorithms to take the long-term consequences of a decision into account when planning, exploring, assigning credit, or simply acting. The options framework (Sutton, Precup, and Singh; 1999) is a promising and well-studied toolkit for investigating these capacities of HRL. For this reason, the topic of the paper is well chosen: continuing to understand how options can benefit and accelerate RL in rich environments is an important direction for research. The idea at the core of the paper is new to my knowledge: learning an encoding of action sequences with a continuous latent representation. It could be a promising technique for HRL. Experiments are conducted to evaluate the effectiveness of the method in several environments, including a continuous gridworld, control tasks, and problems involving transfer learning. 

Verdict: Due to lack of clarity in describing the main methods, and missing comparison to any HRL/option baselines, I recommend rejection.

More Detail:

The paper is lacking clarity in its current form. I view the main contribution as the development of the architecture and loss function that together learn an appropriate latent action representation. There are two key issues with clarity at present: 1) The presentation of the core technical contributions could be improved (see comments below in "Q1"), and 2) Motivation for this style of option learning is missing, with evidence that the proposed method is in fact learning an appropriate thing.

Toward (1): I provide suggestions where clarity could be improved below in "Q1'.

Toward (2): There are a few aspects of the motivation that could be improved. First, the paper mentions that the learned options/representation will help in planning, but planning is not studied in the paper. For example: "Further, the interpolations between two sequences smoothly transfer from one to the other, which is a desired property to have during planning, because the smooth option space provides the RL algorithm with a better search space." By my reading of the paper, this claim is not studied. Similarly, in the intro: "...allow us to do planning at a higher level, and easily transfer the knowledge between different tasks". Including experiments that explicitly evaluate the capacity of the learned representation to carry out planning would help support these claims. Or, alternatively, the contribution could be focused to model-free and policy-based learning, which is where the empirical evidence currently offers the most support. Second, no HRL baselines are compared to in the experiments. One natural comparison to include would be to the Option-Critic, which was the first technique for combining option learning with deep RL. To determine whether TIAC is a sensible approach to learning and using options, a comparison to at least one other option learning method is needed. The paper currently highlights the fact that the option-critic requires a pre-specified number of options: this is true, but it is not discussed why is this problematic, or how the current proposal remedies this difficulty. Others that may be relevant include FuN (Vezhnevets et al. 2017), the recent methods of Nachum et al. (2018), among others (Tiwari and Thomas 2019, Harb et al. 2018, Harutyunyan et al. 2019, Levy et al. 2019).

In short: the results here are promising, so I encourage the authors continue in this direction. The paper will be improved if the presentation of Section 3 is sharpened (see questions regarding clarity below) and a comparison with relevant baselines is included. 

Main Questions:

Q1: The exposition of the main method (Section 3) was unclear to me. Here are a few questions I was left with:

	(a) Why is the posterior (on $o$) conditioned only on the action history, and not state?
	(b) Additionally $o$ is being treated as a random variable through 3.2. So, what is $o$? Where is the randomness coming from?
	(c) Section 3.3 states "it encodes the action sequences with respect to the L2 distance in action space". Does this mean the action space is always a subset of $\mathbb{R}$? But, it looks like $\mathcal{A}$ is just defined as some set: in Section 3.1, "$\mathcal{A}$ is the action set". So, I am confused as to what the $L_2$ is distance defined with respect to. If the actions are always assumed to be real numbers that is entirely okay, but it would be helpful to have that stated early on. From the additional text in Section 3.3, it sounds like the transition function of each action is involved in computing this distance ("...only have small difference in each step of action. Due to the error compounding, the two sequences..."). 
	(d) How is the estimate of the posterior actually used to act? The output of "D" in Figure 2 is $\hat{a_{0...k}}$. What is the type of this entity? Is it guaranteed to be an element of $\mathcal{A}$? If so, then the "option" here is a policy that maps $o$ and the action history $a_{0...k}$ to a new action, correct? Ah, so in Figure 3, it looks like D will have different output depending on how the termination condition is handled. Are the actions output by $D$ then executed by the RL agent, or is there some additional decision making that goes on downstream?
	(e) Early on the section states "In contrast to precisely reconstructing the action sequences, our goal is to extract the latent variable capturing  the information which could benefit RL training." It might be helpful to include some intuition about what this information would look like. It's unclear why action history would be all that meaningful on its own (without say, the state history). It would help the section to provide some intuition for such a latent variable existing; is there an idealized, simple case that would help convey the idea? Note that this proposal comes across as different from the original proposal of the options framework: As an example, Sutton, Precup, and Singh (1999) say: "options enable temporally abstract knowledge and action to be included in the reinforcement learning framework in a natural and general way". This temporally abstract knowledge need not be a function of the entire action history. I like this aspect of the method as it makes the proposed algorithm quite novel, but the motivation for why this should work didn't come through for me.
	(h) Should the mutual information in Eq. 4 be the conditional mutual information given $a_{0...k}$? (Same question for the remaining uses of $I$ and $H$).
	(i) It is unclear how the option learning coordinates with the RL algorithm used. That is, suppose we train the HRL component to learn the mapping from $s, a_{0...k}$ to the constituents identified in Figure 2/3. Where does the actual RL take place? Does the algorithm just execute the actions output by $D$ at each time step? 

Q2: In the first experiment, it is stated: "because the smooth option space provides the RL algorithm with a better search space." Any thoughts as to why this is true? Including some discussion here might help motivate the approach.


Minor Comments:

	C1: I do not understand Figure 6. The color is said to denote "the distribution of options", but I couldn't quite make out what this was, precisely. It would be helpful to know the range of values it can take on, and how those values map to the displayed colors. Moreover, what is the take away from the figure? The text states "with information-theoretic constraints the options and state changes become more correlated" but I am having trouble connecting that claim with the visuals themselves. Some additional discussion here would be really helpful.
	
	C2: In Figure 5, what does "dimension disturbance in option space" mean?

Minor Typos/Writing Suggestions [did not affect evaluation]:
	Abstract:
	- "Applying reinforcement learning (RL) to"::"Applying reinforcement learning (RL) algorithms to"
	- I am having trouble parsing this phrase: "to learn new tasks on higher level more efficiently". Perhaps: "to learn new tasks at a higher level of abstraction more efficiently"
	- "over benchmark learning problems"::"over baseline learning algorithms on benchmark problems"

	Sec. 1 (Intro):
	- Plural acronyms tend to have an 's' at the end. So: Recurrent Variational AutoEncoders (RVAEs).
	- "conveys meaningful information and benefit the RL training"::"conveys meaningful information and can benefit learning"

	Sec. 2 (Related Work):
	- "the policy sketches"::"policy sketches"
	- Personal preference, by I always prefer "use" to "utilize".

	Sec. 3 (Approach):
	- Your $\mapsto$ operators should be replaced by $\rightarrow$. The $\mapsto$ operator indicates what is applied to elements on the left, while $\rightarrow$ specifies the domain and codomain of the function. Thus, the $\mapsto$ variation would be $P : (s,a) \mapsto s'$. The story is the same for $\beta$: it should read "$\beta : \mathcal{S} \rightarrow [0,1]$". Note that this (using $\rightarrow$) is how Sutton, Precup, and Singh (1999) define $\beta$ as well.
	- "Sub-policy is defined as a function over the random variable."::"Now, the sub-policy is defined as a function over the random variable."
	- Not a sentence: "So that the options with similar consequences become closer in the option space." Consider combining with the previous sentence.
	- This sentence runs on: "Given a set of past experiences...". Consider defining $\Lambda$ first as its own sentence, then definines the problem. Something like: "We let $\Lambda = ...$ Then, our problem is to learn...".
	- "it is empirically shown"::"it has been demonstrated empirically"
	- Latex quote issue: "”go reach the door".
	- In Equations 4-9: in general, mutual information is a function of random variables. Is $o$ a random variable? For instance I have trouble expanding $H(o)$. What is $p(o)$?
	- "the encode $E$ is regularized"::"the encoder $E$ is regularized"

	Sec. 4 (Experiments):
	- "task for proof of the concept"::"task as a proof of concept"
	- "that allows us easily visualize the option we learned from the experience"::"that allows us to easily visualize the options learned from experience"
	- "that the RVAE nicely capturing the direction":::"that the RVAE captures the direction"
	- Misuse of $\mapsto$: "learn a control policy $\pi : \mathcal{S} \mapsto \mathcal{A}$" should be "learn a control policy $\pi : \mathcal{S} \rightarrow \mathcal{A}$" or "learn a control policy $\pi : s \mapsto a$".
	- "HarfCheetah"::"HalfCheetah"

References:

Vezhnevets, Alexander Sasha, et al. "Feudal networks for hierarchical reinforcement learning." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.

Nachum, Ofir, et al. "Data-efficient hierarchical reinforcement learning." Advances in Neural Information Processing Systems. 2018.

Tiwari, Saket, and Philip S. Thomas. "Natural option critic." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.

Harb, Jean, et al. "When waiting is not an option: Learning options with a deliberation cost." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.

Harutyunyan, Anna, et al. "The Termination Critic." AISTATS 2019

Levy, Andrew, et al. "Learning multi-level hierarchies with hindsight." ICLR 2019.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The authors propose a Hierarchical Reinforcement Learning (HRL) framework based on learning latent representations of action sequences. They use a Recurrent Variational Autoencoder (RVAE) to encode action sequences from previous experience or expert demonstration. They regularize representations using the fact that these representations should contain information about state changes, but not the states themselves.

The approach is developed both intuitively and theoretically. Detailed visualisations demonstrate that the results match the intuition. The paper is well written and relatively easy to follow. The related work section is wanting - see below.

Comments 

If we understood correctly, E, D, F, and P are pre-trained in an unsupervised way from expert demonstration as in imitation learning. We ask the authors to clarify this in the paper.

In Algorithm 1, we don't see how F is trained. Is this missing or not part of the algorithm at all? Also, in line 10, how is MSE calculated if i != j?

In the experimental section, experience is collected using a PPO agent. A flat policy is used as a baseline. Is the experience collection included in the number of interactions or just used to pre-train (parts of) the model? In the latter case, the comparison might be improper. 

Also, flat policy might be a weak baseline given recent progress on HRL. Comparison with other recent methods such as those in [1][2][3] would be desirable, but not a must.

Typos etc

Page 3, Section 3.3, instead of "however" I suggest "on the other hand" or similar. 
Page 4, Section 3.3, "summation of two conditional entropies" instead of "two conditional entropy". 
Page 9, Section 4.2.2, "noticed" instead of "notice".

Related work

We don't think this is the first time an RVAE has been used for encoding action sequences. SeCTAR [1] also uses an RVAE to encode trajectories (both states and actions) for HRL. The authors should include a reference to the paper and discuss similarities and differences between SeCTAR and their own work.

Other missing recent related works include HIRO [2] and Hierarchical Actor Critic [3].

They write: "the HRL often requires explicitly specifying task structures or sub-goals (Barto & Mahadevan,2003; Arulkumaran et al., 2017). How to learn those task structures or temporal abstractions automatically is still an active studying area." "Some early studies try to find sub-goals or critical states based on statistic methods (Hengst, 2002; Jonsson, 2006; Kheradmandian & Rahmati, 2009). More recent work seeks to learn the temporal abstraction with deep learning (Florensa et al., 2017; Tessler et al., 2017; Haarnoja et al., 2018a). However, many of these methods still require a predefined hierarchical policy structure (e.g. the number of sub-policies), or need some degree of task-specific knowledge (e.g. hand-crafted reward function)."

These are rather recent references. To our knowledge, however, the first HRL with temporal abstraction was published 1990-1991. See the references in section 10 of the overview http://people.idsia.ch/~juergen/deep-learning-miraculous-year-1990-1991.html  "Hierarchical RL (HRL) with end-to-end differentiable NN-based subgoal generators [HRL0], also with recurrent NNs that learn to generate sequences of subgoals [HRL1] [HRL2]. An RL machine gets extra inputs of the form (start, goal). An evaluator NN learns to predict the rewards/costs of going from start to goal. An (R)NN-based subgoal generator also sees (start, goal), and uses (copies of) the evaluator NN to learn by gradient descent a sequence of cost-minimising intermediate subgoals. The RL machine tries to use such subgoal sequences to achieve final goals." See also [HRL4] on another way of discovering appropriate subgoals. How does the work of the authors go beyond this original work on learning temporal abstractions for HRL? 


Additional References mentioned above: 

[1] John Co-Reyes, Yu Xuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings. ICML 2018.
[2] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-Efficient Hierarchical Reinforcement Learning. NeurIPS 2018.
[3] Andrew Levy, George Konidaris, Robert Platt, Kate Saenko. Learning Multi-Level Hierarchies with Hindsight. ICLR 2019.

Overall, we believe this is a promising paper, but we are not sure if it is ripe for publication at ICLR in its current state. For now, we'd lean towards rejecting this submission, but we might change our minds, provided the comments above were addressed in a satisfactory way. Let us wait for the rebuttal.
review_title: Official Blind Review #2

Meta-review: This paper presents a novel hierarchical reinforcement learning framework, based on learning temporal abstractions from past experience or expert demonstrations using recurrent variational autoencoders and regularising the representations.

This is certainly an interesting line of work, but there were two primary areas of concern in the reviews: the clarity of details of the approach, and the lack of comparison to baselines. While the former issue was largely dealt with in the rebuttals, the latter remained an issue for all reviewers.

For this reason, I recommend rejection of the paper in its current form.
Decision: Reject