Learning Temporal Abstraction with Information-theoretic Constraints for Hierarchical Reinforcement Learning
Authors: Wenshan Wang
Yaoyu Hu
Sebastian Scherer
Venue:  (2020)
Keywords: hierarchical reinforcement learning
temporal abstraction

Abstract: Applying reinforcement learning (RL) to real-world problems will require reasoning about action-reward correlation over long time horizons. Hierarchical reinforcement learning (HRL) methods handle this by dividing the task into hierarchies, often with hand-tuned network structure or pre-defined subgoals. We propose a novel HRL framework TAIC, which learns the temporal abstraction from past experience or expert demonstrations without task-specific knowledge. We formulate the temporal abstraction problem as learning latent representations of action sequences and present a novel approach of regularizing the latent space by adding information-theoretic constraints. Specifically, we maximize the mutual information between the latent variables and the state changes.
A visualization of the latent space demonstrates that our algorithm learns an effective abstraction of the long action sequences. The learned abstraction allows us to learn new tasks on higher level more efficiently. We convey a significant speedup in convergence over benchmark learning problems. These results demonstrate that learning temporal abstractions is an effective technique in increasing the convergence rate and sample efficiency of RL algorithms.

Under review as a conference paper at ICLR 2020
LEARNING TEMPORAL
ABSTRACTION INFORMATION-THEOREIC CONSTRAINTS FOR HI- ERARCHICAL REINFORCEMENT LEARNING
WITH
Anonymous authors Paper under double-blind review
ABSTRACT
Applying reinforcement learning (RL) to real-world problems will require rea- soning about action-reward correlation over long time horizons. Hierarchical re- inforcement learning (HRL) methods handle this by dividing the task into hierar- chies, often with band-tuned network structure or pre-defined subgoals. We pro- pose a novel HRL framework TAIC, which Iearns the temporal abstraction from past experience formulate the temporal abstraction problem as learning latent representations action sequences and present a novel approach of regularizing the latent space by adding information-theored information between the latent variables and the state changes. A visualization of the latent space demonstrates that our algorithm learns an effective abstraction of the long action sequences. The learned abstraction allows on higher level more efficiently. We convey a significant speedup in convergence over benchmark learning problems. These results demonstrate that learning tem- poral abstractions is an effective technique in increasing the convergence rate and sample efficiency of RL algorithms.
expert demonstrations without task-specific knowledge. We
constraints. Specifically, we maximize the mutual
learn new tasks
INTRODUCTION
Reinforcement learning (RL) has been successfully applied to many different tasks (Mnih et al., 2015 Zhu et al. 2017). However, applying it to real-world tasks remains a challenging problem, mainly due to the large search space and sparse reward signals. In order to solve this, many research efforts have been focused on the hicrarchical reinforcement learning (HRL), which decomposes an RL problem into sub-goals. By solving the sub-goals, low-level actions composed into high-level temporal abstractions. In this way, the size of the searching space is decreased exponentially. How- the HRL often requires explicitly specifying task structures sub-goals (Barto & Mahadevan, 2003 Arulkumaran et al., 2017). How to learn those task structures or temporal abstractions auto- matically is still active studying area.
Many different strategies are proposed for automatically discovering the task hierarchy or learning the temporal abstraction. Some early studies try to find sub-goals critical states based on statistic methods (Hengst 2002, Jonsson, 2006, Kheradmandian & Rahmati (009). More recent work seeks learn the temporal abstraction with deep learning (Florensa et al. [017] Tessler al.) 2017 Haarnoja et al., 2018a). However, many of these methods still require predefined hierarchical policy structure (e.g. the number of sub-policies), or need some degree of task spec-fpe knowledge (e.g. hand-crafted reward function).
We present a general HRL framework TAIC (Temporal Abstraction with Eeformation-theoree Con- straints), which allows agent to learn the temporal abstraction from past experiences or expert demonstrations without task-specific knowledge. Built upon the ideas of options framework (Sutton et al. 1999) and motor skills (Lin) 1993). we formulate the temporal abstraction problem learn- ing a latent representation of action sequences. In order to obtain good latent representations, we propose novel approach to regularize the latent space by using information-thhoreete constraints. The learned abstract representations of action sequences (we called options) allow us to do RL at a higher level, and easily transfer the knowledge between different tasks.

---

Under review as a conference paper at ICLR 2020
Our contributions are: 1) We formulate the temporal abstraction problem learning latent rep- resentation of action sequences. Motivated by works using Recurrent Variational AutoEncss (RVAE) to model sequential data in neural language processing (NLP) and other areas (Bowman et al., 2015 Ha & Eck] 2017), we employ RVAE to perform temporal abstraction in RI. 2) We propose regularization approach on the option space. It constrains the option to encode more in- formation about its consequence (how the option changes the states). We present both theoretical derivations and practical solutions. 3) We show in the experiments that our learned temporal ab- straction conveys meaningful information and benefit the RL training. In addition, the proposed framework provides an efficient tool for transferring knowledge between tasks.
2 RELATED WORK
HRL is long-standing study topic. Schmidhuberr (1990) proposed HRL with temporal abstraction in early 1990s. [Schmidhuber (1991) proposed system in which recurrent NNs generate sequences of sub-goals, evaluator NN predicts the rewards of going from start to goal and a RL machine tries to use such sub-goal sequences to achieve final goals (Schmidhuber & Wahnsiedler 1993). The options framework is a popular formulation for considering the problem with a two-level hicrar- chy (Sutton et al. 1999). A sequence of primitive actions converted into options. The traditional Markov Decision Process (MDP) problem is extended into semi-MDP with the use of options. Parr developed approach called Hierarchies of Abstract Machines to calculate hierarchically structured MDP policies (Parr& & Russell 1998). Dietterich proposed MAXQ method, which performs value function decomposition over a given task structure (Dietterich) 2000). These early work assume the task hierarchy is predefined by human experts.
For the automatic task decomposition problem, many methods try to find sub- or critical states based on statistic methods (Hengst 2002 Jonsson| 2006 Kheradmandian & Rahmati 2009). These methods cannot handle continuous control problem. More recent work seeks learn the temporal abstraction with deep learning (Florensa et al. 2017| Tessler et al. 2017: Vezhnevets et al. 2017
Haarnoja et al. 2018a] Nachum et al.J (2018). Frans al. (2017) developed a two-layer hierarchy of policies, including one meta-policy and several primitive policies. [Anrress et al. 2017) defined the policy sketches, which annotate tasks with sequences of sub-tasks, and learn the sub-tasks and upper-level tasks jointly. Kulkarni et al. (2016) presented hierarchical-Dor DQN, which integrates hi- erarchical action-valus functions with intrinsic reward based on predefined sub- uub-goals. |Bacon et al.) (017)) combined the options framework with policy gradient and proposed option-critic archi- tecture. These methods either require a predefined hierarchical policy structure (e.g. the number of sub-policies) need to specify sub-goals. We propose framework that allows learning temporal abstraction without predefined sub-goals.
The idea of learning latent representation for HRL has been proposed before (Eysenbach et al. 2018 Hausman et al. 2018), but they did not learn from action sequences. The SeCTAR algorithm learns latent representation from from state sequences in trajectories using RVAE RCo-Reyes al., 2018). Macro-action methods share the similar idea of combining sequences of actions into op- tions (Hauskrecht et al. 1998, Vezhnevets et al. 2016). However, existing literature only combines primitive actions, while our work learns an abstraction, which we will show later is more beneficial for HRL training. Our work is related to Fabius & van Amersfoort (2014); [owmman et al., (2015); Ha & Eck (2017), since they also utilize RVAE to encode sequential data. In this paper, apply RVAE to model the abstraction action sequences.
APPROACH
3.1 PROBLEM FORMULATION
We consider a MDP problem M (S,A,,,R.,) where is the set of states, A is the action set, P (8, A) + S is the transition model, R (S,A) + R is the reward function, 7 is the discount factor. Sutton et al. (1999) presents the option framework, where an option is formulated S is the initiation set, æ is the sub-policy and 3 S + 00. 1) is the
as tuple (I, 3) in which termination condition. The option framework and its followers (Bacon et al., 2017 Kulkarni et al. 2016) either manually designed or learned a finite set of options. The number of option becomes
2

---

Under review as a conference paper at ICLR 2020
hyperparameter, and each option in the set is usually independent with one another. We propose the TAIC framework, in which we model the option o as a continuous random variable, which denotes the latent representation of sequence of actions. Furthermore, following the option framework define (I(o), (o), 3(s, o)) as the function of the random variable o. I(o) is the initiation condition, which is assumed to be the entire state space S for simplification. We define the function =()) as
the sub-policy that maps the latent variable o to a sequence of actions. The termination condition B(s, o) controls the temporal length of the option, and will be detailed later.
This modification of the option definition brings important difference. In the original option frame- work, each option represents one sub-policy. Above those sub-policies, there meta- meta-policy xq that acts as a classifier, choosing one of the sub-policies at a longer time horizon. In contrast, the TAIC framework specifies the option o as continuous random variable, which could have an infinite number of values. Sub-policy is defined Te outputs directly the random variable o. This changes the framework from discrete options to continuous options, and allows us to put constraints on the option space. So that the options with similar consequences become closer in the option space.
function over the random variable. Our meta-policy
Given a set of past experiences A tory {so: dg, To. $1, 01, T1,,1, Sk, ay: rkh. which comes from the agent's past experience or ex- pert demonstration, (I(o), T()), ((s, o)), so that the o should be able to apply to the RI. task, and improve the train- ing efficiency.
(To, TI...., Tm. where
A is one trajec-
problem is to learn a temporal abstraction o, and the corresponding
3.2 TEMPORAL ABSTRACTION AS LEARNING LATENT REPRESENTATION
We consider the problem of learning latent representations from action sequences {ao....u Ao...k....... To model the sequential data with variable length, we use the recurrent auto- encoder (AE), which has been used intensively in natural language processing (NLP) (Bowman) jt al., 2013) and other sequential modeling tasks (Ha Eck (017). Specifically, we deploy recur- rent variational auto-encoders (RVAE) (Kingma & Welling, 2013; Fabius & van Amersfoort 2014)). because empirically shown to be better at finding hidden features of the inputs (Chung et al. 2015).
In general, we would like to calculate the posterior p(olao...x) where the option o captures the intrinsic features of the action sequences. The RVAE solves this by approximating the true posterior by (oloo....)) and then optimizing a lower bound on the log-likelihood (Fabius & van Amersfoort 2014). The log- ligelekel of action sequence do.... can be written
logp(a....k) = K [ol)gollo...))ll(llaaa..
+ L(o)
where (Llp))) is the Kullback- diverrenne, and the C(o) is the evidence lower bound:
((0) - K (((o(ooo....))ll(()) + (oog
In order find a good approximation of p(olag....) with g(olao...k), namely minimize K
we need to maximize the evidence lower bound L(o). The condi- tional distributions (oooo.....) and p(ao...*<o) lo) are approximated by two networks, recurrent encoder E : ap...k + and a recurrent decoder D : o + ao...k Following the common VAE formulation, we model (()) with the Gaussian distribution, and the training objective is to minimize reconstruction loss and KI. loss (Kingma & Welling 2013):
Lyvac LKL + LRecons
Specifically, the encoder E takes in an action sequence ag...k. and outputs the option represented by two vectors Oμ and 0g, namely the mean and standard deviation in the Gaussian distribution. The LxL. is the KI. divergence between this output and the prior Gaussian distribution (/ 1). The decoder D) takes in the sample of option o, and outputs āo..k The T.Recons is defined as the mean-square-error between do... k and ā....k
3.3 INFORMATION TONTAATTTTTTTTTCCCCTTTTTTTTT
CONSTRAINTS IN THE LATENT SPACE
The reconstruction loss in the RVAE setting suggests it encodes the action sequences with respect
to the 1.2 distance in action space. Figure shows a simple example in 2D navigation task. Two
3

---

Under review as a conference paper at ICLR 2020
Option space
$2
$0
(1)
O...3
Figure I: Using RVAE, action sequences are encoded w.r.t the L. distance action space. As a (3) have exactly the same consequence, their codes will be far away in the option space. This is not desired property.
result, although 0
(1) and do...3
(2) action sequences "o...3 and do...3 are with the same length, and only have small difference in each step of action. Due the error compounding, the two sequences end up very different resulting states 81 and 82. However, do...3 and dy... have big L. distance in action space, but they have the same consequence (both move from state 5o to 81). In favor of upper-level task solver, they should be encoded closely together in the option space. Imaging the navigation task in real life, in order to reach the door you could follow different paths (avoiding some dynamic obstacles). Those different sequences all have the same coding in your brain "go reach the door".
In contrast to precisely reconstructing the action sequences, our goal is to extract the latent variable capturing the information which could benefit RL training. Intuitively speaking, the option should encode the consequence of action sequence, namely how the sequence changes the state. Formally, we maximize the mutual information between the option and state changes. We decompose it into two terms. The first term is to maximize the mutual information between the option and two states (states before and after sequence of actions):
max maxx(oo;s.')
(4)
On the other hand, the option should encode the state change, and decoupled from particular start and end states. For example, navigating to the door from this room should be represented similarly as navigating the door at another room. This is formulated as minimizing the mutual information between the option and the individual start and end state:
min[[(o;s) + s()
(5)
Following the definition of the mutual information and the chain rule of the conditional entropy, Equation 4 is transformed into minimizing the summation of two conditional entropy:
max / (0; s, 5') = max (o) ((os,,s')
o
max (()) H(0,8, ((),
max(HH() (H(''ls,)) - Hs|o)) + H())))
(6)
-H(so))
min[H(s'|s s. 0) (s(o))
Similarly, Equation is equivalent to maximizing the conditional entropy of s/s' given H(s)o)) - ( H(s())))
(on( + ((o,5')} = min(((H
-max (s)o)) + H(s'lo))
Notice Equation and Equation Cenaaioonncotioon conflicts with each other, which is also very intuitive. Forcing the option to be irrelevant the start and end state would make harder encode the information of state changes. This is a trade-off depending on how much you want the code to be state-independent. we obtain our final constraints:
Combining Equation 6 and Equation] 7
min H('sss, ss. o) = max Ellog p(s p(s'ls,
(8)
max(II(sso) + HHs())) ([[[[log + E[logp(s'|o)}
(9)
4

---

Under review as a conference paper at ICLR 2020
Oc+
Rst Reward
Option
Environment
Encoder
Simpling
Decoder
Opton
ac+k
dd
O++
Gr++k
Estimated state 8y/85+k
Siate Si
RL
Advarserii paring
Sampling
State 8t
Estimated state S1+46
Decoder D (wwed)
Phase 1: Learning the option
b) Phass 2: Use opton HRL
Figure 2: System architecture. E, D. P and RL are implemented with neural networks. The gradients of three losses Lyvaes Lade and pred will be back-propagated through the encoder E, and regularize the option. Note the gradient of Ladv will change sign before back-propagating to E.
Those two constraints are also approximated with neural networks. For Equation g we utilize a predictor network P (s, o) + s', which predicts the end state s' given the start state and the option o. To optimize the constraint, we minimize the predictive loss Lyred. and backpropagate the gradients into the encoder E. In order to minimize the bog-liielihood in constraint Equation 9 borrow the technology from the adversarial training (Ganin et al., 2016; Shrivastava et al.) 2017). We utilize another network F given the option code. The encoder E' and the state estimator J' are trained competitively with each other. F acts like discriminator, trained in minimization of the loss of estimating the states Lodes while E is trained with the opposite gradients that try to maximize the loss Ludo- Thus, E will be pushed encode option in a way that F is not able recover the start and end states. this way, the encode E is regularized by these extra gradients. In practise, all of these networks are trained jointly on the experience set A (To, TI, Tm ). Figure 2 shows, we have 4 networks (F, D, F and P) collaborating and competing (note RT. is not updated in the option learning process). The learned latent representation o a temporal abstraction that captures the execution consequence of the action sequences.
(s, 8'), which tries recover the start state and the end state
3.4 TERMINATION CONDITION
After learning the option, we can train the agent at a higher level. The HRL policy outputs an option, which is then decoded sequence of actions by the decoder D. In order to apply the learned option to HRL training, we need to consider the termination condition ((s, o). We implement and compare three different methods: Fix-length, Term-output and Term-predict.
3.4.1 OPTION WITH FIXED LENGTH
In the simplest setting, we learn the option from the action sequence with fixed length N. The decoder D terminates the output after N actions, and then the HRL policy will output another option. This termination condition (referred as Fix-len later) does not depend on the option and state.
This straightforward method is easy to implement. We will show in the experiment section that for most tasks, even this naive implementation would bring benefit the HRL training.
3.4.2 OPTION WITH TERMINATION OUTPUT
In order to learn the option code of arbitrary sequence length, add another output the decoder D. At each time step f, D outputs an action a, and termination signal et. which determines whether this is the last action in the sequence (Figure 3 (a)). The termination output is 2-class classifier, trained with supervised cross-entropy loss. Note that this termination condition (referred to as Tutp-o is only dependent the option.
Allowing the RVAE to encode various lengths of sequence provides the HRL with more choices. For example, in states that stable and safe, the HRL can choose longer sequences; while in unstable

---

Under review as a conference paper at ICLR 2020
Option
Termination condition
Ersodw
Estimated
G+
Option
Option
0g+++1
State
Pt
State
Termnnte ener sbove
Decoder RNN
Decoder
Deaader
state
Ter-ooutptt
b) Term premppredctt
Figure 3: Termination conditions. a) The decoder D outputs termination signal C2 each time step t. b) The option terminates when the predictor P fails to predict the next state.
Algorithm Train option with unexpectedness condition
imput:Traecctee т; (8o, oo, 81, 01, -. 8t, a& ), networks F,, D, P
Parameter: Prediction threshold
Output: Option o
1: Let 0, Liredd
2: while Ixpred < 8 do
3: 0, = (ao....)
4:
8141 P(so. 0.)
5: Ipprdd == MSS (SE(( 8141)
6:
backpropagate Lxpred
7:
i:=i =+11
8: end while
9: (o...(1-1) = 1(0
10: LRecoms = ( (1-1), Co..
11: LKL -
12: backpropagate Riecos AKL LKL
13: return 0,-1
states, it can operate cautiously with shorter sequences. However, this termination condition is not dependent on the state, so cannot respond sudden changes in the state.
3.4.3 TERMINATION BY UNEXPECTEDNESS
In this setting, we utilize the predictor network to decide the length of action sequences (Figure (b)). The intuition is that the predictor network P also acts world model (Ha & Schmidhuber, 2018), which models the environment dynamics. When things are within expectation, the decoder D can go ahead and output longer sequence. When the agent encounters unfamiliar states, D should become more cautious and output shorter sequence. This termination condition (referred to as Term- predict) depends both the option and the state.
During training, we set a threshold on the prediction loss Lpred- When Lpred is bigger than 6, the encoder D terminates the output, then the HRL policy will output new option for subsequent actions. The training algorithm is detailed Algorithm The option learned this way is supposed to be more robust state changes. However, this method relies good predictor. The length of the sequence is dependent on the hyper parameter
3.5 POLICY GRADIENT OPTIONS
Given the option, we employ Semi-MDP framework to train an HRL agent at higher level. Because the option is in continuous space, options. Note that although our option is continuous, our temporal abstraction could be applied to discrete problems, long the RVAE outputs discrete actions.
policy gradient algorithm and derive the algorithm

---

Under review as a conference paper at ICLR 2020
a) 2D navigation
b) MoJoCo
c) Ant turn
Il. Ant maze
IlI. Ant push
Ant
Figure 4:
Evaluation tasks. a) 2D navigation task. b) MoJoCo tasks including Ant, HalfCheetah, Hopper and Walker2d, с) Ant tasks including AntTurn AntMaze, AntPush and AntFall.
The gradient computation follows the principle with the flat RL problem formulated by Sutton et al. 2000); Schulman et al. (017)) The gradient over the parameters has the form:
(oU( = Bi(ar.o)><a Vologg (ox |sc; 0) A(s,, 0:)]
(10)
where the x (o)s; 0) the high-level policy planning in state-option space. The advantage function A (s2, o:) is designed the accumulative reward subtracting baseline function:
Alst, ot) R(st, 0r) V(s+)
(11)
where R(se, 0+) is the cumulative discounted reward from time to the future, and the baseline function (88 is updated with the TD algorithm (Sutton Barto 2018) (assuming the upcoming options are of length &1, t2)...):
(s+) + (st) + a(r--4 + + ((s+4,) - V(s:))
(12)
where To-e44, is the discounted cumulative reward during executing o, between and t f1:
T2- 7T+11 + 7 Tt-111
(13)
In the experiments, we employ PPO algorithm (Schulman et al. 2017). In general, the policy gradi- ent over option and the policy gradient action are similar. Most of the existing RL algorithms such as TRPO, TNPG (Schulman et al. 2015) and SAC (Haarnoja et al. 2018b) are applicable.
4 EXPERIMENTS
We first consider 2D navigation task for proof of the concept. Then we apply our temporal abstrac- tion framework robotic control tasks in MuJoCo (Todorov et al., 2012; Brockman et al. 2016). At last we will move to more challenging tasks, which are also used by Haarnoja et al. (20 8a) and Nachum et al. (2018), and are hard solve by oon-hierarc methods.
The experiments in this section are generally performed in three steps: 1) collecting the experiences using a flat PPO agent; 2) learning the option based on the TAIC algorithm; 3) training HRL based on the option. We implement one-layer LSTM (Hochreiter & Schmidhuber] 1997) with 64 hidden units for both the E and D, and three-layer MLPs for the RL, F and P. We use learning rate 0.01 for the E. D and P. learning rate 0.0003 for the RL, 0.001 for F. We also balance multiple losses with different weights: d&i is used for the Lgl in RVAE; Nadv is used for Leado for F.
4.1 2D NAVIGATION TASK
We first test our framework using a navigation task, which is toy example that allows us easily visualize the option we learned from the experience. As Figure 4 (a) shows, environment is 10m by 10m room with obstacles. Both the state and the action are continuous, represented by (x, y) and (Uz, vy) respectively. The goal is to navigate from the start location (orange circle) to the goal location (green circle). The reward for reaching the goal is 100, and -1 otherwise.
We collect the experience using flat PPO agent, running multiple tasks with random start and goal location. Then we train an RVAE network using the collected action sequences. Figure 5 (a) shows the interpolation between options. We first randomly sample two action sequences from

---

Under review as a conference paper at ICLR 2020
Dim1
Dim2
Dima
Dim4
Dim5
DimB
100
TACC
Stow
500
1100)
1300
2005
a) Interpolation
b) Dimension disturbance in option space
c) Environment changing task
Figure 5: 2D navigation task. a) Interpolation in option space. b) Disturbing one dimension in the option space. <) Comparing the flat RL and TAIC on the ability of recovering from sudden environment change (average over 10 trials).
the testing set (shown in red dash lines), and encode them into two options using the network E. We do a linear interpolation between these two options and then decode those options into a set of action sequences using network D) The first thing we notice is that the RVAE nicely capturing the direction and curvature of the sequences and usually outputs a smoother version (solid lines) of the original sequence (red dash lines). Further, the interpolations between two sequences smoothly transfer from one to the other, denoting that we have smooth option space.
In Figure (b), visualize how each dimension in the option space encodes different information. The red dash line shows an action sequence sampled from the testing set. This action sequence is encoded into option which is vector with 6 real values. We modify
dimension and decode the options into action sequences shown with the solid lines. We notice the first two dimensions con- trol the ending point of the action sequence, and dimension and 4 control the curvature, dimension 5 and control more subtle properties. Note that not all the trials of training get the same results. Sometimes the properties are mixed, but FigureBJillln illustrates one of the most common results.
Next, we evaluate the ability to recover from a sudden change in the environment. (uutoon et al. (1999) and Bacon al. (2017) also presented similar experiment. Initially the environment is set up as Figure (a) left. After 1000 episodes, an obstacle blocks the shortest path, and creates a trap (Figure (a) right). We reset the standard deviation output of the PPO algorithm to encourage exploration. The flat RL is hard to recover from this situation since the exploration at each time step still gives a similar trajectory, which leads to the trap. While the exploration the option space results in more diverse trajectories. We average 10 repeated experiments for both settings. As can be seen in Figure (c). the TAIC recovers from the environment changes for all 10 trials. While the flat RL agent find the goal in 20% trials.
4.2 ROBOTIC CONTROL TASKS
In the robotic control domain, we utilize the MuJoCo tasks in the OpenAl Gym environ- ment (Todorov al., 2012 Brockman al. 2016). The goal is learn a control policy π S A, Both the state and the action are vectors of continuous values. We present two ways of evaluating the TAIC framework. Firstly, we qualitatively evaluate the learned option by visualize the option space. Secondly, we apply the learned option to the HRL training and compare the performance.
so that the robot runs forward as fast as possible (Figure (b)). (b)).
4.2.1 VISUALIZATION OF THE OPTION SPACE
Evaluating the high dimensional option space is difficult. We observe that the losses in the option learning process (Lrvae. Lired and Lodu) could not reveal the underlying structure of the option space. We propose a visualization method that provides qualitative evaluation of the option space.
We now consider how to visualize the correlation between the option and state change. We randomly
sample the options (0) g(l) (2)) . from the option space, and decode the options into action
sequences (ag...,, :...ky 0 1.......--- All the action sequences are applied the same start state,
(0) (1) (2)
8

---

Under review as a conference paper at ICLR 2020
(no constraint)
Term-output
Temm-predict
Figure 6: Visualization of option space w.r.t state change. Option obtained by RVAE wo/ con- straints. b) Option w/ constraints and Tern-output. c) Option w/ constraints and Term- Term-predict. The points in (c) most ordered w.r.t. the color. This suggests the option distribution in (c) is most correlated with state change.
obtaining set of end states (s(0),()) s(2) ,..... Since the start states the same, we use the end state to represent the state changes.
Figure 6 shows the visualization of the option space trained for HalfCheetah-vl task. Each point in the figure represents option-state pair (o(*) s() The options and the end states are first converted to 2D vectors using t-SNE (Maaten & Hinton 2008). respectively. The 2D vector of the end states associated the location of the points in the figure, while the color of the points represents the distribution of the options. This is to say, if the option distribution is more coupled with the state change, the points will be arranged better with respect the color. As we can from Figure 6 with information-theorede constraints the options and state changes become more correlated.
4.2.2 HRL USING OPTIONS
We evaluate the option on four benchmark tasks in simulation. Figure 7 compares the HRL perfor- mance with different termination conditions. We observe the HRL using TAIC outperform the flat PPO in 3 out of 4 tasks. In the HarfCheetah and the Ant task, the curve of TAIC starts higher (be- cause the random policy over the option space already gets positive rewards) and converges faster. One thing should be notice is that the HRL policy is updated in a much lower frequency than the flat RL, because the options are executed in multiple time steps (a typical average length is 5 time steps).
In the Walker2d task, our method does not outperform the flat RL. Our hypothesis that the agent is more unstable in this task. In our current setup, the decoder D does not depend on the state S. It acts like an open-loop controller that is more vulnerable to unstable and fast-changing states. However, our option model can be extended close-loop sub-policies x (alo, s) that also depends on the state. We will explore this idea in the future study.
The different termination conditions have pro and cons. The Fix-len termination condition is more simple and stable in most of the cases. The ones with Term-output and Term cemm-rredict termination condition get better results some cases, but they require more tuning of the parameters such as the predict threshold, which controls the average length of the sequences.
The experiences used for training the option model is critical factor that decides what the agent can learn (as humans can also be biased by their previous experiences). We tested two ways of selecting experiences: random selecting the experiences and selecting experiences with higher rewards. From our experiments, the agent learns faster and performs better in the latter setting.
4.3 TRANSFER THE OPTION TO MORE DIFFICULT TASKS
The TAIC framework provides efficient way to transfer the past experiences to unseen and more sophisticated tasks. We apply the options learned from the above MoJoCo Ant-v1 running task, to novel tasks shown in Figure 4 (c). In the Ant turn task, the agent is awarded by making left turn as fast as possible. The goal of the other three tasks (Ant maze, Ant push and Ant fall) is to move from the start location (orange circle) the goal location (green circle). The reward is the negative of the distance between the agent and the goal. The last two tasks require the agent to manipulate the red box in right way, in order avoid being stuck in sub-optimal trap states.
As shown in Figure 8 TAIC (green) achieves higher rewards and quickly converges to the goal state in the first three tasks. In contrast, the flat RL. (blue) method is not able to solve these tasks in

---

Under review as a conference paper at ICLR 2020
HallCheetah
Warker
TOCII cendraalii Jerre vedic
by conemaais berr widcct
3007
2000
1080
oe-commmnns 1art pridit
$61990
203 455 Santerr
138 2000 350
25
3258 1500 3758 3003
503
1000
2000
300
Figure 7: Evaluating the performance of TAIC by HRL training on MuloCo tasks. The proposed framework with different termination conditions are compared with the flat RL. Each case is an average over 5 repeated training.
ASt.
Ant push
bal
-32
-10
-310
-33
-20
Tharkkr
Twerre
-20
-1000
-50
-40
-13
Figure 8: Transfer capability. We compare the proposed TAIC framework with two baseline methods on more challenging tasks. The TAIC framework efficiently transfers the options learned in simpler Ant-v1 task to these novel and more complex tasks.
million interactions without any task transfer treatment. We also apply transfer learning on the flat RL (orange) by using the Ant-vl policy as an initialization. As expected, transfer learning does not bring comparable benefit to the flat RL, indicating sharing weights between different tasks is less efficient. The last two experiments also show that the transferred policy (orange) may even degrade the performance. This does not happen to our TAIC framework since it transfers high level abstracted knowledge in the form of options.
5 CONCLUSION
This paper presented a general HRL framework TAIC for learning temporal abstraction from action sequences. We formulate the temporal abstraction problem as learning latent representations (called options) over action sequences. In order to learn a better representation, we derive theoretically on how to regularize the option space and give an applicable solution of adding constraints to option space. In the experiments, we try to reveal the underlying structure of the option space by visualizing the correlation between options and state changes. We showed qualitatively and quantitatively that our options encode meaningful information and benefit the RL training. Furthermore, the TAIC framework provides an efficient tool to transfer the knowledge learned from one task to another. Our framework can be applied together with all kinds of RI. optimization algorithms, and can be applied to both discrete and continuous problems.
This work brings many new directions for future studies. As we currently learn the RL task and the option separately, the option could not be improved with the improvement of the policy. In theory, is entirely feasible to jointly optimize the two parts, or at least train them alternately. As mentioned above, the current sub-policy acts like an open-loop controller. So learning a close-loop sub-policy beyond the RNN decoder will be one of the focus areas of our future studies. We would also like to apply the TAIC framework to discrete problems and with other RL algorithms such as DQN and SAC. This could bring more insights to further improve the framework.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 166-175. JMLR. org, 2017.
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath, A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05580 2017.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. Thirty-Firsi AAAI Conference Artificial Intelligence, 2017.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(1-2):41-7,, 2003.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. arXiv preprint arXiv:1511006344 2015.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1r0iv:1606.0155 2016.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980-2988. 2015.
John Co-Reyes, YuXuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajec- tory embeddings. In International Conference Machine Learning, pp. 1008-1017, 2018.
Thomas G Dietterich. An overview of maxq hierarchical reinforcement learning. In International Symposium on Abstraction, Reformulation, and Approximation, pp. 26-44. Springer, 2000.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06077 2018.
Otto Fabius and Joost van Amersfoort. Variational recurrent auto-encoders. arXiv preprint arXiv:1412.6581, 2014.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. arXiv preprint (oXiv:/704.03011 2017.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. arXiv preprint arXiv:1710009767000 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Paseal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learning Research, 17(1):009-20300, 2016.
David Ha and Douglas Eck. arXiv:1704.0344 034477. 2017.
neural representation sketch drawings.
arXiv preprint
David Ha and Jürgen Schmidhuber. World models. arXiv preprini arXiv: 1803. 2018.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. arXiv preprint arXiv: 02808, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint 1rXivv1801.01290, 2018b.
M Hauskrecht, N Meuleau. L Kaelbling. T Dean, and C Boutilier. Hierarchical solutions of mdps using macro-actions. In Proc. UAI, volume 98, 1998.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning embedding space for transferable robot skills. 2018.

---

Under review as a conference paper at ICLR 2020
Bernhard Hengst. Discovering hierarchy in reinforcement learning with hexq. In ICML, volume 19, pp. 243-250, 2002.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1997.
Anders Jonsson. A Causal Approach to Hierarchical Decomposition in Reinforcement Learning- University of Massachusetts Amherst, 2006.
Ghorban Kheradmandian and Mohammad Rahmati. Automatic abstraction in reinforcement learn- ing using data mining techniques. Robotics and Autonomous Systems, 17(11):119-1112, 2009.
Diederik Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
arXiv: 14, 2013.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp.
2016.
L-J Lin. Hicrarchical learning of robot skills by reinforcement. In IEEE International Conference on Neural Networks, pp. 181-186. IEEE, 1993.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Nov):2579-26055 2008.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529, 2015.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, pp. 3303-3313, 2018.
Ronald Edward Parr and Stuart Russell. Hierarchical control and learning for Markov decision processes. University of California, Berkeley Berkeley, CA, 1998.
Jürgen Schmidhuber. Towards compositional learning with dynamic neural networks. Inst. für Informatik, 1990.
Jürgen Schmidhuber. Learning generate sub-goals for action sequences. In Artificial neural networks, pp. 967-972, 1991.
Jürgen Schmidhuber and Reiner Wahnsiedler. Planning simple trajectories using neural subgoal generators. From Animals to Animats 2: Proceedings of the Second International Conference on Simulation of Adaptive Behavior, volume 2, pp. 196. MIT Press, 1993.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, and Philipp Moritz, Trust region policy optimization. In leml, volume 37, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2107-2116. 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2)1811-
211. 1999.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural informa- fion processing systems, pp. 1057-1063, 2000.
12

---

Under review as a conference paper at ICLR 2020
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hierar- chical approach to lifelong learning in minecraft. In Thirty-Firee First AAAI Conference on Artificial Intelligence, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Aga- piou, et al. Strategic attentive writer for learning macro-actions, In Advances in neural information processing systems, pp. 3486-3494, 2016.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3540- 3549. JMLR. org, 2017.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3357-3364. IEEE, 2017.
13


Reviews:
[{"review_id": "SkxfupZctr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a Hierarchical Reinforcement Learning (HRL) framework based on learning latent representations of action sequences. They use a Recurrent Variational Autoencoder (RVAE) to encode action sequences from previous experience or expert demonstration. They regularize representations using the fact that these representations should contain information about state changes, but not the states themselves.\n\nThe approach is developed both intuitively and theoretically. Detailed visualisations demonstrate that the results match the intuition. The paper is well written and relatively easy to follow. The related work section is wanting - see below.\n\nComments \n\nIf we understood correctly, E, D, F, and P are pre-trained in an unsupervised way from expert demonstration as in imitation learning. We ask the authors to clarify this in the paper.\n\nIn Algorithm 1, we don't see how F is trained. Is this missing or not part of the algorithm at all? Also, in line 10, how is MSE calculated if i != j?\n\nIn the experimental section, experience is collected using a PPO agent. A flat policy is used as a baseline. Is the experience collection included in the number of interactions or just used to pre-train (parts of) the model? In the latter case, the comparison might be improper. \n\nAlso, flat policy might be a weak baseline given recent progress on HRL. Comparison with other recent methods such as those in [1][2][3] would be desirable, but not a must.\n\nTypos etc\n\nPage 3, Section 3.3, instead of \"however\" I suggest \"on the other hand\" or similar. \nPage 4, Section 3.3, \"summation of two conditional entropies\" instead of \"two conditional entropy\". \nPage 9, Section 4.2.2, \"noticed\" instead of \"notice\".\n\nRelated work\n\nWe don't think this is the first time an RVAE has been used for encoding action sequences. SeCTAR [1] also uses an RVAE to encode trajectories (both states and actions) for HRL. The authors should include a reference to the paper and discuss similarities and differences between SeCTAR and their own work.\n\nOther missing recent related works include HIRO [2] and Hierarchical Actor Critic [3].\n\nThey write: \"the HRL often requires explicitly specifying task structures or sub-goals (Barto & Mahadevan,2003; Arulkumaran et al., 2017). How to learn those task structures or temporal abstractions automatically is still an active studying area.\" \"Some early studies try to find sub-goals or critical states based on statistic methods (Hengst, 2002; Jonsson, 2006; Kheradmandian & Rahmati, 2009). More recent work seeks to learn the temporal abstraction with deep learning (Florensa et al., 2017; Tessler et al., 2017; Haarnoja et al., 2018a). However, many of these methods still require a predefined hierarchical policy structure (e.g. the number of sub-policies), or need some degree of task-specific knowledge (e.g. hand-crafted reward function).\"\n\nThese are rather recent references. To our knowledge, however, the first HRL with temporal abstraction was published 1990-1991. See the references in section 10 of the overview http://people.idsia.ch/~juergen/deep-learning-miraculous-year-1990-1991.html  \"Hierarchical RL (HRL) with end-to-end differentiable NN-based subgoal generators [HRL0], also with recurrent NNs that learn to generate sequences of subgoals [HRL1] [HRL2]. An RL machine gets extra inputs of the form (start, goal). An evaluator NN learns to predict the rewards/costs of going from start to goal. An (R)NN-based subgoal generator also sees (start, goal), and uses (copies of) the evaluator NN to learn by gradient descent a sequence of cost-minimising intermediate subgoals. The RL machine tries to use such subgoal sequences to achieve final goals.\" See also [HRL4] on another way of discovering appropriate subgoals. How does the work of the authors go beyond this original work on learning temporal abstractions for HRL? \n\n\nAdditional References mentioned above: \n\n[1] John Co-Reyes, Yu Xuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings. ICML 2018.\n[2] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-Efficient Hierarchical Reinforcement Learning. NeurIPS 2018.\n[3] Andrew Levy, George Konidaris, Robert Platt, Kate Saenko. Learning Multi-Level Hierarchies with Hindsight. ICLR 2019.\n\nOverall, we believe this is a promising paper, but we are not sure if it is ripe for publication at ICLR in its current state. For now, we'd lean towards rejecting this submission, but we might change our minds, provided the comments above were addressed in a satisfactory way. Let us wait for the rebuttal.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HygjWRsRFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper studies the hierarchical reinforcement learning (HRL) problem. It proposes a framework TAIC that learns temporal abstraction from past experience or expert demons without task-specific knowledge. The method is to formulate the problem by a temporal abstraction problem. That is, they assume that the action sequence is generated by a latent variable o. By regularizing the latent space by adding information-theoretic constraints, they are able to learn the representation. The paper later uses visualization to demonstrate the effectiveness of the learning. \n\nI would think this paper is slightly below the borderline. It is an interesting method of encoding the option sequence by a continuous variable. Therefore, the action space becomes continuous rather than discrete. However, I found it not convincing why continuous option space is better than discrete ones. It appears to me that the experiment section does not provide a comparison with previous discrete option based methods as well. \n\nComments:\n* 4th line of related work: Parr --> \"Parr & Russel\"\n* Page 2, problem formulation: in beta(s,o), s is not defined. Maybe you can denote it as beta_o(.).\n* It appears to be that the\"option\" is a sequence of actions? This can only happen in the deterministic environment. What will you do if applying pi does not give the same sequence of actions? For instance, from (s1,a1) -> (s2, a2), where s2 is generated from a random distribution, and a2 is based on s2.\n* the paper is overlength ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeV0RY3KH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper develops a method for learning a latent action representation based on prior experiences (and specifically, prior action sequences). Additionally, the paper proposes to regularize the learning of this representation using an information-theoretic constraint, yielding Temporal Abstraction with Information-theoretic Constraints (TIAC). Indeed, one promise of HRL is to allow for learning and decision making algorithms to take the long-term consequences of a decision into account when planning, exploring, assigning credit, or simply acting. The options framework (Sutton, Precup, and Singh; 1999) is a promising and well-studied toolkit for investigating these capacities of HRL. For this reason, the topic of the paper is well chosen: continuing to understand how options can benefit and accelerate RL in rich environments is an important direction for research. The idea at the core of the paper is new to my knowledge: learning an encoding of action sequences with a continuous latent representation. It could be a promising technique for HRL. Experiments are conducted to evaluate the effectiveness of the method in several environments, including a continuous gridworld, control tasks, and problems involving transfer learning. \n\nVerdict: Due to lack of clarity in describing the main methods, and missing comparison to any HRL/option baselines, I recommend rejection.\n\nMore Detail:\n\nThe paper is lacking clarity in its current form. I view the main contribution as the development of the architecture and loss function that together learn an appropriate latent action representation. There are two key issues with clarity at present: 1) The presentation of the core technical contributions could be improved (see comments below in \"Q1\"), and 2) Motivation for this style of option learning is missing, with evidence that the proposed method is in fact learning an appropriate thing.\n\nToward (1): I provide suggestions where clarity could be improved below in \"Q1'.\n\nToward (2): There are a few aspects of the motivation that could be improved. First, the paper mentions that the learned options/representation will help in planning, but planning is not studied in the paper. For example: \"Further, the interpolations between two sequences smoothly transfer from one to the other, which is a desired property to have during planning, because the smooth option space provides the RL algorithm with a better search space.\" By my reading of the paper, this claim is not studied. Similarly, in the intro: \"...allow us to do planning at a higher level, and easily transfer the knowledge between different tasks\". Including experiments that explicitly evaluate the capacity of the learned representation to carry out planning would help support these claims. Or, alternatively, the contribution could be focused to model-free and policy-based learning, which is where the empirical evidence currently offers the most support. Second, no HRL baselines are compared to in the experiments. One natural comparison to include would be to the Option-Critic, which was the first technique for combining option learning with deep RL. To determine whether TIAC is a sensible approach to learning and using options, a comparison to at least one other option learning method is needed. The paper currently highlights the fact that the option-critic requires a pre-specified number of options: this is true, but it is not discussed why is this problematic, or how the current proposal remedies this difficulty. Others that may be relevant include FuN (Vezhnevets et al. 2017), the recent methods of Nachum et al. (2018), among others (Tiwari and Thomas 2019, Harb et al. 2018, Harutyunyan et al. 2019, Levy et al. 2019).\n\nIn short: the results here are promising, so I encourage the authors continue in this direction. The paper will be improved if the presentation of Section 3 is sharpened (see questions regarding clarity below) and a comparison with relevant baselines is included. \n\nMain Questions:\n\nQ1: The exposition of the main method (Section 3) was unclear to me. Here are a few questions I was left with:\n\n\t(a) Why is the posterior (on $o$) conditioned only on the action history, and not state?\n\t(b) Additionally $o$ is being treated as a random variable through 3.2. So, what is $o$? Where is the randomness coming from?\n\t(c) Section 3.3 states \"it encodes the action sequences with respect to the L2 distance in action space\". Does this mean the action space is always a subset of $\\mathbb{R}$? But, it looks like $\\mathcal{A}$ is just defined as some set: in Section 3.1, \"$\\mathcal{A}$ is the action set\". So, I am confused as to what the $L_2$ is distance defined with respect to. If the actions are always assumed to be real numbers that is entirely okay, but it would be helpful to have that stated early on. From the additional text in Section 3.3, it sounds like the transition function of each action is involved in computing this distance (\"...only have small difference in each step of action. Due to the error compounding, the two sequences...\"). \n\t(d) How is the estimate of the posterior actually used to act? The output of \"D\" in Figure 2 is $\\hat{a_{0...k}}$. What is the type of this entity? Is it guaranteed to be an element of $\\mathcal{A}$? If so, then the \"option\" here is a policy that maps $o$ and the action history $a_{0...k}$ to a new action, correct? Ah, so in Figure 3, it looks like D will have different output depending on how the termination condition is handled. Are the actions output by $D$ then executed by the RL agent, or is there some additional decision making that goes on downstream?\n\t(e) Early on the section states \"In contrast to precisely reconstructing the action sequences, our goal is to extract the latent variable capturing  the information which could benefit RL training.\" It might be helpful to include some intuition about what this information would look like. It's unclear why action history would be all that meaningful on its own (without say, the state history). It would help the section to provide some intuition for such a latent variable existing; is there an idealized, simple case that would help convey the idea? Note that this proposal comes across as different from the original proposal of the options framework: As an example, Sutton, Precup, and Singh (1999) say: \"options enable temporally abstract knowledge and action to be included in the reinforcement learning framework in a natural and general way\". This temporally abstract knowledge need not be a function of the entire action history. I like this aspect of the method as it makes the proposed algorithm quite novel, but the motivation for why this should work didn't come through for me.\n\t(h) Should the mutual information in Eq. 4 be the conditional mutual information given $a_{0...k}$? (Same question for the remaining uses of $I$ and $H$).\n\t(i) It is unclear how the option learning coordinates with the RL algorithm used. That is, suppose we train the HRL component to learn the mapping from $s, a_{0...k}$ to the constituents identified in Figure 2/3. Where does the actual RL take place? Does the algorithm just execute the actions output by $D$ at each time step? \n\nQ2: In the first experiment, it is stated: \"because the smooth option space provides the RL algorithm with a better search space.\" Any thoughts as to why this is true? Including some discussion here might help motivate the approach.\n\n\nMinor Comments:\n\n\tC1: I do not understand Figure 6. The color is said to denote \"the distribution of options\", but I couldn't quite make out what this was, precisely. It would be helpful to know the range of values it can take on, and how those values map to the displayed colors. Moreover, what is the take away from the figure? The text states \"with information-theoretic constraints the options and state changes become more correlated\" but I am having trouble connecting that claim with the visuals themselves. Some additional discussion here would be really helpful.\n\t\n\tC2: In Figure 5, what does \"dimension disturbance in option space\" mean?\n\nMinor Typos/Writing Suggestions [did not affect evaluation]:\n\tAbstract:\n\t- \"Applying reinforcement learning (RL) to\"::\"Applying reinforcement learning (RL) algorithms to\"\n\t- I am having trouble parsing this phrase: \"to learn new tasks on higher level more efficiently\". Perhaps: \"to learn new tasks at a higher level of abstraction more efficiently\"\n\t- \"over benchmark learning problems\"::\"over baseline learning algorithms on benchmark problems\"\n\n\tSec. 1 (Intro):\n\t- Plural acronyms tend to have an 's' at the end. So: Recurrent Variational AutoEncoders (RVAEs).\n\t- \"conveys meaningful information and benefit the RL training\"::\"conveys meaningful information and can benefit learning\"\n\n\tSec. 2 (Related Work):\n\t- \"the policy sketches\"::\"policy sketches\"\n\t- Personal preference, by I always prefer \"use\" to \"utilize\".\n\n\tSec. 3 (Approach):\n\t- Your $\\mapsto$ operators should be replaced by $\\rightarrow$. The $\\mapsto$ operator indicates what is applied to elements on the left, while $\\rightarrow$ specifies the domain and codomain of the function. Thus, the $\\mapsto$ variation would be $P : (s,a) \\mapsto s'$. The story is the same for $\\beta$: it should read \"$\\beta : \\mathcal{S} \\rightarrow [0,1]$\". Note that this (using $\\rightarrow$) is how Sutton, Precup, and Singh (1999) define $\\beta$ as well.\n\t- \"Sub-policy is defined as a function over the random variable.\"::\"Now, the sub-policy is defined as a function over the random variable.\"\n\t- Not a sentence: \"So that the options with similar consequences become closer in the option space.\" Consider combining with the previous sentence.\n\t- This sentence runs on: \"Given a set of past experiences...\". Consider defining $\\Lambda$ first as its own sentence, then definines the problem. Something like: \"We let $\\Lambda = ...$ Then, our problem is to learn...\".\n\t- \"it is empirically shown\"::\"it has been demonstrated empirically\"\n\t- Latex quote issue: \"”go reach the door\".\n\t- In Equations 4-9: in general, mutual information is a function of random variables. Is $o$ a random variable? For instance I have trouble expanding $H(o)$. What is $p(o)$?\n\t- \"the encode $E$ is regularized\"::\"the encoder $E$ is regularized\"\n\n\tSec. 4 (Experiments):\n\t- \"task for proof of the concept\"::\"task as a proof of concept\"\n\t- \"that allows us easily visualize the option we learned from the experience\"::\"that allows us to easily visualize the options learned from experience\"\n\t- \"that the RVAE nicely capturing the direction\":::\"that the RVAE captures the direction\"\n\t- Misuse of $\\mapsto$: \"learn a control policy $\\pi : \\mathcal{S} \\mapsto \\mathcal{A}$\" should be \"learn a control policy $\\pi : \\mathcal{S} \\rightarrow \\mathcal{A}$\" or \"learn a control policy $\\pi : s \\mapsto a$\".\n\t- \"HarfCheetah\"::\"HalfCheetah\"\n\nReferences:\n\nVezhnevets, Alexander Sasha, et al. \"Feudal networks for hierarchical reinforcement learning.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.\n\nNachum, Ofir, et al. \"Data-efficient hierarchical reinforcement learning.\" Advances in Neural Information Processing Systems. 2018.\n\nTiwari, Saket, and Philip S. Thomas. \"Natural option critic.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.\n\nHarb, Jean, et al. \"When waiting is not an option: Learning options with a deliberation cost.\" Thirty-Second AAAI Conference on Artificial Intelligence. 2018.\n\nHarutyunyan, Anna, et al. \"The Termination Critic.\" AISTATS 2019\n\nLevy, Andrew, et al. \"Learning multi-level hierarchies with hindsight.\" ICLR 2019.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel hierarchical reinforcement learning framework, based on learning temporal abstractions from past experience or expert demonstrations using recurrent variational autoencoders and regularising the representations.

This is certainly an interesting line of work, but there were two primary areas of concern in the reviews: the clarity of details of the approach, and the lack of comparison to baselines. While the former issue was largely dealt with in the rebuttals, the latter remained an issue for all reviewers.

For this reason, I recommend rejection of the paper in its current form.
Decision: Reject