Data-efficient Hindsight Off-policy Option Learning
Authors: Markus Wulfmeier
Dushyant Rao
Roland Hafner
Thomas Lampe
Abbas Abdolmaleki
Tim Hertweck
Michael Neunert
Dhruva Tirumala
Noah Yamamoto Siegel
Nicolas Heess
Martin Riedmiller
Venue:  (2021)
Keywords: Hierarchical Reinforcement Learning
Off-Policy
Abstractions
Data-Efficiency

Abstract: Hierarchical approaches for reinforcement learning aim to improve data efficiency and accelerate learning by incorporating different abstractions. We introduce Hindsight Off-policy Options (HO2), an efficient off-policy option learning algorithm, and isolate the impact of action and temporal abstraction in the option framework by comparing flat policies, mixture policies without temporal abstraction, and finally option policies; all with comparable policy optimization. When aiming for data efficiency, we demonstrate the importance of off-policy optimization, as even flat policies trained off-policy can outperform on-policy option methods. In addition, off-policy training and backpropagation through a dynamic programming inference procedure -- through time and through the policy components for every time-step -- enable us to train all components' parameters independently of the data-generating behavior policy. We continue to illustrate challenges in off-policy option learning and the related importance of trust-region constraints. Experimentally, we demonstrate that HO2 outperforms existing option learning methods and that both action and temporal abstraction provide strong benefits in particular in more demanding simulated robot manipulation tasks from raw pixel inputs. Finally, we develop an intuitive extension to encourage temporal abstraction and investigate differences in its impact between learning from scratch and using pre-trained options. 

Under review as a conference paper at ICLR 2021
DATA-EFFICIENT HINDSIGHT OFF-POLICY OPTION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Hierarchical approaches for reinforcement learning aim to improve data efficiency and accelerate learning by incorporating different abstractions. We introduce Hind- sight Off-policy Options (HO2), an efficient off-policy option learning algorithm, and isolate the impact of temporal and action abstraction in the option framework by comparing flat policies, mixture policies without temporal abstraction, and finally option policies; all with comparable policy optimization. We demonstrate the importance of off-policy optimization, as even flat policies trained off- off-policy can outperform on-policy option methods. In addition, off-policy training and backpropagation through a dynamic programming inference procedure - through time and through the policy components for every time-step enable us train all components' parameters independently of the Kata-generating behavior policy. Experimentally. we demonstrate that HO2 outperforms existing option learning methods and that both action and temporal abstraction provide strong benefits, particularly in more demanding simulated robot manipulation tasks from raw pixel inputs. We additionally illustrate challenges in off- off-policy option learning and high- light the importance of trust-region constraints. Finally, we develop an intuitive extension further encourage temporal abstraction and investigate differences in its impact between learning from scratch and using pre-trained options.
INTRODUCTION
Despite deep reinforcement learning's s considerable successes in recent years (Silver et al. 2017 7
OpenAl et al. 2018 Vinyals al. 2019), applications in domains with limited or expensive data have so far been rare. To address this challenge, data efficiency can be improved through additional structure imposed on the solution space. Hierarchical methods, such the options framework (Sutton et al. [999; Precup 2000). present one approach to integrate different abstractions into the agent as combination of low-level and high-level controllers, option
agent. By representing policies can support reuse of low-level behaviours and can ultimately accelerate learning. The advantages introduced by the hierarchical control scheme imposed by the options framework are partially balanced by additional complexities, including possible degenerate cases (Precup 22000) Harb et al. 2018), trade-offs regarding option length (Harutyunyan et al., 2019), and additional stochasticity. Overall, the interaction of algorithm and environment, as exemplified by the factors above, can become increasingly difficult, especially in an off-policy setting (Precup et al. 2006).
With Hindsight Off-policy Options (HO2), present method for data-efficient, robust off-policy learning of options to partially combat the previously mentioned challenges option learning. We evaluate against current methods for option learning to demonstrate the importance of off-policy learning for data-efficiency. In addition, we compare HO2 with comparable policy optimization methods for flat policies and mixture policies without temporal abstraction. This allows us to isolate the individual impact of action abstraction (in mixture and option policies) and temporal abstraction in the option framework. The algorithm updates the policy via critic-weighte maximum likelloood (similar to mbdolmaleki et al. (2018b); Wulfmeier et al. (2020)) and combines these with an efficient dynamic programming procedure to infer option probabilities along trajectories and update all policy parts via backpropagation through the inference graph (conceptually related to (Rabiner 1989] Shiarlis al. 2018; Smith et al.) 2018)). Intuitively, the approach can be understood as inferring option and action probabilities for off-policy trajectories in hindsight and maximizing the likelihood of good actions and options by backpropagating through the inference procedure. To stabilize

---

Under review as a conference paper at ICLR 2021
policy updates, HO2 uses adaptive trust-region constraints, demonstrating the importance of robust policy optimization for hierarchical reinforcement learning (HRL) line with recent work (Zhang) & W 2019). Rather than conditioning on executed options, the algorithm treats these as unobserved variables, computes the marginalized likelihood, and enables exact gradient computation and renders the algorithm independent of further approximations such as Monte Carlo estimates or continuous relaxation (Li et al. 2019). As an additional benefit, the formulation of the inference graph also allows to impose hard constraints on the option termination frequency, thereby regularizing reward
the learned solution without introducing additional weighted loss terms that are dependent scale. We exploit this to further investigate temporal abstraction for learning from scratch and for pre-trained options. Experimentally, we demonstrate that HO2 outperforms existing option learning methods and that both action and temporal abstraction provide strong benefits, particularly more demanding simulated robot manipulation tasks from raw pixel inputs.
Our main contributions include:
data-efficient off- off-policy option learning algorithm which outperforms existing option learning methods on common benchmarks.
careful analysis to isolate the impact of action abstraction and temporal abstraction in the option framework by comparing flat policies, mixture policies without temporal abstraction, and finally option policies. Our experiments demonstrate individual improvements from both forms of abstraction in complex 3D robot manipulation tasks from raw pixel inputs, and include ablations of other factors such as trust region constraints and off-policy versus on-policy data.
intuitive method to further encourage temporal abstraction beyond the core method, using the inference graph to constrain option switches without additional weighted loss terms. We investigate differences between temporal abstraction in learning from scratch and pre-trained options and show that optimizing for temporally abstract behaviour in addition to simply providing the methods for its emergence mostly provides benefits in the context of pre-trained options.
2 PRELIMINARIES
Problem Setup We consider reinforcement learning setting with an agent operating in a Markov Decision Process (MDP) consisting of the state space S, the action space A, and the transition probability p(st+1| St, ax) of reaching state 8:411 from state when executing action a,, The actions a are drawn from the agent's policy x x(a (aa). where z; can either refer to the current state 8 of the agent or, in order to model dependencies on the previous steps, the trajectory until the current step, hú {st, dt- 1, St-1; .0, J- Jointly, the transition dynamics and policy induce the marginal state visitation distribution p(*+). The discount factor together with the reward rz = r (8,, ax) gives rise
to the expected return, which the agent aims to maximize: J(()) = (p(..)))(ar..))) 1"r.
We start by describing mixture policies as an intermediate between flat policies and the option framework (see Fig which introduce a type of action abstraction via multiple low-level policies. Note that both Gaussian and mixture policies have been trained in prior work via similar policy optimization methods (Abdolmaleki et al. 2018a Wulfmeier al., 2020), which we will extend towards option policies. In particular, we will make use of the connection to isolate the impact of action abstraction and temporal abstraction in the option framework in Section 4.2 The next paragraphs focus on computing likelihoods of actions and options, which forms the foundation for the proposed critic- weighted maximum likelihood algorithm to learn hierarchical policies.
simplification of the options framework without initiation or termination condition, with resampling of options after every step (i.e. no dependency between the options of timestep and L + 1 in Figure The joint probability of action and option is given as:
Mixture Policies can be
(a,, (e 0r s2) = n² (arlse, oi) Th (or)st). with n " (o.|s:) = C (o.]sz).
(1)
where " and n² respectively represent high-level policy (which for the mixture is equal Categorical distribution πh and low-level policy (components of the resulting mixture distribution), and is the index of the sub-policy or mixture component.
2

---

Under review as a conference paper at ICLR 2021
3 METHOD
Option Policies extend mixture policies by incorporating temporal abstraction. We follow the semi-MDP and call-and-return option model (Sutton et al., 1999), defining option triple (T(se: x (arls, 0z), (st, 0t)). The initiation condition describes an option 's probability to start in state and is adapted as [(st, 02) = IVs; E S following |Bacon et al. 2017 Zhang & Whiteson 2019). The termination condition be option's probability to terminate in any given state, and the action distribution for given option is modelled by (a, (8. Every time the agent observes state, the current option's termination condition is sampled. subsequently no option is active, a new option is sampled from the controller xe (o,) st). Finally, sample from either the continued or option to generate new action.
[(s,, 0t) denotes Bernoulli distribution describing the
The transitions between options can therefore described by
p ((o1(s),0-1)) = {1 11- (st, 0r-1) 1) + (sc, Or 0r-1) (or(s)))) ((s,, 1z (8e|
if - =
(2)
otherwise
Equation B describes the direct connection between mixture and option policies. In both cases, the low-level policies n only depend on the current state. However, where mixtures only depend on the current state 8, for the high-level probabilities n" options take into account the history hn directly via the previous option as described in Equation
(e(a,, 0,/h,) = n² (aelse, 0.) n" (oelhr)
Following the graphical model in Figure I the probability of being in component or at timestep across a trajectory hi can be determined in recursive manner based on the option probabilities of the previous timestep. For the first timestep, the probabilities are given by the high-level controller " (oo (h)) = x (op (sp) and for all consecutive steps are computed as follows for M options using the option transition probabilities in Equation 2
The factorization in Equation and the following exact per-timestep marginalization over options in Equation eenalle to efficiently compute the likelihood of actions and options along off-policy trajectories (instead of using option samples) to efficiently perform intra-option learning (Precup) 2000) for all options independently of the executed option.
M
# (o: (he
Σ
[p stc 01-1) n (ot-2lhe-1) T (00-1(s-11,0--1)
(4)
In every step, we normalize the distribution following " (or(ht)) = = (o. (1)) /d) =1 # (d/
This dynamic programming formulation (similar to (Rabiner 1989 (Shiarlis et al. 2018 Smith et al. 2018)) allows the use of automatic differentiation in modern deep learning frameworks (e.g. (Abadi al. 2016)) backpropagate through the graph determine the gradient updates for all policy parameters during the policy improvement step. We compute the probability of the option at time l given past states and actions by marginalizing a,-1, we need to take the previous action that was actually chosen into account when performing the marginalization. The method is computationally more efficient than determining updates over all possible sequences of options independently and reduces variance compared to sampling based approximations with the executed option or new samples during learning. In practice we find that
the previous option. Since 0:-1 affects
Figure 1: Graphical model for flat policies (left), mixture policies (middle)- introducing type of action abstraction, and option policies (right) adding temporal abstraction via autoregressive options. While the action is solely dependent on the state for flat policies, mixture policies introduce the additional component or option o which affects the actions (following Equation Option policies do not change the direct dependencies for actions but instead affect the options themselves, which are now also dependent on the previous option and its potential termination b (following Equation! 2)
3

---

Under review as a conference paper at ICLR 2021
Σ
componante
(2[AA)
(2//v/
Figure 2: Representation of the dynamic programming forward pass - bold arrows represent connec- tions without switching. Left: example with two options. Right: extension of the graph to explicitly count the number of switches. Marginalization over the dimension of switches determines component probabilities. By limiting over which nodes to sum at every timestep, the optimization can be targeted to fewer switches and more consistent option execution.
removing the conditioning of component probabilities on actions (the wT. terms in Equation improves performance and stability. This is furthermore shown in the ablations for the effects of conditioning on potentially stale actions in off-po learning in Section 4.3
We now continue by describing the main policy improvement algorithm, which requires the previously determined option probabilities. The principal three steps are: update the critic following Eq, [ 2. generate an intermediate, con-aaametric policy based on the updated critic (Eq. 6 3. update the parametric policy to align the non-parametric improvement (Eq. 8).
Policy Evaluation In comparison to training mixture policies (Wulfmeier et al, 2020 the critic for option policies is function of 8, and since the current option influences the likelihood of future actions and thus rewards. Note that even though we express the policy as function of the history hy, Q is function of og, St, at, since these are sufficient to render the future trajectory independent of the past (see the graphical model in Figure T]. In the constrained case Q also depends on ne, which we omit here for clarity. We define the TD(O) objective as
min L(Ø) E. (er Qolst, a, =.))].
where the current states, actions, and options are sampled from the current replay buffer D. For the 1-step target Qr T₁ +...E.... approximated with the sample 5.-1 from the replay buffer, and we estimate the value by sampling actions and options according to de+1, 0
[Q'(s++1, at+1, 0c+1]], the expectation over the next state is
n (-1h441) following Equation 2
Policy Improvement We follow an et al.] 2020 Abdolmaleki et al. 20T8b), which first computes an improved non-parametric policy and then updates the parametric policy to match this target. Given Q, all we require to optimize option policies is the possibility to sample from the policy and determine the log- bigelikelihood (gradient) of actions and options under the policy given he- The first step of policy improvement provides us with the non-parametric policy ((a,, or
procedure similar to (Wulfmeier
max J(q) = Ee,,orrn,s (os(r ar, ot)]. s.t. ENL-D
S E
(6)
where KL(-II-) denotes the Kullback-Leibler divergence, and eg defines a bound on the KL. We can find the solution the constrained optimization problem in Equation i| closed-form and obtain q(ac, oyl x (a or(hi) exp (Qa(se. a,, or)/n))
(7)
Practically speaking, this step computes samples from the previous policy and weights them based on the corresponding semperature-calibratee values of the critic. The temperature parameter 17 is computed following the dual of the Lagrangian. The derivation and final form of the dual can be
found in Appendix C.I. Equation 17
To align parametric and non parametaii policies in the second step, we minimize their KL divergence.
0+ arg arg min BALND KL KL
s.t. ED,,D T
Too T (-1he))l=o(-II.)) S <M (8)
The distance function T in Equation
8 has trust-region effect and stabilizes learning by constraining the change in the parametric policy. The computed option probabilities from Equation are used in Equation/7 to enable sampling of options as well as Equation to determine and maximize the likelihood of option samples under the policy. We can apply Lagrangian relaxation again and solve the primal as detailed in Appendix C.2 Finally, we describe the complete pseudo-code for HO2 in target networks for policy n' and Q-function Q' to stabilize training.
Algorithm 1 where
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Hindsight Off-policy Options
input:
initial parameters for 0, and 6, KL regulaizzation parameters e, set of trajectories
while not done do
sample trajectories T from replay buffer
// forward pass along sampled trajectories
determine component probabilities H (or (+) (Eq.
sample actions a, and options o, from ITa (-)) (Eq. to estimate expectations
// compute gradients over batch for policy, Lagrangian multipliers and Q-function
6g <- - -oo Er Ej (exp (Qo(st, aj oz) //) log TT(aj, 0y14x)] following Eq. and g
6n f- vng(n) V Vn EmEE log E, exp (Qo(s), aj (j,0;)/ pj)/n)] following Eq. 17
8 V (oL (Qo(st, de2 0:) - 0+)² following Eq.
update 17, @ 1/ apply gradient updates
if number of iterations target update then // update target networks for policy x' and value function Q'
T'' no, Q' Q+
Using options over longer sequences can help to reduce the search space and simplify exploration (Sutton et al., 1999; Harb et al. 2018). Previous approaches (e.g. (Harb) al. )018) rely on additional weighted loss terms which penalize option transitions. We instead introduce a mechanism to explicitly limit the maximum number of switches between options along a trajectory to increase temporal consistency. In comparison to additional loss terms, parameter for the maximum number of switches can be chosen independently of the reward scale of an environment and provides an intuitive semantic interpretation. Both aspects simplify hyperparameter tuning.
Temporal Consistency
The 2D graph for computing option probabilities along trajectory in Figure iis extended with a third dimension representing the number of switches between options. Practically, this means that we modelling =H (oz, n4 h2) where n, represents the number of switches until timestep f. We can marginalize over all numbers of switches to determine option probabilities. In order to encourage option consistency across timesteps, we can instead sum over only subset of nodes for all n S N with N smaller than the maximal number of switches leading nh (oe(ht) En n (o,, nz hh).
For the first timestep, only switches are possible and ("0, nol(h) = everywhere except for No = and n (oulmu 0, ho) = τ (oylsu). For further timesteps, all edges following option terminations 3 lead to the next step's 's option probabilities with increased number of switches M4+1 = n4 + 1. All edges following option continuation lead to the probabilities for equal number of switches 1 = n4, This results in the computation of the joint distribution for t >> in Equation 9 with the option and switch index transitions p (or, ne St, 01-1; R(-1) described in Equation 19 in the Appendix and normalization as H (o,, n4 (ha) = ă #
(ot, ny (hx) Do
M
Ent= # (d/: n/(h)))
M,N
# (or, =
Σ
por, ne 14-) "
T² (dt-1lsi-1,, 9-1) (9)
01-1=1
4 EXPERIMENTS
In this section, we aim answer a set of questions to better understand the contribution of different aspects to the performance of option learning in particular with respect to the proposed method, HO2. Across domains, we use MPO (Abdolmaleki et al. 2018a) to train flat Gaussian policies, RHPO (Wulfmeier et al. 2020) for
policies as an intermediate step, and HO2 to train option policies- all based on equivalent policy optimization (types of critic-weighted maximum likelihood estimation).
To start, in Section 4. we explore (1) How well does HO2 perform in comparison to existing option learning methods? and (2) How important is off-policy training this context? We use a set of common OpenAl gym (Brockman et al.) 2016) benchmarks to answer these questions (Section 4.1). In Section 4.2 we ask (3) How do action abstraction in mixture policies and the additional temporal abstraction brought by option policies individually impact performance? We use more complex, pixel-based 3D robotic manipulation experiments to investigate these aspects and
5

---

Under review as a conference paper at ICLR 2021
Half
Hopper-v2
immer-122
Walkers22
Average Retum
10000
2500
140
5000
2000
2000
120
1500
1000
80
60
H02
H52
2000
3000
2000
500
20
1.0
1.5 0.0
1/5
0.0
106
106
lc6
166
Figure 3: Results on OpenAl gym. Dashed black line represents DAC (Zhang & Whiteson dotted line represents Option-Critie (Bacon et al.] 2017) and solid line represents IOPG (Smith et al. 2018) (lines reflect the approximate results after 2 x 10th steps from (Zhang & Whiteson 2019 We limit the number of switches to for HO2-limits. HO2 outperforms existing option learning algorithms.
2019
evaluate scalability with respect to higher dimensional input and state spaces. In Section 4.2. we explore (4) How does increased temporal consistency impact performance in particular with respect to sequential transfer with pre-trained options (applying the constrained formulation from Section 3). Finally, we perform additional ablations in Section 4.3 to investigate the challenges of robust off-policy option learning and improve understanding of various environment and algorithm aspects.
4.1 COMPARISON OF OPTION LEARNING METHODS
We compare HO2 (with and without limiting option switches) against competitive baselines for option learning in common, feature- based continuous action space domains. The baselines include Double Actor-Critic (DAC) (Zhang & Whiteson| 2019). Inferred Option Policy Gradient (IOPG) (Smith et al. 2018) and Option-Critic (OC) (Bacon et al. 2017).
As demonstrated in Figure 3 off-policy learning alone (here MPO (Abdolmalkki et al. 2018 (18b) improves data-efficiency and suffice outperform on-policy option algorithms such as DAC, IOPG and Option-Critic for example in the HalfCheetah and Swimmer domains while otherwise at least performing on par. This emphasizes the importance of strong underlying policy optimization method. We achieve further improvements when training mixture policies via RHPO (Wulfmeier et al. 2020) and can accelerate learning with temporal abstraction via HO2. Overall, HOZpertorms better or commensurate to MPO and RHPO across all tasks and show strong gains in the HalfCheetah and Swimmer tasks. Using the switch constraints for increasing temporal abstraction from Section 3 has overall only a minor effect on performance. We further investigate this effect in sequential transfer in Section 1 Given the comparable simplicity of these tasks and correlated solutions, principal gains associated with off- policy training. In the next section, study more complex domains to isolate gains from action and temporal abstraction.
4.2 INVESTIGATING ACTION ABSTRACTION AND TEMPORAL ABSTRACTION
Figure 4: Ball-In-Cup and Stacking. Left: Environments. Right: Example agent observations.
We
next investigate different aspects of option learning via HO2 more complex simulated robotic manipulation tasks stacking and ball-in-cup (BIC) as displayed in Figure 4 based on robot proprioception and raw pixel inputs (64x64 pixel, 2 cameras for BIC and 3 for stacking). Since the performance of HO2 for training from scratch relatively independent as seen in Figure 3 we will focus on the base method only. To reduce data requirements for these more complex tasks, we use set of known methods improve data-efficiency and accelerate learning for all methods. We will apply multi-task learning with a related set of tasks to provide a curriculum and share policies and Q-functions across tasks with details in Appendix A Furthermore,
switch constraints

---

Under review as a conference paper at ICLR 2021
assign rewards for all tasks to any generated transition data in hindsight to share data and simplify exploration (Andrychowicz et al. 2017 Riedmiller et al. 2018 Wulfmeier et al. 2020).
Across all tasks, except for simple positioning and reach tasks (see Appendix A), action abstrac- tion improves performance (mixture policy versus flat Gaussian policies). In particular. the more challenging later stacking tasks shown in Figure [ intuitively benefit from shared sub-behaviours with earlier tasks such as grasping and lifting, demonstrating accelerated learning. The introduction of temporal abstraction (option policy vs mixture policy) further accelcrates learning. Similarly to action abstraction via mixtures (RHPO), the impact increases with task complexity and is particularly pronounced for the stacking tasks.
Bill Cup
HZ2 R/PPO
MAD
20
ge
20
0.0
0.5
2.0
3.0
1.3 2.0
1.2 Ep 10-5odes
3.5
0.2
3.0
3.5
3.0
Ep 10:
sodess
105
Ecisodes
Figure 5: Results for option policies, mixture policies, and single Gaussian policies (respectively HO2, RHPO, and MPO) in multi-task domains with pixel-based ball-in-cup (left) and pixel-based block stacking (right). All four tasks displayed use sparse binary rewards, such that the obtained reward represents the number of timesteps where the corresponding condition such the ball is in the cup - fulfilled. Please see Appendix B for further details and additional tasks.
4.2.1 OPTIMIZING FOR TEMPORAL ABSTRACTION
There is difference between enabling the policy to represent temporal abstraction and explicitly maximizing it. The ability to represent temporal abstract behaviour in HO2 consistently helps in previous experiments. However, there is limited benefit in increasing temporal consistency (via limiting the number of switches) in our experiments for training from scratch in Section 4.1 In this section, we evaluate this factor for sequential transfer with pre-trained options. We first train low-level options for all tasks except for the final, most complex task in each domain. Successively, given this set of pre-trained options, we only train for the final task and compare training with and without limited switches. We build on the same domains from Section 2 block stacking and BIC based on the Sawyer robot. As shown in Figure/6, across both domains, we can see that more consistent options lead to increased performance in the transfer domain. Intuitively, increased temporal consistency and fewer switches lead smaller search space from the perspective of the high-level controller, by persisting with one behavior for longer after selection. While the same mechanism can also hold for training from scratch, it is likely that the added complexity of maximizing temporal consistency simultaneous to learning the low-level behaviours outweighs the benefits.
0.5
Pestralnedd midd Hettranned
Pratrultut: Limited
Protraieed
63
40
30
20-
20-
300 355 4do
X0 430
130
50
150
Episodes
Figure 6: Sequential transfer experiments with limited option switches. Left: BIC. Right: Stack. We see considerable improvements for limited switches. In addition, we visualize the actual agent option switch rate in the environment directly demonstrate the constraint's effect.
4.3 FURTHER ANALYSIS AND ABLATIONS
In this section, we investigate different algorithmic aspects to get a better understanding of the method, properties of the learned options, and how to achieve robust training in the off-policy setting.
Off-policy Option Learning Figure F visualizes that across most of our experiments, we find that conditioning component probabilities on the previous timesteps' action probabilities (see Section 3
7

---

Under review as a conference paper at ICLR 2021
HallCheetah-v2
Hogper-22
Swlmmer-2
Walceny2
10000
2500
4000
8c00
1500
100
3000
4000
500
40
Bo Addaan Conditaredd
1000
0.0
0.5
1.0
1.5
2.0
0.0
1.0 Staps
1.5
0.0
0.5
1.0 Steps
1.5
0.0
0.5
1.0
1.5
2.0
Steps
Figure 7: Results on OpenAl gym with/without conditioning of option probabilities on past actions.
detrimental. explanation for these results can be found in the potential staleness of performed actions with respect to the current policy caused by the off-policy training setup. We additionally include results displaying less impact in the on-policy case in Appendix |A.4 to demonstrate the connection between the effect and the off- off-policy setting. While we use simple empirical change, the problem has lead to various off off-policy corrections for goal-based HRL (Nachum et al. 2018b Levy et al. 2017) which provide a valuable direction for future work.
Trust-regions and Robustness Applying trust-region constraints for policy updates has been shown to be beneficial for non-hierarchical policies (Schulman et al., 2015 Abdolmaleki et al.) 2018b). In this section, we describe the impact of ablating the strength of constraints on the option probability updates (both termination conditions |8 and the high-level controller æc). As displayed in Figure 8 the approach is robust across different values, but very weak or strong constraints can considerably degrade performance. Note that a high value is essentially equal to constraint and causes very low performance. Training option policies strongly relies here on trust-region constraints during optimization. Details for additional tasks in the domain can be found in Appendix A.5
Stack
217
RLe0e-10000
0.0
0.5
1.0
3.5
3.0
Componett
Compsnent
Epiodes
(a) Without IA
(b) With IA
Figure 8: Block stack- ing results with different trust-region constraints.
Figure 9: Analysis on Ant locomotion tasks, showing histogram over
options, and t-SNE scatter plots in action space colored by option.
Decomposition and Option Clustering We apply HO2 to variety of simple locomotion tasks to investigate how the agent uses its capacity and decomposes behavior into options. In these tasks, the agent body (eg. "Ant") must go one of three targets room, with the task specified by the target locations and selected target index. As shown in Figure we find that option decompositions depend on both the task properties and algorithm settings. Information asymmetry (IA), here achieved by providing task information only to the high-level controller, can address degenerate solutions and lead to increased diversity with respect options (as shown by the histogram over options) and more specialized options (represented by the clearer clustering of samples action space). More extensive experiments and discussion with a wider variety of bodies and tasks, with numerous quantitative and qualitative measures of option decomposition, can found in Appendix A..] To summarize,
these analyses yield number of additional interesting observations, showing that (1) for simpler bodies like "Ball", the options are easily interpretable (forward torque, and turning left/right
different rates); and (2) applying the switch constraint introduced in Section 4.2. leads to a much lower switch rate while allowing the agent to solve the task.
5 RELATED WORK
Hierarchy has been investigated in many forms in reinforcement learning to improve data gathering as well data fitting aspects. Goal-based approaches commonly define a grounded interface between high- and low-level policies such that the high level acts by providing goals to the low level, which is
trained to achieve these goals (Dayan & Hinton 1993 Levy et al. 2017 Nachum al. 2018a bi
8

---

Under review as a conference paper at ICLR 2021
Vezhnevets et al., 2017), effectively separating objectives and improving exploration. These methods have been able to overcome very sparse reward domains but commonly require domain knowledge and crafted interface can limit expressiveness and richness of behaviour.
More emergent interfaces within policies have been investigated from an RL-as-inference perspective via policies with continuous latent variables (Haarnoja et al. 20188) Hausman et al.) 2018 Heess) et al. 2016 [gl et al. 2019; Tirumala et al. 2019 Teh al. (017). Related to these approaches, we provide a probabilistic inference perspective to off off-policy option learning and benefit from efficient dynamic programming inference procedures. We furthermore build on the related idea of information asymmetry (Pinto et al.] 2017] Galashov et al.] 2018) Tirumala et al., (019)) providing a part of the observations only to a part of the model. The asymmetry can lead to an information bottleneck
affecting the properties of learned low-level policies. We build on the intuition and demonstrate how
option diversity can be affected in ablations in Section 4.3
Our work builds and investigates the option framework (Precuu 2000; Sutton et al., [1999, which describes policies with an autoregressive, discrete latent space. Option policies commonly use a high-level controller to choose from a set of options or skills which additionally include termination conditions, to enable a skill to represent temporally extended behaviour. Without termination conditions, options can be seen as equivalent to components under a mixture distribution, and this simplified formulation has been applied successfully different methods (Agostini & Celaya 2010) Daniel et al. 2016: Wulfmeier et al.] 2020). The option framework has been further extended and improved for more practical application (Bacon et al., 2017 Harb al. 2018 |Harutyunyan et al. PrecuPre et al. 2006; Riemer et al., 2018) Smith et al.
2018). Finally, , the benefits of options and other modular policy styles have also been applied in the supervised case for learning from demonstration (Fox et al. 2017: Krishnan et al. 2017 Shiarlis et 2011).
One important step towards increasing the robustness of option learning has been taken in Zhang & Whiteson 2019) by building on robust (on-policy) policy optimization with PPO (Schulman iet al.) 2017]. HO2 has similar robustness benefits, but additionally improves data-efficiency by building off- policy learning, hindsight inference of options, and additional trust-region constraints (Abdolmaleki et al. 2018b] Wulfmeier et al.) 2020). Related inference procedures have also been investigated in imitation learning (Shiarlis et | 2018)) as well as on-policy RL (Smith et al.) (018).. In addition to inference of suited options hindsight, off-policy learning enables us to make use of hindsight assignment of rewards for multiple tasks, which has been successfully applied with flat, sinhheearchical policies (Andrychowicz et al. 2017 Riedmiller et al. 2018) and goal-based hierarchical approaches (Levy et al. 2017 Nachum et al. 2018b).
6 CONCLUSIONS
We introduce a novel actor-critic algorithm for robust and efficient off-policy training of option policies. The approach outperforms recent work in option learning common benchmarks and is able to solve complex, simulated robot manipulation tasks from raw pixel inputs more reliably than competitive baselines. HO2 infers option and action probabilities for trajectories in hindsight, and performs critic-weighted
likelihood estimation by backpropagating through the inference procedure. The ability to infer option choices given a trajectory, allows us to train from off-policy trajectories, including those from different tasks and makes it possible to impose hard constraints on the termination frequency without introducing additional weighted objectives.
We separately analyze the impact of action abstraction (via mixture policies), and temporal abstraction (via the autoregressive modeling of options), and find that each abstraction independently improves performance. The additional enforcing of temporal consistency for option choices is beneficial when transferring pre-trained options a new task but displays limited effect when learning from scratch. Furthermore, we investigate the consequences of the off- policyneics of training data and demonstrate the benefits of trust-region constraints for option learning. Finally, examine the impact of different agent and environment properties (such as information asymmetry, tasks, and embodiments) with respect to task decomposition and option clustering; which provides relevant future direction. Finally, since our method performs (weighted) maximum likelihood learning, it can be adapted naturally to learn structured behavior representations in mixed data regimes, e.g. to learn from combinations of demonstrations, logged data, and online trajectories. This opens up promising directions for future work.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In /2th {USENIX) Symposium on Operating Systems Design and Implementation ({OSDI) 16), pp. 265-283, 2016
Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin Riedmiller. Relative entropy regularized policy iteration. arXiv preprint arXiv: 02256, 2018a.
Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and 2018b.
Martin A. Riedmiller. Maximum posteriori policy optimisation. CoRR,
Alejandro Agostini and Enric Celaya, Reinforcement learnigg with a gaussian mixture model. The 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2010.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048-5058, 2017.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critte architecture. In Thirty- First AAAI Conference on Artificial Intelligence, 2017.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. ArXiv, abs/1606.01540 2016.
Christian Daniel, Gerhard Neumann, Oliver Kroemer, and Jan Peters. Hierarchical relative entropy policy search. The Journal of Machine Learning Research, 17(1):3190-3299, 2016.
Peter Dayan and Geoffrey Hinton. Feudal reinforceement In Advances in neural information processing systems, pp. 27 1-187, 1993.
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. arXiv preprint arXiv:1703.08290 2017.
Alexandre Galashov, Siddhant M Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins. Wojciech M Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in kl-regularized rl. 2018.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning, pp. 1856-1855, 2018.
Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not an op- tion: Learning options with a deliberation cost, In Thirty-Second AAAI Conference Artificial Intelligence, 2018
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Rémi Munos, and Doina Precup. The termination critie. CoRR, abs/1902.09996, 2019. URL
19902
09996
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018.
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv: 160.05//.. 2016.
Maximilian Igl, Andrew Gambardella, Nantas Nardelli, N Siddharth, Wendelin Böhmer, and Shimon Whiteson. Multitask soft option learning. arXiv preprint arXiv:1904.01033 2019.
Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. Ddco: Discovery of deep continuous options for robot learning from demonstrations, arXiv preprint ar 2017.
10

---

Under review as a conference paper at ICLR 2021
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. arXiv preprini arXiv:1712.00945, 2017.
Alexander C Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. Sub-policy adaptation for hierarchical reinforcement learning. arXiv preprint arXiv:1906.0560 2019.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv://110.0/255, 201
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3303-3313, 2018b.
OpenAl, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub W. Pachocki, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng. and Wojeiech Zaremba. Learning dexterous in-hand manipulation. CoRR, 0bs//808.0011 2018. URL http: www.xv.crg/aas///..o
00177
Lerrel Pinto, Marcin Andrychowiez, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06542 2017.
Doina Precup. Temporal abstraction in reinforcement learning. University of Massachusetts Amherst, 2000.
Doina Precup, Cosmin Paduraru, Anna Koop, Richard S Sutton, and Satinder P. Singh. Off-policy learning with options and recognizers.
In Y. Weiss, B. Schölkopf, and J. C. Platt (eds.), Advances in Neural Information Processing Systems 18. nips. cc/paper/ pp. 1097-1104. MIT Press, 2006. URL recogiiizess. pdf opt and
Lawrence R Rabiner. A tutorial hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2)257-286, 1989.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing- solving sparse reward tasks from scratch. arXiv preprint arXiv:/802. /0567, 2018
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. In Advances in Neural Information Processing Systems, pp. 10424-10434, 2018. 8.
John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, pp. 1889-1897. JMLR. org, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Kyriacos Shiarlis, Markus Wulfmeier, Sasha Salter, Shimon Whiteson, and Ingmar Posner. Taco: Learning task decomposition via temporal alignment for control. In International Conference on Machine Learning, pp. 4661-4670, 2018.
David Silver, Julian Schritwieser. Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3354, 2017.
Matthew Smith, Herke Hoof, and Joelle Pineau. An inference-based policy gradient method for learning options. In International Conference on Machine Learning, pp. 4703-4712 2018.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for
temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):111211 1999.
Yee Whye Teh, Victor Bapst, Wojeiech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. CoRR, abs/1707. 04175, 2017. URL
04175
11

---

Under review as a conference paper at ICLR 2021
Dhruva Tirumala, Hyeonwoo Noh, Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Teh, and Nicolas Heess. Exploiting hierarchy for learning and transfer in kl-regularized rl. arXiv preprini arXiv: 1903.07488, 2019.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 3540-3549, JMLR. org, 2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki. Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350- 334,454,, 2019.
Markus Wulfineier Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg. Michael Neunert, Noah Siegel. Tim Hertweck, Thomas Lampe, Nicolas Heess, and Martin Riedmiller. Compositional Transfer in Hierarchical Reinforcement Learning. In Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA, July 2020. doi: 10. 10.15607/RSS.2000 20..200000........
Shangtong Zhang and Shimon Whiteson. Dac: The double actor-critic architecture for learning options. In Advances in Neural Information Processing Systems, pp. 2010-2020, 2019.
12

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL EXPERIMENTS
A.1 DECOMPOSITION AND OPTION CLUSTERING
We further deploy HO2 on set of simple locomotion tasks, where the goal is for agent to move to one of three randomized target locations in a square room. These are specified as a set of target locations and a task index to select the target of interest.
The main research questions we aim to answer (both qualitatively and quantitatively) are: (I) How do the discovered options specialize and represent different behaviors?; and (2) How is this decomposition affected by variations in the task, embodiment, or algorithmic properties of the agent? To answer these questions, we investigate a number of variations:
Three bodies: quadruped with two ("Ant") three ("Quad"") torque-controlled joints on each leg, and a rolling ball ("Ball") controlled by applying yaw and forward roll torques.
With or without information asymmetry (IA) between high- and low-level controllers, where the task index and target positions are withheld from the options and only provided to the categorical option controller.
With without limited number of switches in the optimization.
Ctormation-asymmettt (IA) in particular, has recently been shown to be effective for learning general skills: by withholding fask-information from low-level options, they can learn task-agnostic, cemporally-consister behaviors that can be composed by the option controller to solve a task. This mirrors the setup in the aforementioned Sawyer tasks, where the task information is only fed to the high-level controller.
For each of the different cases, we qualitatively evaluate the trained agent over 100 episodes, and generate histograms over the different options used, and scatter plots indicate how options cluster the state/action spaces and task information. We also present quantitative measures (over sceds) to accompany these plots, in the form of (1) Silhouette score, measure of clustering accuracy based on inter- and intra-cluster distance and (2) entropy over the option histogram, to quantify diversity. The quantitative results are shown in Table [ and the qualitative plots are shown in Figure 10 Details and images of the environment are in Section B.4
Scenario
Option entropy
Switch rate
Cluster seore (actions)
Cluster score (states)
Claster seore (tasks)
Ball
No IA
2.105 1 0.074
0.196 1. 0.010
-0.269 1 0.058
-0.110 1 0.025
-0.056 1 0.011
Regular
With IA
2.123 1 0.066
0,346 = 0.024
-0.156 :: 0.024
-0.164 + 0.051
0.003 1 0.008
Ant
No IA
1.583 1: 0.277
0,268 = 0.043
-0.148 : 0.034
-0.182 = 0.068
-0,075 1 0.011
With IA
0.073
0,303 0.019
-0.053 0.021
-0.066 0.021
-0.032 0,006
Quad
No LA
1.792
0.336 = 0.019
-0.078 + 0.061
-0.113 = 0.035
-0 0..89 + 0. 0,050
With IA
2.200
0.403 = 0.014
0,029 + 0.009
-0.040 = 0.003
-0.447 + 0.006
Ball
With TA
No
2.233 +- 0.027
0.214
0.020
0.142
0.009
0.015
-0.304
-0.132 + 0.035
-0.250 0.135
-0.113 0.043
-0.131
00.53 + 0.003
Limited
Switches
Ant
No IA
IA
0.077
0.043
0.141 0.015
0.073 = 0.014
-0.121 0.017
-0.052 0.011
-0.155 = 0.067
-0.054 0.014
-0.081 +: 0.034
-0.050 0.007
No IA
1.549 + 0.293
0.185 = 0.029
-0.075 # 0.036
-0.126 = 0.030
-0.112 # 0.022
Quad
With LA
2:231
+: 0.012
0.167 0.125
-0.1129 = 0.029
-0.032 0.004
-0.053 + 0.009
Table 1: Quantitative results indicating the diversity of options used (entropy), and clustering accuracy in action and state spaces (silhouette score), with and without information asymmetry (IA), and with or without limited number of switches. Higher values indicate greater separability by option / component.
The results show number of trends. Firstly, the usage of IA leads to a greater diversity of options used, across all bodies. Secondly, with IA, the options tend to lead to specialized actions, as demonstrated by the clearer option separation in action space. In the case of the 2D action space of the ball, the options correspond to turning left or right (y-axis) at different forward torques (x-axis). Thirdly, while the simple Ball can learn these high-level body-agnostic behaviors, the options for
'The silhouette score is a value in [-1, 1] with higher values indicating cluster separability, We note that the values obtained in this setting do not correspond to high absolute separability, as multiple options can be used model the same skill or behavior abstraction. We are instead interested in the relative clustering score for different scenarios.
13

---

Under review as a conference paper at ICLR 2021
more complex bodies have greater switch rates that suggest the learned behaviors may be related to lower-level motor behaviors over a shorter timescale. Lastly limiting number of switches during marginalization does indeed lead to lower switch rate between options, without hampering the ability of the agent complete the task.
t-SNE scatter plots States
Histogram options
Actions
Task
Ball, no IA
Ball, with IA
Ant, no IA
Ant, with IA
Quad, no IA
Quad, with IA
Figure 10: Qualitative results for the three bodies (Ball, Ant, Quad) without limited switches, both with and without IA, obtained over 100 evaluation episodes. Left: the histogram over different options used by each agent; Centre to right: scatter plots of the action space, state space, and task information, colored by the corresponding option selected. Each of these spaces has been projected to 2D using t-SNE, except for the two-dimensionall action space for Ball, which is plotted directly. For each case, has been taken choose median representative model out of seeds.
14

---

Under review as a conference paper at ICLR 2021
A.2 ACTION AND TEMPORAL ABSTRACTION EXPERIMENTS
The complete results for all pixel and proprioception based multitask experiments for ball-in-cup and stacking can be respectively found Figures [1 and T2 Both RHPO and HO2 outperform a simple Gaussian policy trained via MPO. HO2 additionally improves performance over mixture policies (RHPO) demonstrating that the ability learn temporal abstraction proves beneficial in these domains. The difference grows as the task complexity increases and is particularly pronounced for the final stacking tasks.
Ball Above
Ball Above Cup Rim
Ball Near Max Height
100
100
80-
80-
Average Retum
80-
60
Average
40
Average
40
40
26
100
200 Episodes
Fpisades
Fpicades
Ball Near Rim
Sparse Catch
60
HD2
Averaag Retum
50-
60
RHPO
40
50
0
2.0
Episodes
Episodes
Figure 11: Complete results on pixel-based ball-in-cup experiments.
Reach
Grosp
Lft
:20
100
140
120
140
100
120
120
80
MFD
0.2
0.
3.5
15
0.0
0.5
1.0
3.5
0.5
0.5
3.5
15
0.0
0.5
1.0
3.5
Parc Normow
Stack
Stack and Leace
100
82
Rall
3.5
1.5
23
0.5
1.0
1.5
2.0
0.5
3.5
15
3.0
Ep 5odddd
143
Es66dds
162
Esioodts
Figure 12: Complete results on pixel-based stacking experiments.
A.3 TASK-AGNOSTIC TERMINATIONS
The complete results for all experiments with task-agnostic terminations can be found in Figure 13
The perspective of options task-independent skills with termination conditions as being part of a skill, leads to termination conditions which are also task independent. We show that at least in this limited set of experiments, the perspective of task-dependent termination conditions i.e. with access to task information which can be understood as part of the high-level control mechanism for activating options improves performance. Intuitively, by removing task information from the termination conditions, we constrain the space of solutions which first accelerates training slightly but limits final performance. It additionally shows that while we benefit when sharing options across tasks, each task gains from controlling the length of these options independently. Based on these results, the termination conditions across all other multi-task experiments are conditioned on the active task.
15

---

Under review as a conference paper at ICLR 2021
Reach
Gresp
Un
Flace Wide
Taikiedtpendett Takk-sperdan
60202525en 15
0.0
0.5
1.0
0.2
1. Epkodes
Episoces
Episides
Eptecdes
Place Narmw
Stark
Stark and Leave
Return
15
2.6
Scdlss
143
Esbouks
165
Esioadss
162
Figure 13: Complete results multi-task block stacking with and without conditioning termination conditions on tasks.
A.4 OFF-POLICY OPTION LEARNING
The complete results for all experiments with and without action-conditional inference procedure can be found in Figure 14
In order to train in more on-policy regime, we reduce the size of the replay buffer by two orders of magnitude and increase the ratio between data generation (actor steps) and data fitting (learer steps) by one order of magnitude. The resulting algorithm is run without any additional hyperparameter tuning to provide an insight into the effect of conditioning on action probabilities under options in the inference procedure. We can see that the on-policy case the impact this change less pronounced. Across all cases, we were unable to generate significant performance gains by including action conditioning into the inference procedure.
HoltCheetahhx2
Hopper-v2
Swimmer-y2
Wallken-22
10000
2500
140
5000
8000
2000
100
3000
1000
2000
2000
500
20
0.0
0.5
1.5
2.0
0.0
0.5
1.0
15
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.5
2.0
Stops
105
Stops
le5
Stops
le5
Stops
105
HallChoctah-v2
Hopper
Swimmoriv2
15000
150
6000
13000
100
5000
1000
50
2000
8.6
1.5
2,0
0
0.0
0.5
1.5
2.0
0
0.0
3.5
0.0
0.5
1.5
2,0
Figure 14: Complete results on OpenAl gym with and without conditioning component probabilities on past executed actions. For the off-policy (top) and on-policy case (bottom). The on-policy approaches uses data considerably less efficiently and the x-axis is correspondingly adapted.
A.5 TRUST-REGION CONSTRAINTS
The complete results for all trust-region ablation experiments can be found in Figure 15
With the exception of very high very low constraints, the approach trains robustly, but performance drops considerably when we remove the constraint fully.
A.6 SINGLE TIME-STEP VS MULTT TIME-STEP INFERENCE
To investigate the impact probabilistic inference of posterior option distributions тH (he) along the whole sampled trajectory instead of using sampling-based approximations until the current timestep, perform additional ablations displayed in Figure 16. Note that required to
16

---

Under review as a conference paper at ICLR 2021
Reach
Greap
Un
Flace Wide
1.50
100
81
2000
50.
0.2
0.5
1.5
15
0.0
0.5
1.0
15
0.2
0.5
1.6
0.5
1.0 Eptecdes
Eprodes
Epieides
Epkodes
Place Narmo
Stark
Stackand Leave
125
3.8773347
SOamOO Retum
6. CEL
6
2.6
scilcs
143
Esbouks
165
Esloads
163
Figure 15: Complete results block stacking with varying trust-region constraints for both termina- tion conditions and the high-level controller TC.
perform probabilistic inference for at least one step to use backpropagation through the inference step to update our policy components. Any completely sampling-based approach would require different policy optimizer (e.g. via likelihood ratio or raparametrization trick) which would introduce additional compounding effects.
We compare HO2 with an ablation of HO2 where we do not compute the option probabilities along the trajectory following Equation 4 but instead use an approximation with only concrete option samples propagating across timesteps for all steps until the current step. For generating action samples, we therefore sample options for every timestep along a trajectory without keeping a complete distribution For determining the likelihood of actions and options for every timestep, we rely on Equation 2 based the sampled options of the previous timestep. By using samples and the critic- weighted update procedure from Equation 8 we can only generate gradients for the policy for the current timestep instead of backpropagating through the whole inference procedure. We find that using both samples from executed options reloaded from the buffer as well as new samples during learning both can reduce performance depending on the domain. However, in the Hopper-v2 env environment, sampling during learning performs slightly better than inferring options.
options and sample actions only from the active option every timestep.
HallCheetah-wz
Hupper-v2
Swimmer-22
Walkar-12
15000
150
6000
10000
100
Executed
scon
Inferred Samples
2000
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Stops
Steps
Stops
Figure 16: Ablation results comparing inferred options with sampled options during learning (sam- pled) and during execution (executed). The ablation is run with five actors instead of single one as used in the OpenAl gym experiments in order to generate results faster.
B ADDITIONAL EXPERIMENT DETAILS
B. OPENAI GYM EXPERIMENTS
with asynchronous learner and actors. We use single actor and report performance over the number of transitions generated. Following (Wulfmeier et al.) 2020). both HO2 and RHPO use different biases for the initial mean of all options or mixture components distributed between minimum and maximum action output. This provides a small but non-negligible benefit and supports specialization of individual options. In line with our baselines (DAC (Zhang & Whiteson 2019). IOPG (Smith al. 2018), Option Critic (Bacon al. 2017)) we use 4 options or mixture components for the OpenAl gym experiments. We run all experiments with 5 samples and report variance and mean. All experiments are run with single actor in distributed setting.
All experiments
17

---

Under review as a conference paper at ICLR 2021
Hyperparameters
HO2
RHPO
MPO
Policy net Number of actions sampled per state Q function net Number of components
256-256
20
256-256
4
NA
0.1
E
5e-4
6y;
5e-5
Co
1e4
NA
le-4
NA
Discount factor (n)
0.99
Adam learning rate
3e-4
Replay buffer size
2c6
Target network update period
200
Batch size
256
Activation function
elu
Layer norm on first layer
Yes
Tanh on output of layer norm Tanh on input actions to Q-function Sequence length
Yes
Yes
8
Table 2: Hyperparameters OpenAl gym
B.2 ACTION AND TEMPORAL ABSTRACTION EXPERIMENTS
Shared across all algorithms, use 3-layer convolutional policy and Q-function torsos with [128, 64, 64) feature channels, [(4, 4), (3, 3). (3, 3)] as kernels and stride For multitask domains, we build on information asymmetry and only provide task information as input to the high-level controller and termination conditions to create additional incentive for the options specialize. The Q-function has access to all observations (see the corresponding tables in this section). We follow (Riedmiller et al., 2018, Wulfmeier al. 2020) and assign rewards for all possible tasks trajectories when adding data to the replay buffer independent of the generating policy.
Stacking The setup consists of Sawyer robot arm mounted on a table and equipped with a Robotiq 2F-85 parallel gripper. In front of the robot there is a basket of size 20x20 em which contains three cubes with an edge length of 5 cm (see Figure 4).
The agent is provided with proprioception information for the amm (joint positions, velocities and torques), and the tool center point position computed via forward kinematics. For the gripper, receives the motor position and velocity, as well as binary grasp flag. It also receives a wrist sensor's force and torque readings. Finally, it is provided with three RGB camera images at 64 64 resolution. At each timestep, a history two previous observations (except for the images) is provided to the agent, along with the last two joint control commands. The observation space is detailed in Table 5 All stacking experiments are run with actors in parallel and reported over the current episodes generated by any actor. Episode lengths are up to 600 steps.
The robot arm is controlled Cartesian velocity mode 20Hz. The action space for the agent is 5-dimensional, as detailed in Table The gripper movement also restricted to a cubic volume above the basket using virtual walls.
iff | <
stol(u,
(10)
tanh? atanh( (v0.95) (ol) else
iff Emin
slin(y, Cmin, (max)
iff Eman
(11)
U-Emia
else
Emas Emin
18

---

Under review as a conference paper at ICLR 2021
Hyperparameters
HO2
RHPO
MPO
Policy torso
(shared across tasks)
256
512
Policy task-dependent heads
100 (categorical)
200
Policy shared heads
100 (components)
NA
Policy task-dependent terminations
100
NA
NA
(terminations)
6p
le-3
657
1e-5
C,
le-4
NA
Et
lc-4
NA
Number of action samples
20
Q function torso
(shared across tasks)
400
Q function head (per task)
300
Number of components
number of tasks le6
NA
Replay buffer size Target network
update period
500
Batch size
256
Table 3: Hyperparameters. Values taken from the OpenAl gym experiments with the above mentioned changes.
Table 4: Action space for the Sawyer Stacking experiments.
Entry
Dims
Unit
Range
Translational Velocity x, y,
m/s
[-0.07, 0.07]
Wrist Rotation Velocity Finger speed
rad/s
[-1. 11
tics/s
[-255, 255]
Table 5: Observations for the Sawyer Stacking experiments. The TCP's pose is represented as its world coordinate position and quaternion. In the table, denotes meters, rad denotes radians, and refers to quaternion in arbitrary units (au).
Entry
Dims
Unit
History
Joint Position (Arm)
2
Joint Velocity (Arm)
rad/s
2
Joint Torque (Arm)
Nm
2
Joint Position (Hand)
tics
2
Joint Velocity (Hand)
tics/s
2
Force-Torqus (Wrist)
6
N. Nm
2
Binary Grasp Sensor
2
TCP Pose
m.
2
Camera images
64 64
R/G/B value
0
Last Control Command
rad/s, tics/s
2
blol(v, e) = {1 iff |u| <e
(12)
0 else
REACH(G): (d G). 0.02, 0.15):
Minimize the distance of the TCP the green cube.
19

---

Under review as a conference paper at ICLR 2021
GRASP:
Activate grasp sensor of gripper ("inward grasp signal" of Robotiq gripper)
LIFT(G):
slin(G, 0.03, 0.10)
Increase z coordinate an object more than 3em relative to the table.
PLACE_ ( Y): stol(d((,, + [0, 0, 0.05)), 0.01, 0.20) Bring green cube to a position 5cm above the yellow cube.
PLACE NARROWIG, Y): stol(d + +0, O, 0.05)), 0.00, 0.01):
Like PLACE __IDEG,, Y) but more precise.
STACK(G, Y): btol (dzy((: Y), 0.03) * (d. Y) + 0.05, 0.01) * (1- GRASP)
Sparse binary reward for bringing the green cube on top of the yellow one (with 3cm tolerance horizontally and Iem vertically) and disengaging the grasp sensor.
(TACK_AND) L.D_LEEVVEEG, Y): slol(d, (TCP, G) + 0.10. 0.03, 0.10) STACK(G, Y) Like STACK(G, Y), but needs to move the arm 10cm above the green cube.
This task consists of a Sawyer robot arm mounted on pedestal. A partially see- through cup structure with radius of 11em and height of 17cm attached to the wrist flange. Between cup and wrist there is a ball bearing, to which a yellow ball of 4. 4.9cm diameter is attached via string of 46.5em length (see Figure 4
Ball-In-Cup
Most of the settings for the experiment align with the stacking task. The agent is provided with proprioception information for the arm (joint positions, velocities and torques), and the tool center point and cup positions computed via forward kinematics. It also provided with RGB camera images at 64 x 64 resolution. At each timestep, a history of two previous observations (except for the images) is provided the agent, along with the last two joint control commands. The observation space is detailed in Table 7 the current episodes generated by any actor. Episode lengths are up to 600 steps.
All BIC experiments are run with 20 actors in parallel and reported over
The position of the ball in the cup's coordinate frame is available for reward computation, but not exposed to the agent. The robot arm is controlled in joint velocity mode at 20Hz. The action space for the agent is 4-dimensional, with only 4 out of 7 joints being actuated, in order avoid self-collision. Details are provided in Table 4
Table 6: Action space for the Sawyer Ball-in-Cup experiments.
Entry
Dims
Unit
Range
Rotational Joint Velocity for joints 1, 2, and
rad/s
[-2, 2]
Table Observations for the Sawyer Ball-in in- Cull-in-Cup experiments. In the table, m denotes meters, rad denotes radians, and refers to quaternion in arbitrary units (au). Note: the joint velocity and command represent the robot's internal state; the 3 degrees of freedom that were fixed provide constant input of O.
Entry
Dims
Unit
Joint Position (Arm)
rad rad/s
Joint Velocity (Arm)
7
TCP Pose
7
m. au
Camera images
2 64 x 64 x
R/G/B value rad/s
Last Control Command
7
Let BA be the Cartesian position in meters of the hall in the cup's coordinate frame (with an origin at the center of the cup's 's bottom), along axes A E {z, 2J
CATCH: 0.17 > B2 > 0 and /Ball2 < 0.11
Binary reward if the ball is inside the volume of the cup.
BBLL_ABOVE_BA BASE: B₃ > 0
Binary reward if the ball is above the bottom plane of the cup.
20

---

Under review as a conference paper at ICLR 2021
BALL_ABOVE RIM: BN > 0.17
Binary reward if the ball above the top plane the cup.
BALL_NEAM MAX: MEX: B 0.3
Binary reward if the ball near the maximum possible height above the cup.
BALL NEAR RIM: 1 tanh? (
||Bays (0, 0, 0.17)Il2)
0.5
Shaped distance of the ball to the center of the cup opening (0.95 loss at a distance of 0.5),
B.3 PRE-TRAINING AND SEQUENTIAL TRANSFER EXPERIMENTS
The sequential transfer experiments performed with the same settings their multitask equivalents. However, they rely on pre-training step in which we take all but the final task in each domain and train HO2 to pre-train options which we then transfer with new high-level controller the final task. Fine-tuning of the options enabled as we find that it produces slightly better performance. Only data used for the final training step is reported but all both approaches were trained for the same amount of data during pretraining until convergence.
B.4 LOCOMOTION EXPERIMENTS
The environment used for simple locomotion tasks with Ball (left), Ant (center) and Quadruped (right).
Figure 17:
Figure 17 shows examples of the environment for the different bodies used. In addition to pro- prioceptive agent state information (which includes the body height, position of the end-effectors. the positions and velocities of its joints and sensor readings from an accelerometer, gyroscope and velocimeter attached its torso), the state space also includes the ego-centric coordinates of all target locations and a categorical index specifying the task interest. Table X contains an overview of the observations and action dimensions for this task. The agent receives sparse reward of +60 if part of its body reaches square surrounding the predicate location, and 0 otherwise. Both the agent spawn location and target locations are randomized at the start of each episode, ensuring that the agent must use both the task index and target locations to solve the task.
Table 8: Observations for the go of targets task with 'Ball' and Ant.
Entry
Dimensionality
Task Index
Target locations
Proprioception (Ball)
16
Proprioception (Ant)
Proprioception (Quad)
Action Dim (Ball)
Action Dim (Ant)
8
Action Dim (Quad)
12
C ADDITIONAL DERIVATIONS
In this section we explain the derivations for training option policies with options parameterized as Gaussian distributions. Each policy improvement step is split into two parts: non-parametric and parametric update.
21

---

Under review as a conference paper at ICLR 2021
C.I NON-PARAMETRIC OPTION POLICY UPDATE
In order to obtain the con-arametric policy improvement we optimize the following equation:
max (p(h)) Boroor (Qo(s, at 0z)]
s.t. s.t.E ((hh)) (L To( (- EE
s. s.t.En,-p(t [ [1]] =
for each step of trajectory, where h4 {5r, at-1; St .., 5o} represents the history of states and actions and p(h+) describes the distribution over histories for timestep t, which in practice are approximated via the of replay buffer D. When sampling he, the state 8; is the first element of the history. The inequality constraint describes the maximum allowed KL distance between intermediate update and previous parametric policy, while the equality constraint simply ensures that the intermediate update represents a normalized distribution.
Subsequently, in order to render the following derivations more intuitive, we replace the expectations and explicitly use integrals. The Lagrangian L(g, n) can be formulated as
|Iʃ (1t)))lle, ae oz) do, da, dhe
L(gm,7)
(13)
+ n eF /I] p(he)a(aa, orl log g(ae, orllee doz du, dhe
(14)
no oelhe)
1 (1- [I] p(ha)g(aa, hel do, da, dhv
(15)
Next maximize the Lagrangian with respect to the primal variable q, we determine its derivative as,
aL(g.n,7) Qalan, or st) 7) log q(a,, orl/4) + 1 log op(ht, - 7.
0q
In the next step, we can set the left hand side to zero and rearrange terms to obtain
g(a, lel No (ar, or ex) exp Qu(se,ae.00
17 +
exp
"
The last exponential term represents normalization constant for q, which we can formulate as
n - log (ff T oeh exp (Qo(s, a1, do da,
(16)
In order to obtain the dual function ((1),, we insert the solution for the primal variable into the Lagrangian in Equation 13 which yields
I(g,n,)) [ʃʃ p(he)gla, o, o.the) a, o,) do, da, dh,
-[[] p(he)q(ar, oelhe) ( log
mater (h2) eep Go (olterneoo)) expl (xp(-=12) do da, dhe
+ cE -
toler.oshhe)
+7 []] p(he)g(ae, ozl oe) do, da, dhs
22

---

Under review as a conference paper at ICLR 2021
We expand the equation and rearrange obtain
(.). /// p(hi)g(ar. or (s ar, oz) do, da, ch1
- -n [[] p(hr)g(ar, oelhte) Qa(sc: 01) log no 0z " n ++7 7
7
do da, dhg
"
IIʃ p(hx)g(as, or log no or oelh4) do da, dh,
+ NE +
+ /|] p(hs)g(aa, oslhe) do da, dhe
In the next step, most of the terms cancel out and after additional rearranging of the terms we obtain
L(g, 11 7) = TEE + / p(
7 7+2
d/t--
We have already calculated the term inside the integral in Equation g(n) min L(g. 7,7)
16 which we now insert to obtain
4
ME + / p(he) log (ʃʃ Te(ar, oslhe) exp
(olena do da, d/,
(Golsnar ar 0z)
NE + xpphh))) log EaLoerTT exp
(17)
The dual in Equation 17 can finally be minimized with respect 77 based samples from the replay
buffer and policy.
C.2 PARAMETRIC OPTION POLICY UPDATE
After obtaining the non-parametric policy improvement, we can align the parametric option policy to the current non-parann parametraie policy. As the non-parametric policy is represented by of samples from the parametric policy with additional weighting, this step effectively employs type of critic- weighted maximum likelihood estimation. In addition, introduce regularization based on distance function which has a trust-region effect for the update and stabilizes learning.
Onew arg min E, -p(ht)
KL (L(g(a orlhe)llooaa on
arg min En p(he)
Enroong log glan, or/ log (a or [he]
arg max En, Bphr)(ha)
log (o or (1a)
s.t. Eterp((he) (TTonew (1160)(To(-(he)) < EM.
where he p(h,) is a trajectory segment, which in practice sampled from the dataset D, T is an arbitrary distance function between the new policy and the previous policy. eM denotes the allowed change for the policy. We again employ Lagrangian Relaxation enable gradient based optimization of the objective, yielding the following primal:
max min L(A, ai) -E,, ~p(he) log To(a,, oz lhe a>0
(18)
a (EN Eni-p(th)) (TT..... (he). TT (-(h))))))))))))
We can solve for by iterating the inner and outer optimization programs independently. In practice find that it is most efficient update both in parallel.
23

---

Under review as a conference paper at ICLR 2021
We also define the following distance function between old and new option policies TTTguuw (-)be), TT(- = TH(h)) + (r()h) + (ha)
(ha) = KL(Cat(fa) (he))j=1....a (htto
M
Tr-(h)) = M
Σ ==1
KL((Ca(((b) (1)))==1.2))CCa(((( (h.)))))..)))))))))
(he M
KL(N (bt), (1))) (1(())))
where Th evaluates the KL between the categorical distributions of the high-level controller, Tr the average KI. between the categorical distributions of the all termination conditions, and Ti corresponds the average KL across Gaussian components. In practice, control over the convergence of model components by applying different eM to different model parts (high-level controller, termination conditions, options).
exert additional
C.3 TRANSITION PROBABILITIES FOR OPTION AND SWITCH INDICES
The transitions for option o and switch index n are given by:
(1 - ((1,, (ss 107
n; = 14-1, 0: = 0:-1
nels, 01-1, R(-1) =
if nn R4-1 +1 otherwise
(19)
24


Reviews:
[{"review_id": "wjYjqwWtlh7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a reinforcement learning algorithm with temporal abstraction using the options framework. It provides empirical results in a variety of domains, demonstrating that the algorithm can improve data efficiency.\n\nThe paper is well motivated. Data efficiency is an important concern in applications of reinforcement learning.  The approach is sufficiently novel. The empirical results are positive, showing performance improvements in a variety of domains. Results in simulated robotic manipulation tasks are particularly positive, as measured by average return. \n\nThe paper can be improved by providing additional analysis to better understand the behaviour of the algorithm and the conditions under which it improves performance. For instance, it would be useful to see what type of options are being learned in the various domains and with different limits on the number of switches allowed. \n\nI had difficulty in evaluating the importance of the performance differences in Figure 5. In the main text of the paper, there is no information on the reward structure of the task. Additional information is present in the  appendix but I could not easily locate the relevant information (if it is indeed there). For instance, it would be useful to know what the behavioural difference is between average returns of 60 and 100.  \n\nIn figure 3, the number of switches are listed as 5. Some experimentation with various different values would be informative. In figure 5, seeing a longer training period would be informative. MPO and RHPO are still improving at the end of the learning curve. \n\nPlease explain Equation 4 in some detail.  I could not follow. In particular,  I do not follow why the  $\\pi^{L}$ term is there.\n\nThe paper is not easy to read. Many sentences do not communicate the intended meaning clearly. As an example, the first paragraph of the introduction would be difficult to understand by readers who are not already familiar with hierarchical reinforcement learning, the options framework, and the papers cited. And some of the writing is not clear regardless of the background of the reader. For instance, the first paragraph ends with \"Overall, the interaction of algorithm and environment can become increasingly difficult, especially in an off-policy setting (Precup et al., 2006).\" It is not clear what is meant by a \"difficult\" interaction here.\n\nThe writing could be more nuanced in the discussion of results presented in Figure 3. The authors write that “off-policy learning alone [..] improves data-efficiency and can suffice to outperform on-policy option algorithms such as DAC, IOPG and Option-Critic.” This is not true in every domain. Similarly, the authors write that they “achieve improvements when training mixture policies via RHPO”. Again, this is not true in all four domains. For instance, in Hopper-v2, RHPO lags behind MPO. \n\nIn the pseudo code for Algorithm 1 on page 5, please specify inputs to the algorithm. And please do not use any undefined symbols (e.g., $\\pi’$, $Q’$). \n\nThere is no reference to Figure 6 in the text. \n\nIn Figure 3, please include DAC, OC, and IOPG in the figure legend. \n\nIn Figure 3, the way the performance of DAC, OC, and IOPG are presented on the plots is misleading. For each algorithm, a constant value is shown from step 0 until step $2 \\times 10^6$ although these constants correspond only to average return obtained after $2 \\times 10^6$ steps. Furthermore, my understanding is that these numbers have been taken from Zhang & Whiteson (2019). For best experimental  practice, these algorithms should be tested by the authors themselves along with the other algorithms shown in the plot (e.g., HO2). This would ensure that the performances are truly comparable and that there are differences in relevant experimental settings. In addition, it would allow the reader to compare the algorithms along the entirety of their learning curves.\n\nDoina Precup's dissertation is listed twice in the references, with different publications years. \n\nMisuse of the comma is prevalent throughout the paper. \n\nThe author response answered some of my questions. But I cannot say that I now better understand the behaviour of the algorithm and the conditions under which it improves performance. I agree with reviewer 2 that analysing behaviour and performance in a simpler setting would be informative. While the writing has improved, it stills lacks the clarity and nuance one would wish to see in a paper at this conference.  \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bnosUaXa3yB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the Hierarchical Reinforcement Learning setting, Options in particular, and proposes an algorithm that allows to learn both the high-level and low-level (option) policies at once, from off-policy samples. An original aspect of the algorithm is that it is easy to constrain the learned policies on how often they terminate an option and start a new one. This prevents the agent from learning tiny options that immediately terminate. It is unclear whether it can also be used to prevent the agent from learning a single big option that does everything.\n\nThe paper is very interesting and has a high educational value. It combines many different approaches and is mathematically sound. The empirical evaluation shows encouraging results. However, the significance of the empirical results is difficult to measure, due to a few cons of this paper:\n\n- The paper is quite difficult to follow, and requires several attentive reads to be understood (by someone having a deep knowledge about options, intra-option learning and the Option-Critic architecture). I believe that the lack of clarity comes from the brevity of the paper, that has to fit in the page limit. I would suggest the authors to remove Figures 1 and 2, that take place while still being very difficult to understand. The text helps understand the figures, the figures do not help understand the text, so I would remove the figures. \"Problem setup\" in the preliminaries could also be removed, and replaced with an introductory paragraph in \"Method\", that clearly states what the contribution will be, and what are its main components/properties.\n- \"multi-task learning\", just before \"Experiments\", is then used in the experiments to increase the sample-efficiency of all the algorithms. Because it is used for all the algorithms and not ablated, I don't see how it contributes to the paper. I think that the paper already proposes many ideas, and that multi-task learning could be omitted and left for a future paper.\n- In the experiments, comparing MPO with HO2 allows to see a benefit from the use of options, with the proposed learning algorithm. This is a good point. However, MPO is not a well-known baseline, and a quick glance at the plots does not allow the reader to see that MPO does not use options. I would suggest either to add a little \"no options\" next to MPO in the figures, or to replace it (or add to it) PPO, ACKTR, ACER or the Soft Actor-Critic (SAC). These algorithms are well-known baselines. Not all of them are off-policy, but I believe that comparing HO2 to state-of-the-art algorithms, without restriction to off-policy ones, would better allow to illustrate all the gains to be obtained by HO2.\n- A good argument for the use of options is to use them for explainability: telling an observer what is the current option, and what it tries to achieve. Is it possible to have a small discussion of what do the options learned by HO2 do? Do they aim at goals, or do they perform small bits of trajectories?\n\nIn summary, I like the proposed algorithm, the core contribution of the paper, the contents of Section 3. However, the clarity of this paper is to me too low, and prevents fully grasping the impact of the proposed research. I therefore recommend against acceptance, and invite the authors to remove parts on multi-task, figures that do not help, and use well-known baselines in their evaluation. With the space gained with these changes, more text can be spent summarizing what the contribution will be, and motivating the use of options.\n\nAuthor response: the authors answered my question about the absence of learning curves, and provided extra details. However, I still think that the paper could be clearer and more focused, a sentiment that I think I share with the other reviewers. Given my hesitation, I would therefore not vote for accepting this paper, but I acknowledge that the proposed method is original and interesting, so I would not mind if this paper were to be accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1HdufbjlTEQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies an important area in RL, hierarchical RL, which improves data efficiency by incorporating abstractions. In this paper, the authors proposes an efficient option learning algorithm, which utilizes a TD(0) type objective and constrains the learned policy being not too far away from the past policy. In terms of different abstractions, the paper studies action abstraction through a mixture policy, and temporal abstraction through explicitly limiting the maximum number of switches between options. \n\nThe method is well-motived in general, however, I feel the notation in Alg 1 is a little bit unclear, what is \\pi' and Q'? The ablation study is well-done, to separate the effects of different types, and gives practitioners some useful guidelines. There is one thing I am curious about, do you try the methods using some online data? Since the paper argues the improvement compared with online option learning, it would be great to also have some experiments using online data for a fair comparison. \n\nI am not that familiar with hierarchical RL, so I could not give a fair judgement of the novelty compared with previous option learning literature. In terms of quality, it is a well-motivated work, clearly written in most part of the paper and gives a method with reasonably good empirical performance. I feel the off-policy argument in this paper is less clear, is it just achieved by using a Q-learning based method? This can also be used online, and how is the online-version compared with the actor-critic option learning method?", "extra_scores": {}, "extra_text": {}}, {"review_id": "0uiGvO2uKSW", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThis paper introduces a novel option-learning policy gradient method, HO2. The method learns a parameterized joint distribution over options and actions and uses a soft-continuation based approach to interrupt or \"switch\" between options before option termination. The method introduces a new meta-parameter which enforces a hard limit on the number of \"switches\" that can occur, significantly reducing the variance of the option-learning method and replacing softer loss penalization based approaches. The paper demonstrates the performance of the proposed algorithm on a handful of 3D virtualized environments as well as on robotic simulation tasks.\n\n## Review\n### Summary\nI am currently leaning towards recommending reject for this paper. While the approach and algorithm are novel, they also appear to be highly complex, don't provide a noticeable or consistent improvement over much simpler benchmarks, and I fear that the improvements that _are_ seen are likely due to variance in the results or are hidden behind the additional machinery in the algorithm. I remain slightly skeptical of the utility of the proposed meta-parameter $n$ for setting a hard limit on the number of \"switches\" between active options, with my skepticism primarily due to concerns on the difficulty of tuning this parameter and the domain-specificity of the parameter.\n\n### Details\nI'm curious about some of the hidden complexities in the algorithm. In the problem setup, a policy is defined on an MDP as possibly being $\\pi(a | h)$ where $h=\\{s_t, a_{t-1}, s_{t-1}, a_{t-2}, \\ldots, s_0\\}$,  that is a full history of interactions. First, I suppose this means we are no longer dealing with the original MDP and are working in a modified MDP where the \"Markov\" state is a full history; already leading towards an exponential growth of the state-space. The algorithm itself depends on a recursive product of distributions for the entire length of a sampled trajectory as a result of this adapted Markov state. I'm initially worried at how difficult it is to keep this probability from decaying towards 0 rapidly. There appear to be multiple partial marginalizations (e.g. $\\sum_{i = 0}^M p(X | Y = y_i) p(Y=y_i)$) which require rescaling the final product to stay within the standard simplex, perhaps this is used to prevent the product of distributions from decaying towards 0?\n\nOne of the primary motivations of the hard switching limit is that an auxiliary penalization on the objective is hard to tune. However, it isn't clear to me that the hard limit parameter $n$ would be any easier to tune. In fact, because the inclusion of $n$ seems to require more partial marginalizations, it almost seems as if this would cause additional complexity in the optimization problem. Did you find this meta-parameter easy to tune? What are the effects of choosing it to be $n=5$ for the experiments instead of (say) 10? It appears to play a very mild variance reduction role in the results (though with only 5 seeds, we _really_ can't say much about variance since this is severely underestimating variance). If the switching limit prefers to stay small, would this suggest that the best form of the algorithm is one without the options framework at all (e.g. the best version of the proposed algorithm is an unaltered actor-critic algorithm)?\n\nThe experiments in the paper start with deep neural networks with all of the necessary machinery to make Deep RL run at the moment, including experience replay, target networks, ADAM optimizer, layer norms, mini-batches, various types of activations on each layer, neural networks of different architectures for each of the three sets of weights, different stepsizes for each of the networks, etc. I fail to see why this approach couldn't have been studied in a much simpler linear function approximation setting where statistically significant results with fewer confounding variables could have been achieved. As it stands, it is entirely unclear to me if the proposed algorithm actually provides any benefit when, in the midst of all of the machinery, the modifications above the benchmark algorithms are modest. Given this, there certainly is something to be said for a novel algorithm that does perform favorably when included in the machinery of a Deep RL feat of engineering.\n\nI , however, am unsure if the proposed algorithm does perform favorably. From the results, it appears that in most cases RHPO performs equivalently to the proposed; certainly not statistically significantly different. With only 5 random seeds, it would be very hard to make sound claims; especially considering the known variance issues in Deep RL (take Henderson et al. 2018 for a deeper discussion). The one place where the proposed algorithm _does_ outperform RHPO is in the robot simulator (though again with only 5 seeds, heavy skepticism is called for). I found this fascinating and am curious if there is some structure that the proposed algorithm is able to take advantage of in this domain that RHPO is unable to replicate.\n\n# After discussion period:\nI have read all other reviews, resulting conversation, and have read the edits to the paper. After extended conversation with the authors and a deeper investigation into the empirical components of the paper, I find I have further concerns than originally realized in my original review and that many of my original concerns remain. I am lowering my recommendation from a 5 -> 3 to reflect the new concerns; namely the validity of the ablation study as detailed in-depth below.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was a fair amount of discussion about the paper.  Several reviewers felt that the paper would have been stronger if it tried to do less but better.  The reviews describe in detail what the reviewers would have found compelling, but the key suggestion is to remove the complexity that is not essential for the approach to provide consistent improvements.  Doing this requires a better understanding of the algorithm's behavior and a valid ablation study, a new concern raised during the discussion with the authors. 

The reviewers felt that the proposed approach is potentially interesting and would like to see this paper done well.
Decision: Reject