Flexible Option Learning
Authors: Martin Klissarov
Doina Precup
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: temporal abstraction
options
hierarchical reinforcement learning
deep reinforcement learning
reinforcement learning

Abstract: Temporal abstraction in reinforcement learning (RL), offers the promise of improving generalization and knowledge transfer in complex environments, by propagating information more efficiently over time. Although option learning was initially formulated in a way that allows updating many options simultaneously, using off-policy, intra-option learning (Sutton, Precup & Singh, 1999) , many of the recent hierarchical reinforcement learning approaches only update a single option at a time: the option currently executing. We revisit and extend intra-option learning in the context of deep reinforcement learning, in order to enable updating all options consistent with current primitive action choices, without introducing any additional estimates. Our method can therefore be naturally adopted in most hierarchical RL frameworks. When we combine our approach with the option-critic algorithm for option discovery, we obtain significant improvements in performance and data-efficiency across a wide variety of domains. 

Flexible Option Learning
Martin Klissarov Mila, McGill University klrtin.kisssaoovmaaa www.rroommaam.lmlml......m mogill.ca
Doina Precup
Mila, McGill University and DeepMind dprecupēcs pprecup@cs.mcgill... mogill.ca ca.
Abstract
Temporal abstraction in reinforcement learning (RL), offers the promise of im- proving generalization and knowledge transfer in complex environments, by prop- agating information more efficiently over time. Although option learning was initially formulated in way that allows updating many options simultaneously, using off-policy, intra-option learning (Sutton, Precup & Singh, 1999), many the recent hierarchical reinforcement learning approaches only update a single option at a time: the option currently executing. We revisit and extend intra-option learning the context of deep reinforcement learning, in order enable updating all options consistent with current primitive action choices, without introducing any additional estimates. Our method can therefore be naturally adopted in most hierarchical RL frameworks. When we combine our approach with the option-critic algorithm for option discovery, we obtain significant improvements in performance and efficiiffc across a wide variety of domains.
Introduction
Temporal abstraction is a fundamental component intelligent agents as it allows for explicit reasoning at different timescales. The options framework [Sutton et al. 1999b] provides clear formalism for such abstractions and proposes efficient ways to learn directly from the environment's s reward signal. As the agent needs to learn about the value of possibly large number of options, it crucial to maximize each interaction to ensure sample efficiency. [Sutton et al. [1999b] therefore propose the intra-option value learning algorithm, which updates all consistent options simultaneously from a single transition.
Recently there have been important developments on how to learn or discover options from scratch when using function approximation [Bacon et al. 2016 Smith et al.) 2018 Riemer et al.) 20188) Bagaria and Konidaris) 2019 Zhang and Whiteson 2019 Harutyunyan et al. 2017]. However, most of recent research only updates single option at time, that is, the option selected by the agent in the sampled state. This is perhaps due to the fact that consistency between options is less likely to arise naturally when using function approximation.
We propose scalable method to better exploit the agent's experience by updating all relevant options, where relevance defined as the likelihood of the option being selected. We present a decomposition of the state-option distribution which we leverage to remain consistent in the function approximation setting while providing performance improvements similarly to the ones shown the tabular setting [Sutton et al. 1999b].
When option set is given the agent and only the value of each option remains to be learned, updating all relevant options lets the agent determine faster when to apply each of them. In the case where all options components are learned from scratch, our approach can additionally help mitigate the issue of degenerate solutions [Harb et al. [2017]. Indeed, such solutions are likely due to the ret-hreerrrrrr phenomenon where an updated option has more chance of being picked again
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

when compared to a randomly initialized option. By avoiding degenerate solutions, one can obtain temporally extended options which leads to meaningful and interpretable behavior.
Unlike recent approaches that leverage inference to update all options [Daniel et al.) 2016 Smith et al.] 2018 Wulfmeier al. 2020], our method is naturally compatible with many state of the art policy optimization frameworks [Mnih et al. 2016) Schulman et al. 2017 Fujimoto et al. 20188) and requires no additional estimators. We empirically verify the merits of our approach on wide variety of domains, ranging from gridworlds using tabular representations [Sutton et al. 1999b Bacon et al.
2016
control with linear function approximation [Moore] 1991]. continuous control [odorov et
2012
Brockman et al. 2016] and rision-based navigation "Chealler-Boisver 2018
2
Related Work
The discovery of meaningful temporally extended behaviour has previously been explored through many different avenues. One such avenue is by learning to reach goals in the environment. This can be done in various ways such as identifying bottleneck states [McGovern and Barto, 2000 Şimşek
et al. 2005 Menache al., [002 Machado al. 2017] or through feudal RI. [Dayan and Hinton 1993
Vezhnevets et 20. Nachum et al. 2018).
Another avenue of research is through the options framework (Sutton et al. 1999b where the discovery of temporal abstractions can be done directly through the environment's s reward signal. This framework has lead to recent approaches [Levy and Shimkin [001]] Bacon et al. 2016 Harb
ct al. 2017 Riemer et al. 2018 (Twarri and Thomas [2188] that rely on policy gradient methods to learn the option policies. As policy gradient methods are derived for the on-policy case, these approaches have only updated the sampled option within a state. Our work complements them by providing way to transform single-option updates into updates that consider all relevant options without introducing any additional estimators.
Within the options framework, [Harutyunyan et al. 2017 have recently decoupled the way options terminate from the target termination functions. This allowed for learning longer options as if their duration was less extended in time. This can be beneficial as shorter options are shown to produce better solutions and longer options are preferred for planning [Mann et al.] 2015). By contrast, our approach looks at updating all relevant options through the experience generated by any such option, irrespective the duration.
Another line of work is proposed by Daniel et al. 2016 in which options are considered latent unobserved variables which allows for updating all options. The authors adopt an expectation- maximization approach which assumes a linear structure of the option policies. [Smith et al. (2018) alleviate this assumption and derive policy gradient objective that improves the data-efficiency and interpretability of the learned options. A conceptually related work is proposed by Wulfmeier et al. 22200] which leverages dynamic programming to infer option and action probabilities in hindsight. As these approaches rely inference methods to perform updates, it is not directly compatible with other option learning methods nor with various policy optimization algorithms, which another important feature of our approach.
3 Background and Notation
We assume a Markov Decision Process M, defined as tuple (S.A, r. P) with finite state space S, a finite action space A, a scalar reward function r : x A - (i)) and transition probability distribution P P:& S x A - DSst(S). A policy æ S + (ist(AA specifies the agent's 's behaviour, and the value function the expected discounted return obtained by following æ starting from any state: V* (S) 1>.e f (.,, As][S, s], where 7 E [0, 1) is the discount factor. The policy gradient theorem [Sutton et al., [999a] provides the gradient of the expected discounted return from initial state distribution d( d(so) with respect to a parameterized stochastic policy =<(+))): @=c(als)
aJ(() = Σ, d'd dr)(ss Σ. dc (s, where d' d'()s Σs, d(sp) Ei P = s|So = sp)
8C is the discounted state occupancy measure.
The options framework [Sutton et al. 1999b) provides a formalism for abstractions over time. An option within the option set O is composed of an intra-option policy [[a]s, o) for selecting actions, termination function B(s, o) to determine how long option executes and an initiation set
Options.

---

[s, o) to restrict the states in which option may be selected. It is then possible to choose which option to execute through the policy over options μ. : DO) The set of options along with the policy over options defines an option-valus function which can be written recursively as,
Qo(s, o) E a(als, o)r((, +
Σ Pa.p (slss))Q(ss''co) 3',0'
where Ps d'|s, o) is the probability of transitioning from (s,0) to any other state-option pair in one step the MDP. This probability is defined as,
P d'|ss o) Pp o) Σ P(s'(s,a)
where wv,a(d'o,,')) 8 ())dd)) + (1 3(', , )o.6 is the upon-arrival option probability and ō is the Kronecker delta. Intuitively, this distribution considers whether or not the previous option terminates in 8', and accordingly either chooses an option or continues with the previous one.
Learning Multiple Options Simultaneously
In this work we build upon the call-and-returr option execution model. Starting from state the agent picks an option according to the policy over options o ~ p(-[s). This option then dictates the action selection process through the intra-option policy x (T-|s, o). Upon arrival at the next state 8', the agent queries the option's 's termination function B(s', o) to decide whether the option continues new option is be picked. This process then continues until the episode terminates.
This execution model, coupled with the assumption that the options are Markov, will leverage notion of how likely an option is to be selected, irrespective of the actual sampled option. This measure will then used to weight the updates of each possible option.
4.1 Option Evaluation
In this section we present an algorithm that updates simultaneously the option value functions of all the options, written as Qo(s, o; f) and parameterized by 0. Let's recall the on-policy one-step TD update for state-option pairs,
o++1 = 0e + a (Re+1 + 0 O.)
0.+1 0, + a (B441 + -0
= 02 + a (1410(S,, 0,) + -S, O()(s,
-d 0(1))
b.
0t1
02 ab. A,O,)
where the option value functions are linear functions, i.. Qo o; 0 ((s, o) of the parameters 0, at time f and the features Φ(s, o). Since the random variables within the update rule, summarized through the vector b, and the matrix A,, are sampled from interacting with the environment, the stability and convergence properties can be verified by inspecting its expected behaviour in the limit, written as,
O.+1 02 + ab AO,)
where A = lim, [[A,] and b = lim, E[b,]. As the matrix A has a multiplicative effect on the parameters, it will be crucial for this matrix to be dositive-definite in order to assure the stability of the algorithm, and subsequently its convergence. In the on- on-policy case, this matrix is written as:
A Σ lim Ps (S Or = o) ((ss0) -7 Panm o)(ss))((s'
s',''
0) the probability of being in a particular state-option pair time In order to update all options, we will exploit the form of this state-option distribution to preserve the positive definite property of the matrix A. In the following lemma we first show that this limiting distribution exists under some assumptions and we derive useful decomposition. To do
where

---

introduce the following two assumptions. Regarding Assumption 1, it is common in standard RL to assume that the policy induces ergodicity on the state space, which we extend to all option policies. Assumption 2 is specific to the hierarchical setting and is way to ensure irreducibility, that is, that each state-option pair is reachable. It would be possible to find less strict assumptions by having access to more information about how the MDP is constructed.
Assumption 1. The option policies { 8, o) : E O each induce an ergodic Markov chain on the state space S.
The termination function ((s,0 o) and policy over options p(oos) have strictly positive probabilities in all stale-option pairs.
Assumption 2.
Lemma 1. Under assumptions Z the following limit exists,
lim Pa = s. O2 = o On) = dx (., o)
(2)
10000
and the augmented chain process is an ergodic Markov chain in the augmented state-option space. Moreover the stationary distribution can be decomposed in the following form,
dT = Σ de.nnol (,,0,,,)))) 8)
where Pv,a s, 0) is the probability over options upon arriving in a state.
Remarks. In the derivation we have avoided mentioning the concept of initiation sets such that we would not encumber the notation and presentation. The derivation would be identical if had assumed that the initation set for each option is equal to the state space, which has been done for most of recent work on options (Bacon et al. 2016). Another way reconcile original options framework would be to consider interest functions [Khetarpal et al. 2020]. which are a generalization of the concept of initiation sets. We could then add an assumption similar to Assumption 2 on the interest function by only allowing positive values. This would let us heavily favor a subset of options for certain region of the state space. However, this assumption would still preclude us from using hard constraints on the option set, such as option affordances [Khetarpal al. 202T]. More empirical evidence would be necessary investigate whether this is limitation in practice.
method with the
Ideally, if we seek to update all options in state, we could weight each of these updates by the stationary distribution d o). However, in most case, it is impractical to assume access to this distribution. A possible solution would be estimate this quantity by making use of recent approaches [Hallak and Mannor 2017 Xie et al. 2019 Zhang et al. 2020). Instead, we propose to leverage the decomposition presented in Equation
Indeed, this decomposition highlights the dependency on the upon-arrival option distribution, (ols( o), which is a known quantity it is only dependent of the agent's policy over options u(ols) and termination function 3(s, a). Although (s, ō) is itself unknown, we can collect its samples by adopting the call-and-return option execution model. More concretely, under this model, an option which is previously selected leads us to state of interest at time t. These random variables, being sampled on-policy, follow the distribution d (s, ō). Upon arriving in any state of interest option O₂ will be sampled according to call-and -enurretun, can use
S, with option Oç-1, although the distribution Pu. (O,- = o St, 0,-1) to estimate how likely it would have been for each option o to be selected. The update rule would become,
Oc41 = 02 + Pr. = 0l)
= 00 (S, = O, = 0)
(4)
where EU =, = o) is the target for given state and option pair. In Section 4.3 we show how to obtain this quantity without using any additional estimation or simulation-resettingg Our update rule would then preserve the positive-definit property of the matrix A.
8,
4.2 Option Control
In this section we extend the approach from evaluating the option value functions to the case where we seek to learn all option policies simultancously. To do so build upon the option-critic (OC)

---

update rules derived by Bacon et al. 2016 Let us first recall the intra-option policy gradient,
@nc(als, 0) d Qo(s,o,a)
Σ u. (s.0)
(5)
8,0
where, d' (., E."'PT T.
O2 o), is the 7-discounted occupancy measure over state-option pairs. As in the previous section, we wish to adopt the same call-and-return option execution model, in particular as is also used by the option-crite critic algorithm in order to perform stochastic gradient updates following Equation [ When updating all options within a state we notice that we can leverage the structure of distribution state-option occupancy measure,
d (., = Σ Pu.elols, ō)
Σ (r((s3)dd ",,
(6)
By expanding the occupancyyy measure we can separate unknown probabilities, that is the next state distribution Par (s () and the previous state-option pair di n, (3, ō), from the known upon- option distribution, that is (o s, ō). Executing Markov options in call-and-return fashion will then provide us samples from the unknown distributions, and for each sampled (S,, 1,-1) J-pair we
can weight the likelihood of choosing option at time I. Concretely, this translates into the following update rule.
log dm (A,(A, 00) Qo(s, ; = 0, A,)
S++1 - acc Epna olS, Oz-1)
(7)
0G.
We highlight that the update rules for option control do not rely on Assumption and 2, as these were specific to the evaluation setting. By updating all options within state, can expect better sample efficiency and final performance. Moreover, this may also contribute to reducing variance. Indeed, the update rule in Equation 7 can be seen as a variation on the all-action policy gradient (Sutton ett al. 1983 Petit et al., 2019 which updates all actions applied to the distribution of options. Such all-action or all-option updates can be derived as applications of the extended (Bratley et al.) 1987]
form of conditional Monte- Carlo estimators [Hammersley] 1956 Liu et al. 2019], which provide conditioning effect on the updates that tend to reduce the variance. As we will see our experiments, such variance reduction can be observed, especially with tabular and linear function estimators.
It is possible combine evaluation and control into single method. We do by instantiating these update rules within the option-criti (OC) algorithm which we present in Algorithm in appendix A.1 and name it Multi-updates Option Critic (MOC) algorithm.
4.3 Suitable targets
We have proposed a way in which all relevant options can be updated simultaneously. the case of option evaluation we relied on having access to appropriate targets ESU,|S, O, for each option within a state. However, as we execute options in the call-and-return model and we do not assume access to the simulator for arbitrary resetting, the one taken by the executing option. To leverage this experience for any arbitrary option, we will leverage standard importance sampling (Precup et al.] 2000]. Using importance sampling naturally excludes the possibility that options be defined through deterministic policies, which can be seen as limitation of our approach. That being said, importance sampling is not necessary for the option control algorithm.
only have one sampled action for each state:
The importance sampling target for any arbitrary option ō given the action in state generated by the executing option o is the following,
(f( 0) 7 (A. alSa s, 0. 5,, R4t
(0+1 o' 8' -0) 6 ('), Pp. = 0' = 5 s' =
(A. a.S. s,
It is straightforward to show that this target is unbiased and corrects for the off-policyness introduced by updating any other option than the sampled one, leading to consistency (Sutton et al., 1999b).
would be possible to consider different approaches to correct for difference between intra-option
policies [Mahmood et al. 2014 Li et al.] 2015 Harutyunyan et al.) 2016 Jiang and Li 2016 Jain
and Precup 2018 which we leave for future work.
5

---

4.4 Controlling the number of updates
possible drawback of updating all options is the prospect of reducing the degree of diversity in the option set. A possible remedy would be to favor diversity, or similar measures, through reward shaping [Gregor et al. 2016 Eysenbach et al., 20118 Pathak et al. 2019 Klissarov and Precup 2020]. In this work, we instead introduce a hyperparameter which represents the probability of updating all options, as opposed to updating only the sampled option. our experiments we will notice that high values of 7) are not necessarily detrimental to the expressivity of the option set.
Another interesting property of this hyperparameter, which we do not leverage in this work, is that could be use in a state dependent way (i.e. ns))) without changing the stability of the algorithm. This generalization adds flexibility to how want update options. In particular, it could be a way to limit the variance of the algorithm by only performing such off- updates for state-option pairs where the importance sampling is below a threshold.
5 Experiments
We now validate our method on a wide range of tasks and try to assess the following questions: (1) By updating all relevant options are able to improve the sample-efficiee and final performance? (2) Does our approach induce temporally extended options without the need of regularization? (3) Is the learned option set diverse and useful?
Most of our experiments are done in a transfer learning setting. Our method compatible with any modern policy optimization framework and we validate this by building our hierarchichal implemen- tations on top of the synchronous variant of the asynchronous advantage actor-critic algorithm [Mnih] al. 2016). that is the A2C algorithm, as well as the proximal policy optimization algorithm (PPO) [Schulman et al. 2017]. All hyperparameters are available in the appendix. Importantly, for each set of experiments use the same value of r7. For all experiments, the shaded area represents the 80% confidence interval.
We would also like to emphasize that although our implementation is based upon the option-critic algorithm [Bacon et al. 2016), our approach can be straightforwardly adapted to most hierarchical RL algorithms [Riemer et al. 2018 Jain et al. 2018 Tinnai et al., 2019 /Zhan and Whiteson] 2019
Bagaria and Konidaris] 2019
5.1 Tabular Domain
We first evaluate our approach in the classic FourRooms domain [Sutton et al., 1999b) where the agent starts in a random location and has navigate to the goal in the right hallway. As we are interested in the algorithms' sample efficiency, we plot the number of steps taken per episodes.
Experimental Setup:
In Figure la we work under the assumption that the options have been given and remain fixed throughout learning. The only element being updated is the option value functions, Qo(s, o). which is used by the policy over options to select among options. We compare our approach, leveraging the multi-options update rule in Equation 4 to the case of only updating the sampled option within state. The options used are the 12 hallway options designed in (Sutton et (1999b] (see their Figure 2 for brief summary).. In this setting, our approach achieves significantly better sample efficiency, illustrating how our method can accelerate learning the utility of a set of well-suited options.
We now move to the case where all the options' parameters are learned and leverage the learning rule for control in Equation 7 In this setting we compare our approach, multi-updates option critic (MOC). to the option-critic (OC) algorithm as well flat agent, denoted as actor-critic (AC). Both hierarchical agents outperform the flat baselines, however, our agent is able to reach the same performance OC in about half the number of episodes. Moreover, we notice that MOC shows less variance across runs, which is in accordance with previous work investigating the variance reduction effect of expected updates (Sutton et al. 1983 Petit et al. 2019).
In Figure in the appendix, we verify the kind of options that are being learned under both the option-critic and our approach. For each option we plot the best action in each state and we highlight in yellow whether the option likely to be selected by the policy over options. We first notice that for the OC agent, there is an option that is selected much more often, whereas the MOC agent learns

---

Options
Options
Learned
160
Singit Option
0.8
140
Matiple Options Update
800
MOC
120
800
100
Stops
Informaten
400
200
02
MDC ym33
0.
0.0
M0C
0.4
Egisodis
1x2
Episodes
Episodes
(a)
(h)
Figure 1: FourRooms domain. In a) we used the fixed set of hallway options as defined in Sutton et al. [999b] and only learned the option-value functions. In b) all the parameters are learned from scratch before the goal location is change mid-training. In c) we plot the information radius between options as a measure of diversity. Results are averaged over 50 independent runs.
a good separation of the state space. Moreover, we notice in purple states (where the option less likely to be chosen), the OC agent's actions do not seem to lead to meaningful behaviour in the environment, which in contrast to our approach. As potential benefit, by sharing information between options, the MOC agent will tend to be more robust to the environment's stochasticity as well as to the learning process the policy over options.
Finally, we also would like to point out possible challenge in our approach that we have mentioned in Section It is possible that by learning all options simultancously we may reduce the diversity between options. We verify this in Figure Ic where we plot the information radius [Sibson 1969 between options as a measure of diversity. We plot the OC agent as well as our approach, MOC, with different values of the hyperparameter which controls the degree to which we apply updates to all options. We notice indeed clear different between the OC agent and our agent when 7 = 1.0 (only updates to all options). However, intermediate values of 1 provide effective to increase the information radius. As in this experiment we use tabular representations, the difference between updated options and non-updated options is accentuated. In the function approximation case we will see that the concern about the diversity of options is mitigated.
5.2 Linear Function Approximation
Experimental Setup: We now perform experiments in the case where the value function is estimated through linear function approximation. The environment we consider is a sparse variant of the classic Mountain Car domain [Moore] 1991 Sutton and Barto 2018] where positive reward is given upon reaching the goal. We consider the transfer setting where after half the total number of timesteps the gravity doubled.
1.0
1.
045
0.9
Information Rotto.s
0.40
0.7
0.7
Bas
0.30
0.4
0.5
MaZ
0.23
o2
0.20
MOC
02
AC
MOC 0.9
050
125 Timestops
1.50
03
00
0.4
0.6
1.0
0.00
0.25
0.75 1:00
1.75 2,00
Timesttpss
(a)
(b)
(c)
Figure MountainCar domain. In a) we plot the performance of our agent, MOC, the option-crite critic
and flat baseline. The options are learned from scratch. At mid mid-training, we modify the gravity to double its original value. In b) we perform hyperparamter study on the value of y which controls the amount of updates to all options. In c) we plot the information radius between options as measure of expressivity and notice that higher values of 1 do not affect it negatively. The performance is averaged across 20 independent runs.
7

---

In Figure 2a we observe similar results to the tabular case, that is, hierarchical agents perform better than standard RL in the transfer task. MOC also improves upon OC's performance and sample efficiency. We verify how this improvement in performance translates with respect to the value of 7) the hyperparameter controlling for the probability of updating all options. In Figure 26 we can observe an almost monotonic relationship between v and the algorithm's 's final performance, which confirms the effectiveness of our approach when using function approximation.
Finally, in Figure 2c we verify the information radius between options for different values of 1). We notice that unlike the tabular case where higher values of 1) lead to smaller information radius, there is no clear correlation in this case. Perhaps surprisingly we notice that in the first task high values of n lead an increased information radius. This might indicate that the beginning of training, the agent can learn to specialize faster by updating all options.
5.3 Experiments at Scale
5.3.1 Continuous Control
Experimental Setup: We turn our attention to the deep RL setting where all option components are estimated through neural networks. We first perform experiments in the MuJoco domain [Todoroy] ct al. 2012) where both states and actions are continuous variables. We build on classic tasks |Brockman et al. 2016] and modify them to make the learning process more challenging. The description of each task as well as visual representation is available in the appendix.
We leverage the PPO algorithm (Schulman et al., 2017] to build our hierarchical agents. We also compare our approach, MOC, to the interest option-critic (IOC) (Khetarpal et al.) [020)] which has been shown to outperform the option-critic algorithm derived for the continuous control setting (Harb) et al. 2017 Klissarov et al.] 2017]. Additionally, we compare to flat agent using PPO and to the inferred option policy gradient (IOPG) algorithm [Smith et al. 2018), an HRI. algorithm also designed for updating all options by leveraging inference.
HarCChestnDDDv
AntWallevi
Walker2dStand-v1
800
1200
Avsrage Returre
800-
Average Reture
Returns
10c
400
1800
800
FPO
200
800
600
400
-200
10C
IDPS
IDPG
PPO
2
6
0.00
0.25
0.00
0.75
1.00
1.25
1.50
1.75
0.25
0.75
1.25
1.75
Timestape
Timoooop
Timestops
(a)
(b)
MBCC Octibra on Mull Ch-r
Golbraa Dell
MD2 Colbina Anthhbal
Ostiona AnchNillen
Arerp Opten Dundin
Taosspo
Imssdge
(d)
(c)
(f)
Figure 3: Transfer on MuJoco. In a-b-c) we compare the performance of our approach, MOC, to various hierarchical baselines and to a flat agent. At about a quarter into the training process, the original task is modified to a more challenging, but related, setting where the agents must transfer their knowledge. In d-e-f-g), we plot the option duration for both the MOC agent and the OC baseline. We notice that our approach tends to produce options that are extended in time, which can lead to meaningful behavior. All experiments are plotted using 10 independent runs.
Across environments in Figure Babe we witness that MOC shows better sample efficient and per- formance compared to the other hierarchical agents as well as the PPO baseline. In particular, although IOPG updates all options similarly to our approach, this doesn't translate a practical advantage. This highlights the adaptability of the update rules we propose to any state-of-thh-art policy optimization algorithm. We also emphasize that in all these experiments, the same value of n is used, making it general approach.
8

---

1200
Average Retumss
800
003
400
independerce
MDC
200
10C
10C
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.76
Timesteps
(b)
Figure 4: a) Option visualization. In this experiment, AntWalls-v1, the agent has first learned how to walk in the forward direction before we add obstacles (walls) which it needs to circumvent. We highlight in purple the states in which our agent uses the first option and in yellow the second option. b) Performance of MOC and IOC algorithms for different neural network architectures. Independence indicates that the second last hidden layer does not shared across optiosn.
In Figure Siefg we investigate the options' duration for both the IOC agent and our approach. We investigate both methods using two options, so as to provide more interpretable results We argue that such a metric can be an important indicator of the usefulness of options, which are meant to be temporally extended. We notice that both in HalsChectahDir-v/ and AntWalls-vi. 1,, both options learned by MOC are used by the agent over multiple timesteps. This is in contrast to the IOC agent where options at the end of training exhibit an imbalance. We also highlight that although MOC improves this aspect of option learning, the shaded are is still large which indicates that it does not fully solve the problem.
For the MOC agent, it is interesting to notice temporally extended options are truly present once the agent faces a new task. This perhaps suggests that future inquiry should look into whether challenging the agent with multiple tasks (such as in the continual learning setting) is actually catalyst for the discovery of options.
In Figure 4h, we provide a visualization of meaningful behaviour exhibited by the MOC agent. We notice that in the AntWalls-vl environment, one agent learns an option move forward (highlighted in purple), while the other one is used to move around the walls (highlighted in yellow). We provide additional visualization (including from the IOC agent) in the appendix. We mention that although such meaningful behaviour was witnessed more often in our approach, we did not completely avoid degenerate solutions for all random seeds or all values of the learning rate.
In our experiments we compared HRL algo- rithms when using two options. We now verify how MOC and IOC scale when we grow the op-
Table I: Walker2dStand-vl
tion set four and eight options. In Table we
Algorithm
2 options
4 options
8 options
notice that most of the gains from updating all
options are persistent for each cardinality. How-
MOC
617 (98)
574 (76)
535 (81)
ever, we notice eventually a significant drop in
IOC 326 (92)
238 (55)
212 (64)
performance when using 8 options. This could
attributed to the fact that the domains we exper- iment with are simple enough that such a high
number of options is unnecessary and leads to reduced sample efficiency. One way to verify this would be to apply the same algorithm in a continual learning setting. It could also be the case that the OC baseline on which we implement our flexible update rules could be itself improved for better performance. More experiments are needed to investigate these questions, which we leave for future work.
Finally, promising possibility our method re-organize the way parameters are shared between options. The neural network architecture used by OC-based algorithms (including ours) shares most of the parameters between options: only each head (last layer of the network) is specific to each option and independent of others. In Figure Hp. we investigate the performance of MOC and IOC when the parameters of the second to last hidden layer are also independent for each option. As we notice through the dashed lines, MOC remains more robust to this configuration. This would indicate

---

that by using our proposed update rules, more sophisticated network architectures could be devised without sacrificing sample efficiency.
5.3.2 MiniWorld Vision Navigation
Experimental Semu: In this section we verify how our approach extends to tasks with high dimensional inputs such as the first-person visual navigation domain of MiniWorld (Chevalier-Boisverr 2018
We provide a description for each environment in the appendix.
In WiniWorrd-YMMazeTTannff YMaze the transfer is not drastic and the flat agent is able to recover well in the transfer task. Moreover, MOC seemed to be slower to learn at first, although was able to transfer with smaller drop in performance. For the more challenging MiniWorld MiniWorld-WalllappTT Traasferrv.ef notice that HRL algorithms are more relevant and adapt better to the change, with MOC being more sample efficient.
1.0
66
VD0
012
CC
Opton A
00
0.25 0350
0.5
1.00 125
1:50 175
2.07
0
Tanation
(a)
(b)
Figure 5: Transfer in MiniWorld. In a-b) we plot the performance of our approach and compare it to the option-critt critic (OC) and a flat baseline (PPO). In particular in b) we can appreciate the need for hierarchy when transferring to a similar, but more challenging task. e) we present qualitative results on the kind of options that are being learned by our agent. In these experiments we average the performance across six independent runs.
Finally, in cp. we plot the options learned by our MOC agent in the MiniWorte "MMazTTransfer environment. The first option learns to get to the middle of the maze and locate the red goal. When the goal has been seen, the second option moves towards it. Learning such structure can potentially help an agent adapt to a new and related task.
Conclusion
In this work we have proposed a way to update all relevant options for a visited state. Our method does not introduce any additional estimators and is therefore readily applicable to many state-of-the-art RL, and HRL, algorithms in order to improve their performance and sumple-efficiency. Updating all options simultancously has been an integral part of the intra-option learning proposed in the options framework [Sutton et al. 1999b). This work therefore aims to extend this flexibility to function approximation and deep RL.
Our proposed update rules can be seen updates done in expectation with respect to options. As such, they are reminescent of generalizations of standard RL algorithms, such Expected SARSA [van
Seijen et al., (209)] and Expected Policy Gradients [Ciosek and Whiteson] 2018 Amongst other qualities, such approaches have theoretically been shown to reduce variance. A similar analysis could be performed in our framework to verify what theoretical guarantees can be obtained from updating all relevant options.
We also proposed a flexible way to control between updating all relevant options and updating only the sampled one through the hyperparameter 7). Potential future work could investigate whether state-de dete-deeendent 1) function, similarly to interest functions [Mahmood et al. 2015), could be used to highlight important transitions. such as bottlenecks, and update all options only in those states. Finally, learning all options simultaneously could be further leveraged in the case where we learn very large number of them simultaneously [Precup) 2018 Schaul al., 2015), before deciding which are useful. Such an investigation would most likely be most promising in continual learning setting. Updating all relevant options also means that we could implement less parameter sharing between options and discover neural network architectures better suited for HRL.
10

---

Acknowledgments and Disclosure of Funding
The authors would like to thank the National Science and Engineering Research Council of Canada (NSERC) for funding this research; Khimya Khetarpal for early discussions on the project and the anonymous reviewers for providing critical and constructive feedback.
References
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. abs/1609.051400, 2016. URL http:://axxi..orggg///ooo/////// wttp://rrrgaaabb/..00.0555
CoRR,
Akhil Bagaria and George Konidaris. Option discovery using deep skill chaining. In International Conference Learning Representations, 2019.
Paul Bratley, Bennett L. Fox, and Linus E. Schrage. Guide to Simulation (2nd Ed.). Springer- Verlag, Berlin, Heidelberg, 1987. ISBN 0387964673.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR,
2016. URL http://axaxx orgiiii.rrrgaaaa /bs/
1606 01540
Maxime Beovaalier-Boiveerr gym-miniworld environment for openai gym. https://gitti: com/ maximecb/ httpi/eeebggymmmmiiwoloo/ooo 2018.
Kamil Ciosek and Shimon Whiteson. Expected policy gradients, 2018.
Özgür Şimşek, Alicia P. Wolfe, and Andrew G. Barto. Identifying useful subgoals in reinforce- ment learning by local graph partitioning. In Proceedings of the 22nd International Confer- ence Machine Learning, ICML '05, page 816-823, New York, NY, USA, 2005. Associa- tion for Computing Machinery. ISBN 1595931805. doi: 10. 10.1145/1102351.1 1102454. URL htap:://doi.org//1/ 1145/1102351 1102454
Christian Daniel, Herke Van Hoof, Jan Peters, and Gerhard Neumann. Probabilistic inference for determining options in reinforcement learning. Machine Learning, 104(2):337-357, 2016.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In S. Hanson, Cowan, and C. Giles, editors, Advances in Neural Information Processing Systems, volume Morgan-Kaufmann. 1993. URL https: httpss:/pproceedings.cem dl222066Gaer766033352226666
.pf
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https
//github. wwt.co//opeeaaaaseeeeeeeeee 2017.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you
need: Learning skills without reward function. CoRR, abs/l 1bs/1800,,0070, 2018. URL htthttp!
// //arxiv.. org/ abs wtt.s/882..666 06070
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function appromaattooooooo error in actor- critic methods. CoRR, abs/1802.00947,, 2018. URL htRL:tttt///arrrv.rr/////////
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. CoRR, abs/161 1.7507, 2016. URL wwlLaatp.//arrdv.orr//a//////
Assaf Hallak and Shie Mannor. Consistent on-line off-policy evaluation, 2017.
J. M. Hammersley. Conditional monte carlo. J. ACM 3(2):33-76 April 1956. ISSN 0004-5411. doi:
10.1145/320825.22082 URL / 11455/2082555 320227
Jean Harb, Pierre-Lue Bacon, Martin Klissarov, and Doina Precup. When waiting is not option Learning options with a deliberation cost. CoRR, abs/1709.04 2017. htI.http::// //arxivv
arg/abs/1700 140bb///199005

---

Anna Harutyunyan, Mare G. Bellemare, Tom Stepleton, and Remi Munos. Q()) with off-policy corrections, 2016.
Anna Harutyunyan. Peter Vranex, Pierre-Luc Bacon, Doina Precup, and Ann Nowe. Learning with options that terminate off-policy, 2017.
Arushi Jain, Khimya Khetarpal, and Doina Precup. Safe option-critic: Learning safety in the option- critic architecture. CoRR, Abs/1807.08060 2018. http:///aaaxi org/ 1807. 08060
Ayush Jain and Doina Precup. Eligibility traces for options. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 18, page 1008-1016, Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings The 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 652-661, New York, New York, USA, 20-22 Jun 2016. PMLR. URL http:// http:://proceedings mlr.
html
Yuu Jinnai, Jee Won Park, David Abel, and George Konidaris. Discovering options for exploration by minimizing cover time. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3130-3139. PMLR, 09-15 Jun 2019. URL htttp://proceediinnii
mlr. wwwpp/eserr77/ j/na/nnnnnnnibbb. html
Khimya Khetarpal, Martin Klissarov, Maxime Chevalier-Boisvert Pierre-Luc Bacon, and Doina Precup. Options of interest: Temporal abstraction with interest functions. CoRR, abs/2001.002 2020. hRtp://///arxx www.:oo////a.oo.....//// 00271
Khimya Khetarpal, Zafarali Ahmed, Gheorghe Comanici, and Doina Precup. Temporally abstract partial models, 2021.
Martin Klissarov and Doina Precup. Reward propagation using graph convolutional net- works. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 12895-12908, Cur- Associates, Inc., 2020. URL htttpps://proceddinggss nwwrinu httpeiiig...a.ie.....pppe/////// /c/pp//////pppp//////e/ee//// .pdf
Martin Klissarov, Pierre-Lue Bacon, Jean Harb, and Doina Precup. Learnings options end-to-end for continuous action tasks. CoRR, abs/1712.00004, 2017. URL http::/://aar http:/bbb//111 1712 00004]
Ilya Kostrikov. Pytorch implemetttonn of reinforcement learning algorithms. https://githi:
com/ikostrikko/pyttrchh a2c ppo- -apt---acctt-----------aa-----e- -gail 2018
Kfir Y Levy and Nahum Shimkin. Unified inter and intra options learning using policy gradient methods. In European Workshop on Reinforcement Learning. pages 163-164. Springer, 2011.
Lihong Li, Remi Munos, and Csaba Szepesvari. Toward minimax off-policy value estimation. In Proceedings of the 18th International Conference on Artificial In- telligence and Statistics (AISTATS). JMLR: Workshop and Conference Proceedings,
May 2015. URL https://www mi crosoft. cttm:/eerns/eeeeaarcc///lllaaaaa off wwwe-ooo-lllvlleeeeeseeeeee-eeee www.uueesssmmmmmioii.eoo..
Yao Liu, Pierre-Lue Bacon, and Emma Brunskill. Understanding the curse horizon off- policy evaluation via conditional importance sampling. CoRR, abs/1910.06508, 2019. URL
http::/aaxivv.rr/abbb
Marlos C Machado, Mare Bellemare, and Michael Bowling. laplacian framework for option discovery in reinforcement learning. arXiv preprint arXiv:1703. 170300096,,000, 2017.
12

---

A. Rupam Mahmood, Hado P van Hasselt, and Richard S Sutton. Weighted importance sampling for off poolcyy learning with linear function approximation. Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems. volume 27, Curran Associates, Inc., 2014. URL
.ewrrise www..ips.cc/ppper.. .pff
Ashique Rupam Mahmood, Huizhen Yu, Martha White, and Richard S. Sutton. Emphatic temporal- difference learning. CoRR,
2015, htttt:ttt://xtt/ http://axxxv...g/bbbaa/aa 1507 01569
Timothy A. Mann, Shie Mannor, and Doina Precup. Approximate value iteration with temporally extended actions. J. Artif. Intell. Res., 15:77--338, 2015.
Amy McGovern and Andrew G Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. In ICML, volume I, pages 361-368, 2001.
Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-cut-dynamic discovery of sub-goals in reinforcement learning. In European Conference Machine Learning pages 295-306, Springer, 2002.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. CoRR, abs/1602.01183, 2016. URL http://a/aaa org/ www.ab//.....e....a 01783
Andrew Moore. Efficient Memory-based Learning for Robot Control. PhD thesis, Carnegie Mellon University, Pittsburgh, PA, March 1991.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-effiiient hierarchical reinforce- ment learning. CoRR, Abs//885508896 2018 URL http:://aaraiv..rr///////B/ 08296
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Seffssapperisseeeepploaaoo via disagreement. CoRR, abs/ 0006004161, 2019.
Benjamin Petit, Loren Amdahl-Culleton, Yao Liu, Jimmy Smith, and Pierre-l Luc Bacon. All-action policy gradient methods: A numerical integration approach. CoRR, abs/1910.09093, 2019. URL
bttp:://rxiv.orr//aa /090000000090000
Doina Precup. Temporal abstraction. 2018 URL http:://iideollecuueee.e http://iioo.cttuureeeeeeeeeeeaad dd/eg
abt.raactoaaatttatiiooi..oo.oo.. pdf
Doina Precup. Richard S. Sutton, and Satinder P. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning. ICML '00, page 759-766, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc ISBN 1558607072.
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. CoRR, abs/18 1810. 1158,, 2018. http///t://arxii. http:/awwwv....raaaaa. ab1g//00s/1110...1.1 1at./11810.111333311
Tom Schaul, Dan Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, ICML 15, page 1312-1320. JMLR.org. 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL httt:://ttaxiiii http//awxivv.r.araa//////// www./rgg/////eo/.eo/..o/.eeeeeee aas////s///// 06347
Robin Sibson. Information radius. Zeitschrift für Wahnschheinicchketstteerr und verwandte Gebiete, 1442):149-160, 1969.
Matthew Smith. Herke van Hoof, and Joelle Pineau. An inference-based policy gradient method for learning options. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4703 4712. PMLR, 10-15 Jul 2018. URI, html
mll. presss/ ppress/v0//eet Tieess/880/mmth111 10100mmmth18a www..mmwrr.ess..eee
13

---

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. BtRLattt.///ncommmtcttleeaasssss net/book/
oet. tooeort/theboott-- wwo.bbook22oe 2nd. html
Richard S. Sutton, Satinder Singh, and David Mcallester. Comparing policy-gradient algorithms. IEEE Trans. on Systems, Man, and Cybernetics, 1983.
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS, pages 1057-1063, 1999a.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2)1811-11, 1999b.
Saket Tiwari and Philip Thomas. Natural option critic. CoRR, abs/1 812.01488 2018. URL http://arxiv.orz org/ abs/ 1812. 004888
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In pages 5026-5033, IEEE, 2012. ISBN 978-1-4673-1737-55
atml#TooorvEE12
Harm van Seijen, Hado Hasselt, Shimon Whiteson, and Marco Wiering. theoretical and empirical analysis of expected sarsa. In 2009 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, pages 177-184, 2009. doi: 10.109//DDPL
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pages 3540-3549. PMLR, 2017.
Markus Wulfmeier, Dushyant Rao, Roland Hafner, Thomas Lampe, Abbas Abdolmaleki, Tim
Hertweck, Michael Neunert, Dhruva Tirumala, Noah Y. Siegel, Nicolas Heess, and Martin A. Riedmiller. Data-efficient hindsight policy option learning. CoRR, abs/2007.15588, 2020. URL
https://arxiv org /207 15588
Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. CoRR, abs/1906.03393, 2019. URL http: //arxiv. orgo abaaaaaa//// /90ee//96..... 03393
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values, 2020.
Shangtong Zhang and Shimon Whiteson. DAC: the double actor-critic architecture for learning options. CoRR, abs/1904. 12691, 2019. URL http://arxii org/ ttabs/9994.. 12691
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
14

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/releusing new assets...
(a) your work uses existing assets, did you cite the creators? [No]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "l9lTbyzCYO", "reviewer": "Reviewer_KX8a", "summary": "- This paper examines the possibility of improving sample efficiency in option learning by updating all options simultaneously, rather than the single option chosen for execution. The authors that posit that such an all-options update can improve sample efficiency both when a set of pre-computed options are provided to the agent a priori as well as when the agent must learn the options themselves from scratch along with how to best compose them\n- The primary contributions of this paper center around the theoretical formulation that facilitates updating of multiple options derived from the experiences collected under a single option along with an empirical instantiation of the idea with tabular, linear, and non-linear function-approximation settings.\n- Empirical results are presented confirming the efficacy of the proposed update when a set of pre-computed options are provided as well as when two options are being learned by the agent from scratch. Crucially, the authors are able to report modest performance gain across all tasks in the evaluation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper scores highly on clarity, significance, and (to the best of my knowledge) originality. On that last point, these ideas seem analogous to those in the (flat) reinforcement learning setting [1] which may be an interesting point of exploration for more rigorous theoretical work in this space. However, I do see numerous issues on the axis of quality, with varying degrees of severity. If the authors can suitably resolve them, I would be happy to raise my score.\n\nBased on how the authors have presented options, they have left themselves open to the more general setting where not all options are available at each state (as opposed to the fairly common, but less general, assumption that all options are available from all states). Under this setting, the initiation set does impact the value function definition and distributions outlined in Section 3. Fortunately, this can be fixed easily by simply defining the set of available options, as done for example by Khetarpal et al. 2020, or by introducing an appropriate initiation event and conditioning, as done in the original options paper by Sutton, Precup, & Singh 1999. That said, the rest of the paper seems to be assuming all options are available everywhere and, in fact, perhaps this is necessary to achieve ergodicity in Lemma 1 (otherwise some state-option pairs automatically become unreachable). In any case, the authors should either adjust the definitions accordingly or make this assumption explicit.\n\nOn the topic of Assumption 2, I wonder if there is some level of redundancy? In particular, I don't think one could achieve the ergodicity in Assumption 1 if either the termination function or the option policy yields a probability of zero for any state-option pair. Perhaps the authors should simply add the text of Assumption 2 to Assumption 1 so that the implications of ergodicity are made transparent to readers? That said, it seems like this assumption goes against the spirit of what options are trying to achieve. Naturally, we think of hierarchical multi-task reinforcement-learning agents endowed with a broad, diverse array of skills while the individual tasks presented to the agent may only demand a subset of them at a time. Even in the single-task setting, we expect some skills to have a natural sequentiality such that one would never execute skill B before having reached a particular state from the termination of skill A. Could the authors comment on why Assumption 2 is reasonable given the scale at which we expect hierarchical reinforcement-learning agents to be able to operate? Otherwise, it seems like the long-term possibilities for the proposed algorithm are quite limited.\n\nIn Lemma 1, I would have expected the stationary distributions appearing on the LHS and RHS of Equation 2 to match. Instead, the authors introduce a new distribution on the RHS, \\bar{d}, which hasn't been defined anywhere in the paper. If this is not a typo, could the authors please clarify what this distribution represents? In the rest of this review, I'll assume this was a typo and the two distributions were meant to be the same, evaluated with different arguments on the LHS and RHS.\n\nThe authors correctly introduce new notation in L151 to distinguish the discounted stationary state-option distribution from the distribution defined by Equation 2. Given the analogous discussion for standard policy-gradient methods in (flat) reinforcement learning [2], could the authors comment on the mismatch between these two distributions, with an algorithm only being able to generate samples from the latter, and how that might affect the use of the intra-option policy gradient in their experiments (I suspect the effect would be shared among all the OptionCritic methods evaluated, not just MOC)? This question, while interesting, is perhaps beyond the scope of this work and, accordingly, the authors should not prioritize answering it in their rebuttal if space is constrained.\n\nIn Section 4.4, the introduction of importance-sampling ratios introduces two new assumptions of absolute continuity (one for each ratio used in the equation of that section). From a theoretical perspective, this again comes back to a question of reasonability for these assumptions and why it should be desirable for specific option policies to necessarily retain non-zero probability mass on all actions so as to entertain these off-policy updates? Intuitively, these assumptions seem overly restrictive and unreasonable. From a practical perspective, I wonder if there are any unmentioned reguarlization techniques or other heuristics (for example, entropy regularization) used to ensure that these absolute continuity conditions were met in the experiments?\n\nThe information radius criterion used in Figure 1c seems rather opaque as it is currently discussed. It would be of great benefit to readers if the authors could offer a bit more insight into the measure itself to clarify the importance of the result.\n\nIn Algorithm 1, the upon-arrival probability (roughly line 14) is currently an unbound variable; the authors should include psuedocode for its computation. The same is true of V_\\Omega(s') two lines later although, presumably, this is just taking the expectation on Q_\\Omega with respect to \\mu_z.\n\nOne missing hyperparameter I can't find explicitly stated in each experiment is the total number of options learned. From scanning through, it seems as though no experiment learns more than two options; could the authors please confirm this? A natural follow-up question here is if the authors experimented with more options? Otherwise, it seems a bit difficult to claim that the proposed method does not suffer from issues of diversity when learning useful options (as diversity of two things is a rather low bar to clear). For a domain like FourRooms, I would have wanted one experiment to use precisely four options just to test that MOC does not fall into the degenerate solution of aliasing the four actions of the original MDP. I also wonder if this limitation to two options is done so as to allow the experimental setup to adhere to the aforementioned assumptions of this work; does MOC begin to break down when trying to learn 10 options? 20 options? 30?\n\n===== Post-Rebuttal =====\n\nI thank the authors for their response and clarifying many of the points I raised. I'm optimistic that they can suitably adjust the text in the main paper to reflect these clarifications. \n\nI still find the need to keep non-zero probability mass on all actions/options for all policies to be restrictive. Even in a transfer setting, the margin for uncertainty mentioned in the author response could simply pertain to a decision of whether or not the agent should attempt to expand its library of options/skills in this task as opposed to discerning optimal behavior solely with the options on hand. \n\nMore importantly, I believe that further empirical investigation is needed to investigate MOC when learning far more than 2 options, and on domains far more complex than MountainCar. The now classic OptionCritic paper seems to go as far as 8 and, genuinely, if the premise of MOC to allow for updates across all options holds, then we should be able to push far past that point or, in an equally-important fashion, shine a spotlight on precisely where the breaking point resides. Even if the authors were to run such additional experiments (which I am not asking them to do), the critical piece would be the result and associated exposition in the main paper, which currently cannot be judged in the post-rebuttal phase. I'm incrementing my score to reflect the clarifications made, however, I believe the authors would be best served in providing a more rigorous empirical study with more learned options to sharpen their contribution. Hopefully, if the paper is accepted here, the authors will consider doing this anyways for the camera-ready version as I suspect this question/point of concern is a natural one apparent to anyone in the community hoping to leverage this approach.\n\n[1] Ciosek, Kamil, and Shimon Whiteson. \"Expected Policy Gradients for Reinforcement Learning.\" Journal of Machine Learning Research 21 (2020): 1-51.\n\n[2] Nota, Chris, and Philip S. Thomas. \"Is the Policy Gradient a Gradient?.\" In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 939-947. 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe this is clear from my main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "iOD-E7qicOw", "reviewer": "Reviewer_Q89w", "summary": "This paper presents a method to augment Option-Critic based methods with a modified objective that allows all options to be updated with every sample, as opposed to just the option that was executed. Performance improvements over Option-Critic and other baselines on multiple environments, the flexibility of their objective to augment many similar methods, and evaluations that are mostly comprehensive are why I recommend this paper to be accepted.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Paper Strengths\n**Motivation**\nThe motivation is intuitive and sensible. We want to be able to simultaneously update many options, instead of just one, so that we can encourage better sample efficiency and possibly more useful options.\n\n**Mathematical Formulation**\nThe assumptions made to formulate the objective are sensible and the modified objectives are formulated in a principled manner starting from the one-step TD update.\n\n**Method**\nThe method is a series of objectives that can be applied across many option-learning architectures. It is also, to my knowledge, novel.\n\n**Experiments**\nThe authors test across 3 domains, with multiple environments in the continuous control domain. They also perform interesting analysis on Ant, with option visualizations that show the usefulness of the second option. There are modest to strong performance improvements across their environments over baselines. \n\n## Paper Weaknesses\n**Experiments**\nOne area where HRL/Option methods shine is sparse-reward, long-horizon tasks in continuous control (typically navigation environments). I would’ve liked to see evaluations on these types of tasks in the continuous control section. It’s also possible that visualizations on those environments would show interesting learned option behaviors.\n\n## Questions\nShouldn’t Eq 1 be either $b_t + A_t \\theta_t$ or there be $A$ be negated after line 117?\n\nWhy do you think the variance across runs is reduced despite any potential variance introduced by importance sampling?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No explicit discussion on limitations, I think the authors should add this to the discussion section.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "RGT841GxUwR", "reviewer": "Reviewer_XUis", "summary": "The paper proposes a principled way to update all options at once in an option-learning framework, and conducts extensive experiments to support the proposed algorithm.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I recommend accepting this paper for publication, conditional on the issues listed below being addressed in the final version.\n\n**Quality:**\n\nOverall the quality is very good. The proposed method is principled, and the experiments are numerous, extensive, and well-done.\n\n**Clarity:**\n\nThe paper is written very clearly and is easy to understand. There are some grammatical errors throughout (the ones I noticed are listed below).\n\n**Significance:**\n\nThe proposed method is very general and very effective, making it likely to be significant moving forward.\n\n**Originality:**\n\nThe proposed algorithm is novel to the best of my knowledge.\n\n**Misc. comments, questions, and suggestions for improvement:**\n\nLine 89: should be \"selected\"\n\nLine 99: \"This options\" should be \"This option\"\n\nIt's good to comment on the assumptions made in the paper, explaining why they're made, whether they're common, how restictive they are, etc.\n\nCould the authors justify Assumption 2, and briefly comment on it in the paper? Assumption 2 seems to conflict with the idea of an initiation set by implying that the initiation set for all options must be all states. Presumably the initiation set was included in the options framework for a reason, so it would be good to comment on why this assumption is being made. Is it a simplifying assumption analogous to assuming that all actions are available in all states?\n\nLine 130: \"each\" is repeated.\n\nThe discount factor can be interpreted as the probability of the agent terminating at each timestep. So why is the paper using discounting in addition to the termination function of each option?\n\nLine 177: \"hyperparamter\" should be \"hyperparameter\". This also occurs in the Figure 2 caption.\n\nLine 191: \"that\" should be \"than\"\n\nLine 197: Missing a \"to\"\n\nLine 205: \"fo\" should be \"of\"\n\nSection 4.4: Using importance sampling in this way can lead to high-variance updates, right? Is this variance issue somewhat mitigated by the hyperparameter $\\eta$ introduced in section 4.3? Additionally, using importance sampling rules out deterministic policies, which is a limitation of the algorithm that wasn't mentioned in the paper. In fact, the appendix describes how the original FourRooms options had to be modified to be stochastic so that they would work with importance sampling. It would be better to be clear about how exactly they were modified, and how incorporating importance sampling limits the policies allowed.\n\nLine 234: \"a good separation of the state-state has been learned.\"\n\nFigures: What are the shaded regions? Confidence intervals? Standard error? It's best to plot confidence intervals because non-overlapping confidence intervals imply statistically significant results, whereas non-overlapping standard error does not. That way the reader can see at a glance whether results are statistically significant or not. If the confidence intervals end up overlapping, it can be ok to lower the confidence level; it still provides some evidence to support the proposed method.\n\nExperiments: When a paper uses \"ours\" (\"our approach\", \"our method\", etc.) excessively it sounds like an annoying advertisement. Framing experiments this way makes it feel like the authors' only goal is to showcase the strengths of the proposed method, instead of conducting a fair and impartial comparison. What's weird is that the experiments seem to be done pretty fairly, so why the excessive salesmanship? It was starting to really annoy me by the end of the paper.\n\nRelatedly, the discussion of some of the figures (Figure 2a, Figure 5a for example) feels biased and bordering on overclaiming. The results are really good; there is no need to exaggerate (discussion of figure 2a), and it's ok to point out situations where MOC took longer to learn initially (figure 5a)---it could be an avenue for further research.\n\nFigure 2: What value of $\\eta$ was used in 2a?\n\nLine 259: should be \"number of updates\"\n\nAppendix A.3.1: The caption for Table 2 says \"FourRooms\" when it should be sparse mountain car.\n\nI was initially wary of the introduction of the hyperparameter $\\eta$, because often papers will introduce a new hyperparameter, search for the best-performing value, and then claim a performance improvement with no guidance on how to set the hyperparameter on a new environment without testing. However, I was very happy to see that $\\eta=1$ was (close to) the best performing value for the function approximation setting across different forms of function approximation and different base algorithms, which is really cool!\n\nLine 327: \"simultaenously\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper did not seem to address limitations at all. It would be good to state that using importance sampling rules out deterministic policies, clarify whether assumption 2 rules out the use of initiation sets, and state any further limitations that the authors are aware of.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an intuitive approach to updating multiple options together while learning in a hierarchical deep reinforcement learning setting. The approach seems principled, and performs well empirically.
All reviewers have advocated for acceptance, with the expectation that the requested clarifications in the paper are made.
Decision: Accept (Spotlight)