Flexible Option Learning

1 Introduction . Temporal abstraction is a fundamental component of intelligent agents as it allows for explicit reasoning at different timescales . The options framework [ Sutton et al. , 1999b ] provides a clear formalism for such abstractions and proposes efficient ways to learn directly from the environment ’ s reward signal . As the agent needs to learn about the value of a possibly large number of options , it is crucial to maximize each interaction to ensure sample efficiency . Sutton et al . [ 1999b ] therefore propose the intra-option value learning algorithm , which updates all consistent options simultaneously from a single transition . Recently there have been important developments on how to learn or discover options from scratch when using function approximation [ Bacon et al. , 2016 , Smith et al. , 2018 , Riemer et al. , 2018 , Bagaria and Konidaris , 2019 , Zhang and Whiteson , 2019 , Harutyunyan et al. , 2017 ] . However , most of recent research only updates a single option at a time , that is , the option selected by the agent in the sampled state . This is perhaps due to the fact that consistency between options is less likely to arise naturally when using function approximation . We propose a scalable method to better exploit the agent ’ s experience by updating all relevant options , where relevance is defined as the likelihood of the option being selected . We present a decomposition of the state-option distribution which we leverage to remain consistent in the function approximation setting while providing performance improvements similarly to the ones shown in the tabular setting [ Sutton et al. , 1999b ] . When an option set is given to the agent and only the value of each option remains to be learned , updating all relevant options lets the agent determine faster when to apply each of them . In the case where all options components are learned from scratch , our approach can additionally help mitigate the issue of degenerate solutions [ Harb et al. , 2017 ] . Indeed , such solutions are likely due to the `` rich-get-richer '' phenomenon where an updated option has more chance of being picked again 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . when compared to a randomly initialized option . By avoiding degenerate solutions , one can obtain temporally extended options which leads to meaningful and interpretable behavior . Unlike recent approaches that leverage inference to update all options [ Daniel et al. , 2016 , Smith et al. , 2018 , Wulfmeier et al. , 2020 ] , our method is naturally compatible with many state of the art policy optimization frameworks [ Mnih et al. , 2016 , Schulman et al. , 2017 , Fujimoto et al. , 2018 ] and requires no additional estimators . We empirically verify the merits of our approach on a wide variety of domains , ranging from gridworlds using tabular representations [ Sutton et al. , 1999b , Bacon et al. , 2016 ] , control with linear function approximation [ Moore , 1991 ] , continuous control [ Todorov et al. , 2012 , Brockman et al. , 2016 ] and vision-based navigation [ Chevalier-Boisvert , 2018 ] . 2 Related Work . The discovery of meaningful temporally extended behaviour has previously been explored through many different avenues . One such avenue is by learning to reach goals in the environment . This can be done in various ways such as identifying bottleneck states [ McGovern and Barto , 2001 , Şimşek et al. , 2005 , Menache et al. , 2002 , Machado et al. , 2017 ] or through feudal RL [ Dayan and Hinton , 1993 , Vezhnevets et al. , 2017 , Nachum et al. , 2018 ] . Another avenue of research is through the options framework [ Sutton et al. , 1999b ] where the discovery of temporal abstractions can be done directly through the environment ’ s reward signal . This framework has lead to recent approaches [ Levy and Shimkin , 2011 , Bacon et al. , 2016 , Harb et al. , 2017 , Riemer et al. , 2018 , Tiwari and Thomas , 2018 ] that rely on policy gradient methods to learn the option policies . As policy gradient methods are derived for the on-policy case , these approaches have only updated the sampled option within a state . Our work complements them by providing a way to transform single-option updates into updates that consider all relevant options without introducing any additional estimators . Within the options framework , Harutyunyan et al . [ 2017 ] have recently decoupled the way options terminate from the target termination functions . This allowed for learning longer options as if their duration was less extended in time . This can be beneficial as shorter options are shown to produce better solutions and longer options are preferred for planning [ Mann et al. , 2015 ] . By contrast , our approach looks at updating all relevant options through the experience generated by any such option , irrespective of the duration . Another line of work is proposed by Daniel et al . [ 2016 ] in which options are considered latent unobserved variables which allows for updating all options . The authors adopt an expectationmaximization approach which assumes a linear structure of the option policies . Smith et al . [ 2018 ] alleviate this assumption and derive a policy gradient objective that improves the data-efficiency and interpretability of the learned options . A conceptually related work is proposed by Wulfmeier et al . [ 2020 ] which leverages dynamic programming to infer option and action probabilities in hindsight . As these approaches rely on inference methods to perform updates , it is not directly compatible with other option learning methods nor with various policy optimization algorithms , which is another important feature of our approach . 3 Background and Notation . We assume a Markov Decision ProcessM , defined as a tuple 〈S , A , r , P 〉 with a finite state space S , a finite action space A , a scalar reward function r : S ×A → Dist ( R ) and a transition probability distribution P : S × A → Dist ( S ) . A policy π : S → Dist ( A ) specifies the agent ’ s behaviour , and the value function is the expected discounted return obtained by following π starting from any state : Vπ ( s ) =̇Eπ [ ∑∞ i=t γ t−ir ( Si , Ai ) ∣∣St = s ] , where γ ∈ [ 0 , 1 ) is the discount factor . The policy gradient theorem [ Sutton et al. , 1999a ] provides the gradient of the expected discounted return from an initial state distribution d ( s0 ) with respect to a parameterized stochastic policy πζ ( ·|s ) : ∂J ( ζ ) ∂ζ = ∑ s d γ ( s ) ∑ a ∂πζ ( a|s ) ∂ζ Qπ ( s , a ) where d γ ( s ) = ∑ s0 d ( s0 ) ∑∞ t=0 γ tPπ ( St = s|S0 = s0 ) is the discounted state occupancy measure . Options . The options framework [ Sutton et al. , 1999b ] provides a formalism for abstractions over time . An option within the option set O is composed of an intra-option policy π ( a|s , o ) for selecting actions , a termination function β ( s , o ) to determine how long an option executes and an initiation set I ( s , o ) to restrict the states in which an option may be selected . It is then possible to choose which option to execute through the policy over options µ : S → Dist ( O ) . The set of options along with the policy over options defines an option-value function which can be written recursively as , QO ( s , o ) =̇ ∑ a π ( a|s , o ) r ( s , a ) + γ ∑ s′ , o′ Pπ , µ , β ( s ′ , o′|s , o ) QO ( s′ , o′ ) where Pπ , µ , β ( s′ , o′|s , o ) is the probability of transitioning from ( s , o ) to any other state-option pair in one step in the MDP . This probability is defined as , Pπ , µ , β ( s ′ , o′|s , o ) =̇pµ , β ( o′|s′ , o ) ∑ a P ( s′|s , a ) π ( a|s , o ) where pµ , β ( o′|o , s′ ) = β ( s′ , o ) µ ( o′|s′ ) + ( 1 − β ( s′ , o ) ) δo , o′ is the upon-arrival option probability and δ is the Kronecker delta . Intuitively , this distribution considers whether or not the previous option terminates in s′ , and accordingly either chooses an option or continues with the previous one . 4 Learning Multiple Options Simultaneously . In this work we build upon the call-and-return option execution model . Starting from a state s , the agent picks an option according to the policy over options o ∼ µ ( ·|s ) . This option then dictates the action selection process through the intra-option policy a ∼ π ( ·|s , o ) . Upon arrival at the next state s′ , the agent queries the option ’ s termination function β ( s′ , o ) to decide whether the option continues or a new option is to be picked . This process then continues until the episode terminates . This execution model , coupled with the assumption that the options are Markov , will let us leverage a notion of how likely an option is to be selected , irrespective of the actual sampled option . This measure will then be used to weight the updates of each possible option . 4.1 Option Evaluation . In this section we present an algorithm that updates simultaneously the option value functions of all the options , written as QO ( s , o ; θ ) and parameterized by θ. Let ’ s recall the on-policy one-step TD update for state-option pairs , θt+1 = θt + α ( Rt+1 + γQ̂ ( St+1 , Ot+1 ) − Q̂ ( St , Ot ) ) ∇θtQ̂ ( St , Ot ) θt+1 = θt + α ( Rt+1 + γθ > t φ ( St+1 , Ot+1 ) − θ > t φ ( St , Ot ) ) φ ( St , Ot ) = θt + α ( Rt+1φ ( St , Ot ) ︸ ︷︷ ︸ bt +φ ( St , Ot ) ( φ ( St , Ot ) − γφ ( St+1 , Ot+1 ) ) > ︸ ︷︷ ︸ At θt ) θt+1 = θt + α ( bt − Atθt ) ( 1 ) where the option value functions are linear functions , i.e . QO ( s , o ; θ ) =̇θ > t φ ( s , o ) of the parameters θt at time t and the features φ ( s , o ) . Since the random variables within the update rule , summarized through the vector bt and the matrix At , are sampled from interacting with the environment , the stability and convergence properties can be verified by inspecting its expected behaviour in the limit , written as , θt+1 = θt + α ( b− Aθt ) where A = limt→∞ E [ At ] and b = limt→∞ E [ bt ] . As the matrix A has a multiplicative effect on the parameters , it will be crucial for this matrix to be positive-definite in order to assure the stability of the algorithm , and subsequently its convergence . In the on-policy case , this matrix is written as : A = ∑ s , o lim t→∞ Pπ , µ , β ( St = s , Ot = o ) φ ( s , o ) ( φ ( s , o ) − γ ∑ s′ , o′ Pπ , µ , β ( s ′ , o′|s , o ) φ ( s′ , o′ ) ) > where Pπ , µ , β ( St = s , Ot = o ) is the probability of being in a particular state-option pair at time t. In order to update all options , we will exploit the form of this state-option distribution to preserve the positive definite property of the matrix A . In the following lemma we first show that this limiting distribution exists under some assumptions and we derive a useful decomposition . To do so , we introduce the following two assumptions . Regarding Assumption 1 , it is common in standard RL to assume that the policy induces ergodicity on the state space , which we extend to all option policies . Assumption 2 is specific to the hierarchical setting and is a way to ensure irreducibility , that is , that each state-option pair is reachable . It would be possible to find less strict assumptions by having access to more information about how the MDP is constructed . Assumption 1 . The option policies { π ( a|s , o ) : o ∈ O } each induce an ergodic Markov chain on the state space S. Assumption 2 . The termination function β ( s , o ) and policy over options µ ( o|s ) have strictly positive probabilities in all state-option pairs . Lemma 1 . Under assumptions 1-2 the following limit exists , lim t→∞ Pπ , µ , β ( St = s , Ot = o|S0 , O0 ) = dπ , µ , β ( s , o ) ( 2 ) and the augmented chain process is an ergodic Markov chain in the augmented state-option space . Moreover the stationary distribution can be decomposed in the following form , dπ , µ , β ( s , o ) = ∑ ō d̄π , µ , β ( ō , s ) pµ , β ( o|s , ō ) ( 3 ) where pµ , β ( o|s , ō ) is the probability over options upon arriving in a state . Remarks . In the derivation we have avoided mentioning the concept of initiation sets such that we would not encumber the notation and presentation . The derivation would be identical if we had assumed that the initation set for each option is equal to the state space , which has been done for most of recent work on options [ Bacon et al. , 2016 ] . Another way to reconcile our method with the original options framework would be to consider interest functions [ Khetarpal et al. , 2020 ] , which are a generalization of the concept of initiation sets . We could then add an assumption similar to Assumption 2 on the interest function by only allowing positive values . This would let us heavily favor a subset of options for a certain region of the state space . However , this assumption would still preclude us from using hard constraints on the option set , such as option affordances [ Khetarpal et al. , 2021 ] . More empirical evidence would be necessary to investigate whether this is a limitation in practice . Ideally , if we seek to update all options in a state , we could weight each of these updates by the stationary distribution dπ , µ , β ( s , o ) . However , in most case , it is impractical to assume access to this distribution . A possible solution would be to estimate this quantity by making use of recent approaches [ Hallak and Mannor , 2017 , Xie et al. , 2019 , Zhang et al. , 2020 ] . Instead , we propose to leverage the decomposition presented in Equation 3 . Indeed , this decomposition highlights the dependency on the upon-arrival option distribution , pµ , β ( o|s , ō ) , which is a known quantity as it is only dependent of the agent ’ s policy over options µ ( o|s ) and termination function β ( s , ō ) . Although d̄π , µ , β ( s , ō ) is itself unknown , we can collect its samples by adopting the call-and-return option execution model . More concretely , under this model , an option which is previously selected leads us to a state of interest at time t. These random variables , being sampled on-policy , follow the distribution d̄π , µ , β ( s , ō ) . Upon arriving in any state of interest St with option Ot−1 , although an option Ot will be sampled according to call-and-return , we can use the distribution pµ , β ( Ot = o|St , Ot−1 ) to estimate how likely it would have been for each option o to be selected . The update rule would become , θt+1 = θt + α ∑ o pµ , β ( Ot = o|St , Ot−1 ) ( E [ Ut|St , Ot = o ] − θ > t φ ( St , Ot = o ) ) φ ( St , Ot = o ) ( 4 ) where E [ Ut|St = s , Ot = o ] is the target for a given state and option pair . In Section 4.3 we show how to obtain this quantity without using any additional estimation or simulation-resetting . Our update rule would then preserve the positive-definite property of the matrix A .

Summary: - This paper examines the possibility of improving sample efficiency in option learning by updating all options simultaneously, rather than the single option chosen for execution. The authors that posit that such an all-options update can improve sample efficiency both when a set of pre-computed options are provided to the agent a priori as well as when the agent must learn the options themselves from scratch along with how to best compose them - The primary contributions of this paper center around the theoretical formulation that facilitates updating of multiple options derived from the experiences collected under a single option along with an empirical instantiation of the idea with tabular, linear, and non-linear function-approximation settings. - Empirical results are presented confirming the efficacy of the proposed update when a set of pre-computed options are provided as well as when two options are being learned by the agent from scratch. Crucially, the authors are able to report modest performance gain across all tasks in the evaluation.