Option Discovery using Deep Skill Chaining

1 INTRODUCTION . Hierarchical reinforcement learning ( Barto & Mahadevan , 2003 ) is a promising approach for solving long-horizon sequential decision making problems . Hierarchical methods lower the decision making burden on the agent through the use of problem specific action abstractions ( Konidaris , 2019 ) . While the use of temporally extended actions , or options ( Sutton et al. , 1999 ) , has been shown to accelerate learning ( McGovern & Sutton , 1998 ) , there remains the question of skill discovery : how can agents autonomously construct useful skills via interaction with the environment ? While a large body of work has sought to answer this question in small discrete domains , skill discovery in high-dimensional continuous spaces remains an open problem . An early approach to skill discovery in continuous-state environments was skill chaining ( Konidaris & Barto , 2009b ) , where an agent constructs a sequence of options that target a salient event in the MDP ( for example , the goal state ) . The skills are constructed so that successful execution of each option in the chain allows the agent to execute another option , which brings it closer still to its eventual goal . While skill chaining was capable of discovering skills in continuous state spaces , it could only be applied to relatively low-dimensional state-spaces with discrete actions . We introduce a new algorithm that combines the core insights of skill chaining with recent advances in using non-linear function approximation in reinforcement learning . The new algorithm , deep skill chaining , scales to high-dimensional problems with continuous state and action spaces . Through a series of experiments on five challenging domains in the MuJoCo physics simulator ( Todorov et al. , 2012 ) , we show that deep skill chaining can solve tasks that otherwise can not be solved by nonhierarchical agents in a reasonable amount of time . Furthermore , the new algorithm outperforms state-of-the-art deep skill discovery algorithms ( Bacon et al. , 2017 ; Levy et al. , 2019 ) in these tasks . 2 BACKGROUND AND RELATED WORK . Sequential decision making problems can be formalized as Markov Decision Processes ( MDPs ) . We consider goal-oriented episodic MDPs , where S denotes the state space , A is the action space , R is the reward function , T is the transition function , γ is the discount factor and g ∈ S is the terminating goal state ( Sutton & Barto , 2018 ) . Unlike goal-conditioned algorithms ( Sutton et al. , 2011 ; Schaul et al. , 2015 ) , we do not require that g be known ; instead we assume access to an indicator function 1g : s ∈ S −→ { 0 , 1 } which the agent can query to determine if it has reached the MDP ’ s goal . 1Video of learned policies : https : //youtu.be/MGvvPmm6JQg 2Code : https : //github.com/deep-skill-chaining/deep-skill-chaining One way to learn a policy in an MDP is to first learn an action-value function . The action-value function Qπ ( st , at ) is defined as the expected sum of discounted future rewards if the agent takes action at from st and then follows policy π thereafter : Qπ ( st , at ) = Eπ [ rt + γmaxat+1 Qπ ( st+1 , at+1 ) ] . Q-learning ( Watkins & Dayan , 1992 ) is a commonly used off-policy algorithm that uses the actionvalue function for control through a greedy policy π ( st ) = argmaxat Q ( st , at ) . Inspired by recent success in scaling Q-learning to high-dimensional spaces ( Mnih et al. , 2015 ; Van Hasselt et al. , 2016 ; Lillicrap et al. , 2015 ; Tesauro , 1994 ) , we learn the action-value functionQπφ ( st , at ) using non-linear function approximators parameterized by φ , by minimizing the loss L ( φ ) = Eπ [ ( Qφ ( st , at ) − yt ) 2 ] where the Q-learning target yt is given by the following equation ( Van Hasselt et al. , 2016 ) : yt = rt + γQφ′ ( st+1 , argmax at+1 Qφ ( st+1 , at+1 ) ) . ( 1 ) Deep Q-Learning ( DQN ) ( Mnih et al. , 2015 ) casts minimizingL ( φ ) as a standard regression problem by using target networks ( parameterized by φ′ ) and experience replay ( Lin , 1993 ) . 2.1 THE OPTIONS FRAMEWORK . The options framework ( Sutton et al. , 1999 ) models skills as options . An option o consists of three components : ( a ) its initiation condition , Io ( s ) , which determines whether o can be executed in state s , ( b ) its termination condition , βo ( s ) , which determines whether option execution must terminate in state s and ( c ) its closed-loop control policy , πo ( s ) , which maps state s to a low level action a ∈ A. Augmenting the set of available actions with options results in a Semi-Markov Decision Process ( SMDP ) ( Sutton et al. , 1999 ) where the next state depends on the current state , action and time . 2.2 SKILL DISCOVERY ALGORITHMS . Skill discovery has been studied extensively in small discrete domains ( McGovern & Sutton , 1998 ; Şimşek & Barto , 2004 ; Şimşek et al. , 2005 ; Bakker & Schmidhuber , 2004 ; Schmidhuber , 1991 ; Pickett & Barto , 2002 ; Dietterich , 2000 ) . Recently however , there has been a significant body of work aimed at discovering skills in continuous spaces . Option-critic methods : Option-Critic ( Bacon et al. , 2017 ) uses an end-to-end gradient based algorithm to learn options in high-dimensional continuous spaces . Option-Critic was a substantial step forward in skill discovery and led to a family of related methods ( Klissarov et al. , 2017 ; Tiwari & Thomas , 2019 ; Riemer et al. , 2018 ; Liu et al. , 2017 ; Jain et al. , 2018 ) . Proximal Policy Option Critic ( PPOC ) ( Klissarov et al. , 2017 ) extends Option-Critic to continuous action spaces and is the version of Option-Critic that we compare against in this paper . Our method bypasses two fundamental shortcomings of the Option-Critic framework : ( a ) unlike Option-Critic , we explicitly learn initiation sets of options and thus do not assume that all options are executable from everywhere , and ( b ) we do not treat the number of skills required to solve a task as a fixed and costly hyperparameter . Instead , our algorithm flexibly discovers as many skills as it needs to solve the given problem . Feudal methods : An alternative to the options framework is Feudal RL ( Dayan & Hinton , 1993 ) , which creates a hierarchy in which managers learn to assign subgoals to workers ; workers take a subgoal state as input and learn to reach it . Feudal Networks ( FuN ) ( Vezhnevets et al. , 2017 ) used neural networks to scale the Feudal-RL framework to high-dimensional continuous spaces ; it was extended and outperformed by HIRO ( Nachum et al. , 2018 ) in a series of control tasks in the MuJoCo simulator . More recently , Hierarchical Actor-Critic ( HAC ) ( Levy et al. , 2019 ) outperformed HIRO in a similar suite of continuous control problems . While HIRO relies on having a dense “ distanceto-goal ” based reward function to train both levels of their feudal hierarchy , HAC ’ s use of Hindsight Experience Replay ( HER ) ( Andrychowicz et al. , 2017 ) allows it to work in the more general sparsereward setting . Given its strong performance in continuous control problems and its ability to learn effectively in sparse-reward settings , we compare against HAC as a representative feudal method . Learning backward from the goal : The idea of sequencing locally applicable controllers is well established in robotics and control theory in the form of pre-image backchaining ( Kaelbling & LozanoPérez , 2017 ) and LQR-Trees ( Tedrake , 2009 ) . Such methods either require individually engineered control loops or a model of the system dynamics . Our work fits in the model-free RL setting and thus requires neither . More recently , reverse curriculum learning ( Florensa et al. , 2017 ) also learns backward from the goal . However , they define a curriculum of start states to learn a single policy , rather than learning skills . Relay Networks ( Kumar et al. , 2018 ) segment the value function backward from the goal using a thresholding scheme , which makes their method reliant on the accurate estimation of the value function . By contrast , our algorithm is agnostic to errors in value estimation , which are unavoidable when using function approximation in high-dimensional spaces . Planning with learned skills : Options have been shown to empirically speed up planning in several domains ( Silver & Ciosek , 2012 ; Jinnai et al. , 2019 ; James et al. , 2018 ; Francis & Ram , 1993 ; Konidaris , 2016 ; Sharma et al. , 2019 ) . However , Konidaris et al . ( 2018 ) show that for resulting plans to be provably feasible , skills must be executable sequentially . While they assume that such skills are given , we show that they can be autonomously discovered in high-dimensional spaces . 3 DEEP SKILL CHAINING . Deep skill chaining ( DSC ) is based on the intuition that it is easier to solve a long-horizon task from states in the local neighborhood of the goal . This intuition informs the first step of the algorithm : create an option that initiates near the goal and reliably takes the agent to the goal . Once such an option is learned , we create another option whose goal is to take the agent to a state from which it can successfully execute the first option . Skills are chained backward in this fashion until the start state of the MDP lies inside the initiation set of some option . The inductive bias of creating sequentially executable skills guarantees that as long as the agent successfully executes each skill in its chain , it will solve the original task . More formally , skill chaining amounts to learning options such that the termination condition βoi ( st ) of an option oi is the initiation condition Ioi−1 ( st ) of the option that precedes it in its chain . Our algorithm proceeds as follows : at time t , the policy over options πO : st ∈ S −→ o ∈ O determines which option to execute ( Section 3.2 ) . Control is then handed over to the selected option oi ’ s internal policy πoi : s ∈ S −→ at ∈ R|A| . πoi outputs joint torques until it either reaches its goal ( βoi : = Ioi−1 ) or times out at its predetermined budget T ( Section 3.1 ) . At this point , πO chooses another option to execute . If at any point the agent reaches the goal state of the MDP or the initiation condition of a previously learned option , it creates a new option to target such a salient event . The machinery for learning the initiation condition of this new option is described in Section 3.3 . We now detail the components of our architecture and how they are learned . Readers may also refer to Figures 4 & 7 and the pseudo-code in Appendix A.5 to gain greater intuition about our algorithm .

Summary: This paper studies the problem of learning suitable action abstractions (i.e., options or skills) that can be composed hierarchically to solve control tasks. The starting point for the paper is the (classic) observation that one skill should end where another can start. The paper then proposes a recursive algorithm for learning skills that obey this property. After finding some number of trajectories that reach the goal, the last few states of these trajectories are taken to define the initiation set for the ultimate skill and the termination set for the penultimate skill. The procedure is repeated, yielding a sequence (a "chain") of skills that extends from the initial state distribution to the terminal state. The fact that the number of skills is not defined apriori seems to be a strength, and the extension to trees of skills is neat.