Incremental Hierarchical Reinforcement Learning with Multitask LMDPs

Exploration is a well known challenge in Reinforcement Learning . One principled way of overcoming this challenge is to find a hierarchical abstraction of the base problem and explore at these higher levels , rather than in the space of primitives . However , discovering a deep abstraction autonomously remains a largely unsolved problem , with practitioners typically hand-crafting these hierarchical control architectures . Recent work with multitask linear Markov decision processes , allows for the autonomous discovery of deep hierarchical abstractions , but operates exclusively in the offline setting . By extending this work , we develop an agent that is capable of incrementally growing a hierarchical representation , and using its experience to date to improve exploration . 1 INTRODUCTION . The exploration-exploitation trade-off is much studied in the Reinforcement Learning ( RL ) literature . A central question regards how to explore efficiently in large state spaces . In this direction , a number of classical exploration ideas have emerged for problems with enumerable state spaces which could be represented in tabular format Roger ( 1999 ) . In fact , approximately optimal exploration is achievable in these settings Kolter & Ng ( 2009 ) . When the state space is large and neural networks are used to approximate the value function , an additional suite of techniques exist to promote exploration Stadie et al . ( 2015 ) ; Tang et al . ( 2016 ) ; Plappert et al . ( 2017 ) . An always good-in-principle idea is to first construct an abstract representation of the problem , and explore at this higher level rather than in the space of primitives . This might correspond to a person exploring a house by choosing to explore at the level of rooms , rather than exploring exclusively inch-by-inch . Intuitively , this suggests that exploring a large state space may be best conducted at different levels of granularity , corresponding to different layers of abstraction . The challenge here is that finding suitable abstractions autonomously is , in general , largely an unsolved problem , although some progress has been made in this direction Vigorito & Barto ( 2010 ) ; Bacon & Precup ( 2015 ) . Additionally , while further layers of abstraction would enhance the agent ’ s ability to explore at these various scales , many architectures in the hierarchical reinforcement learning literature focus on just two layers of control , e.g. , the options framework Sutton & Singh ( 2015 ) ; Machado et al . ( 2018 ) . Recent work using the linearly-solvable Markov decision process ( LMDP ) framework provides a mechanism for autonomously learning deeper control hierarchies Saxe et al . ( 2017 ) . However , this method operates in the off-line setting . By incorporating ideas from incremental learning on static classification tasks , we develop an agent that is able to both build , and utilize , a hierarchical architecture fully online . Our agent operates by periodically increasing the capacity of its hierarchical representation , and allowing subsequent experience to fine-tune the new weights . In so much as an analogy to the well-known options framework is helpful , the new approach can be thought of as periodically adding a randomly initialized options policy , and allowing subsequent experience to adjust all of the available options in light of the new-found capacity . We demonstrate that learning the hierarchical representation online is not only possible , but can actually aid exploration when coupled with a simple count-based exploration boost . 2 PRELIMINARIES . The LMDP framework Todorov ( 2006 ) ; Kappen ( 2005 ) considers a special class of MDPs for which the Bellman optimality condition becomes linear in the exponentiated cost-to-go . In this formalism , the MDP is defined as a three-tuple L = 〈S , Pπ̄ , R〉 , where S = [ 1 , . . . , N ] is a set of states , and Pπ̄ : S × S → [ 0 , 1 ] is the so-called passive transition probability between states . This transition probability is defined as the resultant transition probability under some reference policy π̄ . And R : S → R is the expected instantaneous reward . The reference policy π̄ is typically taken to be the uniformly random policy , corresponding to the entropy-regularized RL problem Schulman et al . ( 2017 ) ; Haarnoja et al . ( 2017 ) . Intuitively , the passive dynamics can be thought of as the diffuse process followed by the agent in the absence of a specified control . In this setting , the action taken by the agent is a full distribution over next states a ( ·|s ) ∈ RN , which acts to shift the passive transition probability by redistributing probability mass ; making preferred transitions more likely , and disfavoured transitions less likely . A control cost is associated with this choice : actions corresponding to distributions over next states that are very different from the passive transition probability distribution are expensive , while those that are similar are cheap . In this way the problem is said to be regularized by the reference policy . Intuitively , this can be thought of as specifying an intrinsic preference for energy-efficient actions , by regularizing the controlled transition probabilities towards the reference policy . The LMDP has rewards Ri ( s ) for each interior state , and Rb ( s ) for each boundary state in the finite exit formulation . The LMDP can be solved by finding the so-called desirability function z ( s ) = eV ( s ) , specified in terms of the state value function . The optimal policy can then be computed in closed form as : a∗ ( s′|s ) = Pπ̄ ( s ′|s ) z ( s′ ) G [ z ] ( s ) , ( 1 ) where the normalizing constant G [ z ] ( s ) = ∑ s′ Pπ̄ ( s ′|s ) z ( s′ ) Todorov ( 2009 ) . Intuitively , the hard maximization of standard MDP has been replaced by a soft maximization log ( ∑ exp ( · ) ) , and the continuous action space enables closed form computation of the optimal policy . While in principle any reference policy could be used ( perhaps coming from an expert demonstration ) , entropy regularization is a common choice , and has been shown to recover more adaptive , multi-modal policies in large domains Haarnoja et al . ( 2017 ) . As an illustrative example , consider a robot whose task it is to navigate around some object . Suppose that two equally valid solutions exist , corresponding to a trajectory around the obstacle to the left , and another to the right . Where a standard DQN-style agent would collapse on one path or the other , in the entropy regularized problem the optimal solution is a stochastic policy in which the agent chooses which path to take at random . Suppose that , after training , the path to the left was blocked-off by a new obstacle . While the DQN-style agent would struggle to recover from its firm commitment to the left path , the entropy-regularized agent is able to adapt easily . It is worth noting that unlike the standard MDP formulation , the optimal policies uncovered in the LMDP formulation are stochastic . These have equivalent interpretations as being stochastic policies over deterministic actions , or as deterministic policies over stochastic actions . Explicitly , although the LMDP formalism transforms away the notion of discrete actions , the optimal solutions to LMDP problems may always be realized in terms of stochastic policies over the discrete actions of the more familiar MDP formulation . 2.1 THE MULTITASK LMDP . The linearity of the Bellman optimality condition implies a natural form of policy composition . Suppose that we solve two LMDPs L1 = 〈S , Pπ̄ , [ Ri , R1b ] 〉 and L2 = 〈S , Pπ̄ , [ Ri , R2b ] 〉 , which operate in the same state and action space , but differ in their instantaneous boundary rewards , and obtain the corresponding optimal policies z1 and z2 . If we then encounter a third LMDP L3 = 〈S , Pπ̄ , [ Ri , R3b ] 〉 such that if r3b = αr1b + βr2b , then the corresponding optimal policy is immediately realizable as z3 = αz1 + βz2 Todorov ( 2009 ) . Importantly , no further learning need take place . This property was exploited in Saxe et al . ( 2017 ) , to develop a powerful multitasking framework . In this setting , an agent faces a number of tasks which have the same state and action spaces , but differ in their boundary reward functions . The multitask LMDP ( MLMDP ) operates by learning a set of Nt tasks each defined as an LMDP Lt = 〈S , P , R = [ ri , rtb ] 〉 , t = 1 , · · · , Nt . This set of LMDPs represents an ensemble of tasks with different ultimate goals . These tasks may then be collected into a task basis ( or task library ) . By utilizing this library of pre-learned policies , when the agent encounters a new task , it is able to immediately obtain the optimal policy by representing the policy for the new task as a weighted blend of policies from the task library . If the new task is not exactly realizable as a linear combination of tasks from the library , significant jump-start performance may nevertheless be obtained by initializing the new task as an approximate blend of library tasks Saxe et al . ( 2017 ) . A particular instantiation of a set of tasks is referred to as being a task ‘ module ’ . 2.2 THE HIERARCHICAL MLMDP . Hierarchical RL holds the promise of dramatically simplifying the learning problem by abstracting away the finer details of the base problem , and instead solving a consistent , but simpler , higher layer problem . By repeatedly applying this abstraction procedure , an agent may in principle overcome the curse of dimensionality which typically occurs as the size of the state and action spaces increase . Of course , this simply replaces one difficulty with another : instead of directly solving the original problem , we need to solve the new problem of how to uncover a suitable abstraction in the first place . Typically this abstraction is hand-crafted by a designer , however a number of techniques have been proposed for autonomously uncovering useful abstractions under the banner of “ subtask/options discovery ” Konidaris ( 2016 ) ; Stolle & Precup ( 2002 ) ; Bonarini et al . ( 2006 ) . These discovery procedures are themselves limited in that they uncover only a single layer of abstraction . It is often not at all clear how these procedure might be recursed to uncover deep representations akin to ‘ options-over-options ’ ( but see Vigorito & Barto ( 2010 ) ) . Furthermore , abstractions that involve temporally extended actions often suffer from the fact that they artificially inflate the action space . This results in an exponential increase in the number of parameters to be learned , and is often crippling . However , by leveraging the compositionality of the MLMDP , a simple recursive scheme was described in Saxe et al . ( 2017 ) for constructing arbitrarily deep hierarchical abstractions . This is achieved by stacking the multitask module discussed above ( Saxe et al. , 2017 ) . The stacking is realized by iteratively constructing higher order MLMDPs in which higher levels select the instantaneous reward structure that defines the current task for lower levels in a feudal-like architecture Dayan & Hinton ( 1993 ) ; Vezhnevets et al . ( 2017 ) . This recursive procedure is carried out by firstly augmenting the layer l state space S̃l = Sl ∪ Slt with a set of Nt terminal boundary states Slt called subtask states . Transitioning into a subtask state corresponds to a decision by the layer l MLMDP to access the next level of the hierarchy . The transitions into the subtask states are governed by a new N lt -by-N l i passive dynamics matrix P l t . In the augmented MLMDP , the full passive dynamics then become P̃ l = [ P li ; P l b ; P l t ] , corresponding to transitions to interior states , boundary states , and subtask states respectively . The higher layer MLMDP is then defined by specifying transitions dynamics , [ P l+1i ; P l+1 b ] and reward function [ r l+1 i ] , such that the higher layer MLMDP is consistent with the lower layer MLMDP ( Saxe et al. , 2017 ) , and solving the easier higher layer problem ultimately solves the base problem too . An example of this hierarchical scheme is shown in Fig. ( 1 ) . Notice that the new subtask states form a separate bona fide LMDP Ll+1 = 〈St , [ P l+1i , P l+1 b ] , [ R l+1 i , Rb ] 〉 . This forms the basis for subsequent recursion . Solving the higher layer MLMDP will yield an optimal action al+1 ( ·|s ) . This action shifts the passive transition dynamics making some transitions more likely , indicating that they are desirable for the current task , and others less likely , indicating that they should be avoided for the current task . The instantaneous rewards for the lower layer are then set to be proportional to the difference between the controlled and passive dynamic , rlt ∝ al+1i ( ·|s ) − p l+1 i ( ·|s ) . In order to stack these modules , the subtask transition matrix P lt must be defined at each layer . It was shown in Earle et al . ( 2018 ) that a suitable P lt could be recovered autonomously by finding a low-rank approximation to the full task basis ZlRN×N at each layer . Intuitively , this exploits the fact that there is often significant additional structure in the policies that make up the task basis . By way of example , suppose that we are in a 4-rooms domain , and the task basis is comprised of a policy to get to each individual state . A low-rank approximation of this basis should uncover the fact that the polices are grouped by the room contains their goal state . Since the task basis Zl is the Base LMDP Higher layer LMDP Augmented LMDP i ] are suitably specified . The resulting construction is itself an LMDP , forming the basis for recursion ( RIGHT ) . exponentiated cost-to-go for the composite policies , and are therefore strictly non-negative , Earle et al . ( 2018 ) propose to utilize non-negative matrix factorization to uncover the low-rank approximation as Zl ≈ DlW l , and set P lt = αDl . This allowing for the fully autonomous discovery of deep control hierarchies . The method proceeds recursively by first constructing a task basis Zl , then finding a low rank approximation of that basis and assigning P lt = αD l , and finally computing the transition dynamics P l+1 and reward structure Rl+1 for the abstracted MLMDP . Howver , a key limitation of the method presented in Earle et al . ( 2018 ) is that the computation of the hierarchy happens in an off-line setting . While the method is therefore applicable to probabilistic path planning and to multitask/multiagent RL , it can not immediately be used online .

Summary: This work builds on the ICML paper from Saxe et al (2017) in which the compositionality property of LMDPs was exploited to solve multi-task hierarchies. The paper extends this work by proposing a method that learns incrementally such hierarchies instead of pre-defining them by design. Some experimental results illustrate the method on two toy problems, a 1D corridor and a corridor of rooms.