Hierarchical Reinforcement Learning with Timed Subgoals

1 Introduction . Hierarchical reinforcement learning ( HRL ) has recently begun to live up to its promise of sampleefficient learning on difficult long-horizon tasks . The idea behind HRL is to break down a complex problem into a hierarchy of more tractable subtasks . A particularly successful approach to defining such a hierarchy is to let a high-level policy choose a subgoal which a low-level policy is then tasked with achieving [ 8 ] . Due to the resulting temporal abstraction such subgoal-based HRL methods have been shown to be able to learn demanding tasks with unprecedented efficiency [ 31 , 23 , 19 ] . In order to realize the full potential of HRL it is necessary to design algorithms that enable concurrent learning on all levels of the hierarchy . However , the changing behavior of the lower level during training introduces a major difficulty . From the perspective of the higher level , the reinforcement learning environment and the policy on the lower level constitute an effective environment which determines what consequences its actions will have . During training , the learning progress on the lower level renders this effective environment non-stationary . If this issue is not addressed , the higher level will usually not start to learn efficiently before the lower level is fully converged . This situation is similar to a manager and a worker trying to solve a task together while the meaning of the vocabulary they use for communication is continuously changing . Clearly , a stable solution can then only be found once the worker reacts reliably to instructions . Hence , to enable true concurrent learning , all 1Videos and code , including our algorithm and the proposed dynamic environments , can be found at https : //github.com/martius-lab/HiTS . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . levels in a hierarchy should see transitions that look like they were generated by interacting with a stationary effective environment . Existing algorithms partially mask the non-stationarity of the effective environment by replacing the subgoal chosen by the higher level appropriately in hindsight . Combined with the subtask of achieving or progressing towards the assigned subgoal as fast as possible , this approach was shown to enable fast learning on a range of challenging sparse-reward , long-horizon tasks [ 23 , 19 ] . What these methods do not take into account , however , is that , if adaptive temporal abstraction is used , the higher level in the hierarchy is effectively interacting with a semi-Markov decision process ( SMDP ) , i.e. , transition times vary . If the objective of the lower level is to reach a subgoal as fast as possible , then the amount of time that elapses until it reaches a given subgoal and returns control to the higher level will decrease during training . Hence , the distribution of the transition times the higher level sees will shift to lower values which introduces an additional source of non-stationarity . When trying to quickly traverse a static environment , such as a maze , this shift is in line with the overall task and will contribute to the learning progress . Yet , as soon as dynamic elements that are beyond the immediate control of the agent are present , the situation changes radically . Consider , for example , the task of returning a tennis ball to a specified point on the ground by hitting it with a racket . This clearly requires the agent to time its actions so as to intercept the ball trajectory with the racket while it has the right orientation and velocity . Even if the higher level found a sequence of subgoals ( specifying the state of the racket ) that brought about the right timing , this solution would stop working as soon as the lower level learns to reach them faster . This would require the higher level to choose a different and possibly longer sequence of subgoals , a process that would continue until the lower level was fully converged . Hence , exposing the higher level to a non-stationary distribution of transition times will lead to training instability and slow learning . As it is the rule rather than the exception for real-world environments to contain dynamic elements beyond the immediate control of the agent – think of humans collaborating with a robot or an autonomous car navigating traffic – this problem can be expected to hinder the application of HRL to real-world tasks . In order to solve the non-stationarity issue in dynamic environments , we propose to let the higher level choose not only what subgoal is to be reached but also when . By emitting such timed subgoals , consisting of a desired subgoal and a time interval that is supposed to elapse before it is reached , the higher level has explicit control over the transition times of the SMDP it interacts with . This completely removes the non-stationarity of transition times and allows for stable concurrent learning in dynamic environments when combined with a technique for hiding the non-stationarity of transitions in the subgoal space . It furthermore gives the higher level explicit control over the degree of temporal abstraction , giving it more direct access to the trade off between a small effective problem horizon and exercising tight control over the agent . The main contribution of this work is to distill these insights into the formulation of a sample-efficient HRL algorithm based on timed subgoals ( Section 3 ) which we term Hierarchical reinforcement learning with Timed Subgoals ( HiTS ) . We demonstrate that HiTS is competitive on four standard benchmark tasks and propose three novel tasks that exemplify the challenges introduced by dynamic elements in the environment . While HiTS succeeds in solving them , prior methods fail to learn a stable solution ( Section 4 ) . In a theoretical analysis , we show that the use of timed subgoals in combination with hindsight action relabeling [ 19 ] removes the non-stationarity of the SMDP generating the data the higher level is trained on ( Section 3 ) . 2 Background . 2.1 Reinforcement learning and subgoal-based hierarchical reinforcement learning . We assume the reinforcement learning problem is formulated as a Markov decision process ( MDP ) defined by a tuple ( S , A , p , r , ρ0 , γ ) consisting of state space S , action space A , transition probabilities p ( s′ | s , a ) , reward function r ( s , a ) , initial state distribution ρ0 ( s0 ) and discount factor γ ∈ [ 0 , 1 ) . At each time step , the agent is provided with a state st ∈ S and samples an action at ∈ A from the distribution π ( · | st ) defined by its policy π . The environment then provides feedback in the form of a reward r ( st , at ) ∈ R and transitions to the next state st+1 ∈ S . The objective is to find an optimal policy π∗ that maximizes the expected discounted return J ( π ) = Eπ , s0∼ρ0 [ ∞∑ t=0 γtr ( st , at ) ] . ( 1 ) In this section , we consider the subgoal-based HRL approach to solving this problem where a lower level is tasked with achieving a subgoal provided by a higher level [ 8 , 23 , 19 ] . The lower level ( index 0 ) acts directly on the environment by outputting a primitive action in A , whereas the higher level ( index 1 ) proposes subgoals for level 0 . Hence , the action of level 1 is the subgoal for level 0 , i.e . g0 : = a1t . Since level 0 performs a variable number of steps τ in the environment before “ finishing ” its subtask , level 1 effectively interacts with a semi-Markov decision process ( SMDP ) [ 16 , 17 ] . Its transition probabilities p ( st+τ , τ | st , a1t ) determine the distribution not only of the next state st+τ level 1 observes but also of the time τ that elapses beforehand . Existing subgoal-based algorithms reward the lower level either for being close to the subgoal [ 23 ] or for progressing in a given direction in subgoal space [ 31 ] or penalize every action that does not lead to the immediate achievement of the subgoal [ 19 ] . In this section , we consider the latter case of a shortest path objective with a reward r0 ( s′ , g0 ) = { 0 if φ ( s′ , g0 ) = 1 , −1 otherwise ( 2 ) for transitioning to a new state s′ . The function φ ( s′ , g0 ) evaluates to 1 only if s′ is sufficiently close to the subgoal g0 in some metric ( e.g . Euclidean distance ) . The higher level is queried for a new subgoal either if the lower level achieved a state sufficiently close to the subgoal or if the lower level has used up a fixed budget of actions . 2.2 Non-stationarity of the induced SMDP and hindsight action relabeling . The higher level of the hierarchy interacts with an SMDP that is induced by the environment dynamics and the behaviour of the lower level . Changes to the low-level policy during training render this SMDP non-stationary which prevents the higher level from learning a stable solution . Applying the recently proposed hindsight action relabeling technique can alleviate this problem for static environments [ 19 ] . We show that for dynamic environments the non-stationarity reappears . For reasons of sample-efficiency , data should be reused after a policy update instead of being discarded . We therefore consider training with an off-policy algorithm that uses a replay buffer . On level 1 transitions of the form ( s , a1 , r1 , s′ ) are being stored , containing the state s , the action a1 , the reward r1 and the state s′ at the end of the subtask execution . For now , we assume that the action space of level 1 ( which is equal to the goal space of level 0 ) is the full state space . Due to the changing low-level policy π0 , the distribution of the next state s′ and the reward r1 given a fixed state action pair ( s , a1 ) will change in the course of training . Hence , the SMDP level 1 interacts with is non-stationary and learning a stable high-level policy becomes feasible only after the lower level is fully converged . Hindsight action relabeling , as proposed in [ 19 ] , addresses this problem by “ pretending ” that the low-level policy is already optimal , i.e. , is able to reach assigned subgoals . This is achieved by replacing the high-level action ( corresponding to the assigned subgoal ) with the state the lower level actually achieved before storing transitions in the replay buffer . If the reward depends on the action , it has to be adapted as well : ( s , a1 , r1 , s′ ) =⇒ ( s , â1 : = s′ , r̂1 : = r1 ( s , â1 ) , s′ ) ( hindsight action relabeling ) . ( 3 ) In Fig . 1a we illustrate this relabeling procedure . Proposition 1 Applying hindsight action relabeling ( Eq . 3 ) at level 1 generates transitions that follow a stationary state transition and reward distribution , i.e . pt ( s , â1 , r̂1 , s′ ) = pt ( s , â1 ) p ( s′ , r̂1 | s , â1 ) where p ( s′ , r̂1 | s , â1 ) is time-independent provided that the subgoal space is equal to the full state space . The proof is given in Suppl . A . The remaining caveat is that the higher level should learn to restrict itself to subgoals which are feasible for the lower level . By conducting testing transitions [ 19 ] , which are exempt from hindsight action relabeling and penalize infeasible subgoals , an appropriate incentive for the higher level can be introduced . Note that this reintroduces a non-stationarity to the data in the replay buffer . For the remaining part of this section we do not consider testing transitions as they are not directly related to the issue we are concerned with . Dynamic environments . We now consider environments with dynamic elements which are not directly controllable by the agent , for instance a flying ball or a moving platform . In this case , choosing the full state space as the subgoal space is problematic as it would include parts of the state the agent has no control over . Consequently , the higher level would be forced to learn the full dynamics of the environment in order to be able to propose feasible subgoals . In such a setting it is therefore more appropriate to restrict the subgoal space to the directly controllable part of the state . Formally , we assume the state space to factorize as S = Sc × Se into a directly controllable part Sc , e.g . the agent , and a part Se corresponding to the rest of the environment [ 18 , 15 , 11 ] . For simplicity , we assume that sc does not influence se in this section , i.e. , p ( s′ e | se ) = p ( s′e | se , sc , a0 ) . A typical choice for a subgoal space would then be Sc , i.e . G0 = Sc , as it allows the lower level to focus on controlling the agent alone and allows for transfer to related tasks . However , due to the restriction of the subgoal space , Prop . 1 does not apply anymore . In particular , hindsight action relabeling does not result in a stationary state-transition distribution anymore as illustrated in Fig . 1 ( b ) . Even though relabeling the action of level 1 according to â1 = s′c hides the influence of the lower level on the achieved goal , the state of the environment s′e varies according to how long the lower level was in control . This transition time τ will shrink during the course of training as the lower level tries to optimize a shortest path objective . As a consequence , the same relabeled state-action pair ( st , â1 = s′c ) comes with different environmental outcomes set+τ and s e t+τ̃ depending on the learning progress of the lower level . This makes it impossible for the agent to adapt its behavior to the environment before the lower level is fully converged . As a consequence , existing subgoal-based HRL methods with adaptive temporal abstraction struggle with dynamic environments .

Summary: This paper presents a new framework for subgoal-based HRL where the high-level chooses not only what subgoal to be reached but also when to reach the subgoal, i.e., a timed subgoal, for the lower-level. By emitting such timed subgoals, the non-stationarity of transition times during learning at the high-level can be removed, which is supported by theoretical analysis, and thus, making it possible for stable learning in dynamic environments. Combining timed subgoals with hindsight action relabeling, a practical algorithm HiTS is proposed and it outperforms baseline methods on a set of dynamic environments.