Hierarchical Reinforcement Learning with Timed Subgoals

1 Introduction . Hierarchical reinforcement learning ( HRL ) has recently begun to live up to its promise of sampleefficient learning on difficult long-horizon tasks . The idea behind HRL is to break down a complex problem into a hierarchy of more tractable subtasks . A particularly successful approach to defining such a hierarchy is to let a high-level policy choose a subgoal which a low-level policy is then tasked with achieving [ 8 ] . Due to the resulting temporal abstraction such subgoal-based HRL methods have been shown to be able to learn demanding tasks with unprecedented efficiency [ 31 , 23 , 19 ] . In order to realize the full potential of HRL it is necessary to design algorithms that enable concurrent learning on all levels of the hierarchy . However , the changing behavior of the lower level during training introduces a major difficulty . From the perspective of the higher level , the reinforcement learning environment and the policy on the lower level constitute an effective environment which determines what consequences its actions will have . During training , the learning progress on the lower level renders this effective environment non-stationary . If this issue is not addressed , the higher level will usually not start to learn efficiently before the lower level is fully converged . This situation is similar to a manager and a worker trying to solve a task together while the meaning of the vocabulary they use for communication is continuously changing . Clearly , a stable solution can then only be found once the worker reacts reliably to instructions . Hence , to enable true concurrent learning , all 1Videos and code , including our algorithm and the proposed dynamic environments , can be found at https : //github.com/martius-lab/HiTS . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . levels in a hierarchy should see transitions that look like they were generated by interacting with a stationary effective environment . Existing algorithms partially mask the non-stationarity of the effective environment by replacing the subgoal chosen by the higher level appropriately in hindsight . Combined with the subtask of achieving or progressing towards the assigned subgoal as fast as possible , this approach was shown to enable fast learning on a range of challenging sparse-reward , long-horizon tasks [ 23 , 19 ] . What these methods do not take into account , however , is that , if adaptive temporal abstraction is used , the higher level in the hierarchy is effectively interacting with a semi-Markov decision process ( SMDP ) , i.e. , transition times vary . If the objective of the lower level is to reach a subgoal as fast as possible , then the amount of time that elapses until it reaches a given subgoal and returns control to the higher level will decrease during training . Hence , the distribution of the transition times the higher level sees will shift to lower values which introduces an additional source of non-stationarity . When trying to quickly traverse a static environment , such as a maze , this shift is in line with the overall task and will contribute to the learning progress . Yet , as soon as dynamic elements that are beyond the immediate control of the agent are present , the situation changes radically . Consider , for example , the task of returning a tennis ball to a specified point on the ground by hitting it with a racket . This clearly requires the agent to time its actions so as to intercept the ball trajectory with the racket while it has the right orientation and velocity . Even if the higher level found a sequence of subgoals ( specifying the state of the racket ) that brought about the right timing , this solution would stop working as soon as the lower level learns to reach them faster . This would require the higher level to choose a different and possibly longer sequence of subgoals , a process that would continue until the lower level was fully converged . Hence , exposing the higher level to a non-stationary distribution of transition times will lead to training instability and slow learning . As it is the rule rather than the exception for real-world environments to contain dynamic elements beyond the immediate control of the agent – think of humans collaborating with a robot or an autonomous car navigating traffic – this problem can be expected to hinder the application of HRL to real-world tasks . In order to solve the non-stationarity issue in dynamic environments , we propose to let the higher level choose not only what subgoal is to be reached but also when . By emitting such timed subgoals , consisting of a desired subgoal and a time interval that is supposed to elapse before it is reached , the higher level has explicit control over the transition times of the SMDP it interacts with . This completely removes the non-stationarity of transition times and allows for stable concurrent learning in dynamic environments when combined with a technique for hiding the non-stationarity of transitions in the subgoal space . It furthermore gives the higher level explicit control over the degree of temporal abstraction , giving it more direct access to the trade off between a small effective problem horizon and exercising tight control over the agent . The main contribution of this work is to distill these insights into the formulation of a sample-efficient HRL algorithm based on timed subgoals ( Section 3 ) which we term Hierarchical reinforcement learning with Timed Subgoals ( HiTS ) . We demonstrate that HiTS is competitive on four standard benchmark tasks and propose three novel tasks that exemplify the challenges introduced by dynamic elements in the environment . While HiTS succeeds in solving them , prior methods fail to learn a stable solution ( Section 4 ) . In a theoretical analysis , we show that the use of timed subgoals in combination with hindsight action relabeling [ 19 ] removes the non-stationarity of the SMDP generating the data the higher level is trained on ( Section 3 ) . 2 Background . 2.1 Reinforcement learning and subgoal-based hierarchical reinforcement learning . We assume the reinforcement learning problem is formulated as a Markov decision process ( MDP ) defined by a tuple ( S , A , p , r , ρ0 , γ ) consisting of state space S , action space A , transition probabilities p ( s′ | s , a ) , reward function r ( s , a ) , initial state distribution ρ0 ( s0 ) and discount factor γ ∈ [ 0 , 1 ) . At each time step , the agent is provided with a state st ∈ S and samples an action at ∈ A from the distribution π ( · | st ) defined by its policy π . The environment then provides feedback in the form of a reward r ( st , at ) ∈ R and transitions to the next state st+1 ∈ S . The objective is to find an optimal policy π∗ that maximizes the expected discounted return J ( π ) = Eπ , s0∼ρ0 [ ∞∑ t=0 γtr ( st , at ) ] . ( 1 ) In this section , we consider the subgoal-based HRL approach to solving this problem where a lower level is tasked with achieving a subgoal provided by a higher level [ 8 , 23 , 19 ] . The lower level ( index 0 ) acts directly on the environment by outputting a primitive action in A , whereas the higher level ( index 1 ) proposes subgoals for level 0 . Hence , the action of level 1 is the subgoal for level 0 , i.e . g0 : = a1t . Since level 0 performs a variable number of steps τ in the environment before “ finishing ” its subtask , level 1 effectively interacts with a semi-Markov decision process ( SMDP ) [ 16 , 17 ] . Its transition probabilities p ( st+τ , τ | st , a1t ) determine the distribution not only of the next state st+τ level 1 observes but also of the time τ that elapses beforehand . Existing subgoal-based algorithms reward the lower level either for being close to the subgoal [ 23 ] or for progressing in a given direction in subgoal space [ 31 ] or penalize every action that does not lead to the immediate achievement of the subgoal [ 19 ] . In this section , we consider the latter case of a shortest path objective with a reward r0 ( s′ , g0 ) = { 0 if φ ( s′ , g0 ) = 1 , −1 otherwise ( 2 ) for transitioning to a new state s′ . The function φ ( s′ , g0 ) evaluates to 1 only if s′ is sufficiently close to the subgoal g0 in some metric ( e.g . Euclidean distance ) . The higher level is queried for a new subgoal either if the lower level achieved a state sufficiently close to the subgoal or if the lower level has used up a fixed budget of actions . 2.2 Non-stationarity of the induced SMDP and hindsight action relabeling . The higher level of the hierarchy interacts with an SMDP that is induced by the environment dynamics and the behaviour of the lower level . Changes to the low-level policy during training render this SMDP non-stationary which prevents the higher level from learning a stable solution . Applying the recently proposed hindsight action relabeling technique can alleviate this problem for static environments [ 19 ] . We show that for dynamic environments the non-stationarity reappears . For reasons of sample-efficiency , data should be reused after a policy update instead of being discarded . We therefore consider training with an off-policy algorithm that uses a replay buffer . On level 1 transitions of the form ( s , a1 , r1 , s′ ) are being stored , containing the state s , the action a1 , the reward r1 and the state s′ at the end of the subtask execution . For now , we assume that the action space of level 1 ( which is equal to the goal space of level 0 ) is the full state space . Due to the changing low-level policy π0 , the distribution of the next state s′ and the reward r1 given a fixed state action pair ( s , a1 ) will change in the course of training . Hence , the SMDP level 1 interacts with is non-stationary and learning a stable high-level policy becomes feasible only after the lower level is fully converged . Hindsight action relabeling , as proposed in [ 19 ] , addresses this problem by “ pretending ” that the low-level policy is already optimal , i.e. , is able to reach assigned subgoals . This is achieved by replacing the high-level action ( corresponding to the assigned subgoal ) with the state the lower level actually achieved before storing transitions in the replay buffer . If the reward depends on the action , it has to be adapted as well : ( s , a1 , r1 , s′ ) =⇒ ( s , â1 : = s′ , r̂1 : = r1 ( s , â1 ) , s′ ) ( hindsight action relabeling ) . ( 3 ) In Fig . 1a we illustrate this relabeling procedure . Proposition 1 Applying hindsight action relabeling ( Eq . 3 ) at level 1 generates transitions that follow a stationary state transition and reward distribution , i.e . pt ( s , â1 , r̂1 , s′ ) = pt ( s , â1 ) p ( s′ , r̂1 | s , â1 ) where p ( s′ , r̂1 | s , â1 ) is time-independent provided that the subgoal space is equal to the full state space . The proof is given in Suppl . A . The remaining caveat is that the higher level should learn to restrict itself to subgoals which are feasible for the lower level . By conducting testing transitions [ 19 ] , which are exempt from hindsight action relabeling and penalize infeasible subgoals , an appropriate incentive for the higher level can be introduced . Note that this reintroduces a non-stationarity to the data in the replay buffer . For the remaining part of this section we do not consider testing transitions as they are not directly related to the issue we are concerned with . Dynamic environments . We now consider environments with dynamic elements which are not directly controllable by the agent , for instance a flying ball or a moving platform . In this case , choosing the full state space as the subgoal space is problematic as it would include parts of the state the agent has no control over . Consequently , the higher level would be forced to learn the full dynamics of the environment in order to be able to propose feasible subgoals . In such a setting it is therefore more appropriate to restrict the subgoal space to the directly controllable part of the state . Formally , we assume the state space to factorize as S = Sc × Se into a directly controllable part Sc , e.g . the agent , and a part Se corresponding to the rest of the environment [ 18 , 15 , 11 ] . For simplicity , we assume that sc does not influence se in this section , i.e. , p ( s′ e | se ) = p ( s′e | se , sc , a0 ) . A typical choice for a subgoal space would then be Sc , i.e . G0 = Sc , as it allows the lower level to focus on controlling the agent alone and allows for transfer to related tasks . However , due to the restriction of the subgoal space , Prop . 1 does not apply anymore . In particular , hindsight action relabeling does not result in a stationary state-transition distribution anymore as illustrated in Fig . 1 ( b ) . Even though relabeling the action of level 1 according to â1 = s′c hides the influence of the lower level on the achieved goal , the state of the environment s′e varies according to how long the lower level was in control . This transition time τ will shrink during the course of training as the lower level tries to optimize a shortest path objective . As a consequence , the same relabeled state-action pair ( st , â1 = s′c ) comes with different environmental outcomes set+τ and s e t+τ̃ depending on the learning progress of the lower level . This makes it impossible for the agent to adapt its behavior to the environment before the lower level is fully converged . As a consequence , existing subgoal-based HRL methods with adaptive temporal abstraction struggle with dynamic environments .

Summary: This paper proposes a new method for HRL in which the higher level chooses both a subgoal and a time at which the subgoal should be achieved ("Hierarchical RL with Timed Subgoals, or HITS). The main claim of the work is stated early on: "Hence, the use of timed subgoals extends the class of tasks that can be solved efficiently by subgoal-based HRL methods".  This claim is motivated by several elements. First, the paper identifies an issue of non-stationarity in dynamic environments: the higher level's reliance on the lower level's behavior induces a particular kind of non-stationarity; as learning progresses, the lower level behaviors will improve, thereby changing the outcome of the same high level actions. This discrepency motivates the claim that "all levels in a hierarchy should see transitions that look like they were generated by interacting with a stationary effective environment." Alongside this, it is noted that dynamic environments with elements not controlled by the agent are particularly challenging as they further introduce non-stationarity. Second, the paper focuses on a decomposition of state into the controlled and uncontrolled parts of state, which allows for subgoals to concentrate on the controlled aspects of the environment. These two points are examined to motivate timed subgoals alongside hindsight relabeling techniques to form the core of HITS. Otherwise, by my reading, HITS is quite simple (which I take to be a virtue). Three new benchmark domains are introduced that feature time-based reasoning, such as a ship that needs to carefully furl its sail in order to quickly pass through a drawbridge. These domains are chosen to highlight the positive aspects of HITS, and to differentiate what HITS does differently from other HRL methods. The results quite clearly demonstrate that HITS is well suited to solve these domains.