Learning Multi-Level Hierarchies with Hindsight

1 INTRODUCTION . Hierarchy has the potential to accelerate learning in sequential decision making tasks because hierarchical agents can decompose problems into smaller subproblems . In order to take advantage of these shorter horizon subproblems and realize the potential of HRL , an HRL algorithm must be able to learn the multiple levels within the hierarchy in parallel . That is , at the same time one level in the hierarchy is learning the sequence of subtasks needed to solve a task , the level below should be learning the sequence of shorter time scale actions needed to solve each subtask . Yet the existing HRL algorithms that are capable of automatically learning hierarchies in continuous domains ( Schmidhuber , 1991 ; Konidaris & Barto , 2009 ; Bacon et al. , 2017 ; Vezhnevets et al. , 2017 ; Nachum et al. , 2018 ) do not efficiently learn the multiple levels within the hierarchy in parallel . Instead , these algorithms often resort to learning the hierarchy one level at a time in a bottom-up fashion . Learning multiple levels of policies in parallel is challenging due to non-stationary state transition functions . In nested , multi-level hierarchies , the transition function for any level above the ground level depends on the current policies below that level . For instance , in a 2-level hierarchy , the high-level policy may output a subgoal state for the low level to achieve , and the state to which this subgoal state leads will depend on the current low-level policy . When all policies within the hierarchy are trained simultaneously , the transition function at each level above ground level will continue to change as long as the policies below that level continue to be updated . In this setting of non-stationary transition functions , RL will likely struggle to learn the above ground level policies in the hierarchy because in order for RL methods to effectively value actions , the distribution of states to which those actions lead should be stable . However , learning multiple policies in parallel is still possible because the transition function for each level above ground level will stabilize once all lower level policies have converged to optimal or near optimal policies . Thus , RL can be used to learn all policies in parallel if each level above ground level had a way to simulate a transition function that uses the optimal versions of lower level policies . Our framework is able to simulate a transition function that uses an optimal lower level policy hierarchy and thus can learn multiple levels of policies in parallel . We introduce a new HRL framework , Hierarchical Actor-Critic ( HAC ) , that can significantly accelerate learning by enabling hierarchical agents to jointly learn a hierarchy of policies . Our framework is primarily comprised of two components : ( i ) a particular hierarchical architecture and ( ii ) a method for learning the multiple levels of policies in parallel given sparse rewards . The hierarchies produced by HAC have a specific architecture consisting of a set of nested , goalconditioned policies that use the state space as the mechanism for breaking down a task into subtasks . The hierarchy of nested policies works as follows . The highest level policy takes as input the current state and goal state provided by the task and outputs a subgoal state . This state is used as the goal state for the policy at the next level down . The policy at that level takes as input the current state and the goal state provided by the level above and outputs its own subgoal state for the next level below to achieve . This process continues until the lowest level is reached . The lowest level then takes as input the current state and the goal state provided by the level above and outputs a primitive action . Further , each level has a certain number of attempts to achieve its goal state . When the level either runs out of attempts or achieves its goal state , execution at that level ceases and the level above outputs another subgoal . Figure 1 shows how an ant agent trained with HAC uses its 3-level policy hierarchy ( π2 , π1 , π0 ) to move through rooms to reach its goal . At the beginning of the episode , the ant ’ s highest level policy , π2 , takes as input the current state , which in this case is a vector containing the ant ’ s joint positions and velocities ( [ θ , θ̇ ] ) , and its goal state , represented by the yellow box . π2 then outputs a subgoal state , represented by the green box , for π1 to achieve . π1 takes as input the current state and its goal state represented by the green box and outputs the subgoal state represented by the purple box . Finally , π0 takes as input the current state and the goal state represented by purple box and outputs a primitive action , which in this case is a vector of joint torques . π0 has a fixed number of attempts to move to the purple box before π1 outputs another subgoal state . Similarly , π1 has a fixed number of subgoal states that it can output to try to move the agent to the green box before π2 outputs another subgoal . In addition , HAC enables agents to learn multiple policies in parallel using only sparse reward functions as a result of two types of hindsight transitions . Hindsight action transitions help agents learn multiple levels of policies simultaneously by training each subgoal policy with respect to a transition function that simulates the optimal lower level policy hierarchy . Hindsight action transitions are implemented by using the subgoal state achieved in hindsight instead of the original subgoal state as the action component in the transition . For instance , when a subgoal level proposes subgoal state A , but the next level policy is unsuccessful and the agent ends in state B after a certain number of attempts , the subgoal level receives a transition in which the state B is the action component , not state A . The key outcome is that now the action and next state components in the transition are the same , as if the optimal lower level policy hierarchy had been used to achieve subgoal state B . Training with respect to a transition function that uses the optimal lower level policy hierarchy is critical to learning multiple policies in parallel , because the subgoal policies can be learned independently of the changing lower level policies . With hindsight action transitions , a subgoal level can focus on learning the sequences of subgoal states that can reach a goal state , while the lower level policies focus on learning the sequences of actions to achieve those subgoal states . The second type of hindsight transition , hindsight goal transitions , helps each level learn a goal-conditioned policy in sparse reward tasks by extending the idea of Hindsight Experience Replay ( Andrychowicz et al . ( 2017 ) ) to the hierarchical setting . In these transitions , one of the states achieved in hindsight is used as the goal state in the transition instead of the original goal state . We evaluated our approach on both grid world tasks and more complex simulated robotics environments . For each task , we evaluated agents with 1 , 2 , and 3 levels of hierarchy . In all tasks , agents using multiple levels of hierarchy substantially outperformed agents that learned a single policy . Further , in all tasks , agents using 3 levels of hierarchy outperformed agents using 2 levels of hierarchy . Indeed , our framework is the first to show empirically that it can jointly learn 3-level hierarchical policies in tasks with continuous state and action spaces . In addition , our approach outperformed another leading HRL algorithm , HIRO ( Nachum et al. , 2018 ) , on three simulated robotics tasks . 2 RELATED WORK . Building agents that can learn hierarchical policies is a longstanding problem in Reinforcement Learning ( Sutton et al. , 1999 ; Dietterich , 2000 ; McGovern & Barto , 2001 ; Kulkarni et al. , 2016 ; Menache et al. , 2002 ; Şimşek et al. , 2005 ; Bakker & Schmidhuber , 2004 ; Wiering & Schmidhuber , 1997 ) . However , most HRL approaches either only work in discrete domains , require pre-trained low-level controllers , or need a model of the environment . There are several other automated HRL techniques that can work in continuous domains . Schmidhuber ( 1991 ) proposed a HRL approach that can support multiple levels , as in our method . However , the approach requires that the levels are trained one at a time , beginning with the bottom level , which can slow learning . Konidaris & Barto ( 2009 ) proposed Skill Chaining , a 2-level HRL method that incrementally chains options backwards from the end goal state to the start state . Our key advantage relative to Skill Chaining is that our approach can learn the options needed to bring the agent from the start state to the goal state in parallel rather than incrementally . Nachum et al . ( 2018 ) proposed HIRO , a 2-level HRL approach that can learn off-policy like our approach and outperforms two other popular HRL techniques used in continuous domains : Option-Critic ( Bacon et al . ( 2017 ) ) and FeUdal Networks ( FUN ) ( Vezhnevets et al . ( 2017 ) ) . HIRO , which was developed simultaneously and independently to our approach , uses the same hierarchical architecture , but does not use either form of hindsight and is therefore not as efficient at learning multiple levels of policies in sparse reward tasks . 3 BACKGROUND . We are interested in solving a Markov Decision Process ( MDP ) augmented with a set of goals G ( each a state or set of states ) that we would like an agent to learn . We define an MDP augmented with a set of goals as a Universal MDP ( UMDP ) . A UMDP is a tuple U = ( S , G , A , T , R , γ ) , in which S is the set of states ; G is the set of goals ; A is the set of actions ; T is the transition probability function in which T ( s , a , s′ ) is the probability of transitioning to state s′ when action a is taken in state s ; R is the reward function ; γ is the discount rate ∈ [ 0 , 1 ) . At the beginning of each episode in a UMDP , a goal g ∈ G is selected for the entirety of the episode . The solution to a UMDP is a control policy π : S , G → A that maximizes the value function vπ ( s , g ) = Eπ [ ∑∞ n=0 γ nRt+n+1|st = s , gt = g ] for an initial state s and goal g. In order to implement hierarchical agents in tasks with continuous state and actions spaces , we will use two techniques from the RL literature : ( i ) the Universal Value Function Approximator ( UVFA ) ( Schaul et al. , 2015 ) and ( ii ) Hindsight Experience Replay ( Andrychowicz et al. , 2017 ) . The UVFA will be used to estimate the action-value function of a goal-conditioned policy π , qπ ( s , g , a ) = Eπ [ ∑∞ n=0 γ nRt+n+1|st = s , gt = g , at = a ] . In our experiments , the UVFAs used will be in the form of feedforward neural networks . UVFAs are important for learning goal-conditioned policies because they can potentially generalize Q-values from certain regions of the ( state , goal , action ) tuple space to other regions of the tuple space , which can accelerate learning . However , UVFAs are less helpful in difficult tasks that use sparse reward functions . In these tasks when the sparse reward is rarely achieved , the UVFA will not have large regions of the ( state , goal , action ) tuple space with relatively high Q-values that it can generalize to other regions . For this reason , we also use Hindsight Experience Replay ( Andrychowicz et al. , 2017 ) . HER is a data augmentation technique that can accelerate learning in sparse reward tasks . HER first creates copies of the [ state , action , reward , next state , goal ] transitions that are created in traditional off-policy RL . In the copied transitions , the original goal element is replaced with a state that was actually achieved during the episode , which guarantees that at least one of the HER transitions will contain the sparse reward . These HER transitions in turn help the UVFA learn about regions of the ( state , goal , action ) tuple space that should have relatively high Q-values , which the UVFA can then potentially extrapolate to the other areas of the tuple space that may be more relevant for achieving the current set of goals .

Summary: This paper presents a novel approach for doing hierarchical deep RL. Each level of the hierarchy is rewarded for reaching a goal state. The top level's goal state is the environment goal, lower level goals states are the actions of the higher levels. The lowest level's actions are primitive actions. Each level can act until it reaches it goal or a maximum of T steps. Then HER is used to still learn from missed subgoals. For example, if the lowest level is given a subgoal and fails to achieve it, it is trained with a new experience where the goal was the achieved state. In addition, the level above is trained with an experience where the action it chose (the subgoal that was not achieved) is replaced with the subgoal that was achieved. So HER is replacing goals on one level and replacing actions on the higher level. The paper shows nice empirical results across 6 domains.