Hierarchical Reinforcement Learning by Discovering Intrinsic Options

1 INTRODUCTION . Imagine a wheeled robot learning to kick a soccer ball into a goal with sparse reward supervision . In order to succeed , it must discover how to first navigate in its environment , then touch the ball , and finally kick it into the goal , only receiving a positive reward at the end for completing the task . This is a naturally difficult problem for traditional reinforcement learning ( RL ) to solve , unless the task has been manually decomposed into temporally extended stages where each stage constitutes a much easier subtask . In this paper we ask , how do we learn to decompose the task automatically and utilize the decomposition to solve sparse reward problems ? Deep RL has made great strides solving a variety of tasks recently , with hierarchical RL ( hRL ) demonstrating promise in solving such sparse reward tasks ( Sharma et al. , 2019b ; Le et al. , 2018 ; Merel et al. , 2019 ; Ranchod et al. , 2015 ) . In hRL , the task is decomposed into a hierarchy of subtasks , where policies at the top of the hierarchy call upon policies below to perform actions to solve their respective subtasks . This abstracts away actions for the policies at the top levels of the hierarchy . hRL makes exploration easier by potentially reducing the number of steps the agent needs to take to explore its state space . Moreover , at higher levels of the hierarchy , temporal abstraction results in more aggressive , multi-step value bootstrapping when temporal-difference ( TD ) learning is employed . These benefits are critical in sparse reward tasks as they allow an agent to more easily discover reward signals and assign credit . Many existing hRL methods make assumptions about the task structure ( e.g. , fetching an object involves three stages : moving towards the object , picking it up , and combing back ) , and/or the skills needed to solve the task ( e.g. , pre-programmed motor skills ) ( Florensa et al. , 2016 ; Riedmiller et al. , 2018 ; Lee et al. , 2019 ; Hausman et al. , 2018 ; Lee et al. , 2020 ; Sohn et al. , 2018 ; Ghavamzadeh & Mahadevan , 2003 ; Nachum et al. , 2018 ) . Thus these methods may require manually designing the correct task decomposition , explicitly formulating the option space , or programming pre-defined options for higher level policies to compose . Instead , we seek to formulate a general method that can learn these abstractions from scratch , for any task , with little manual design in the task domain . The main contribution of this paper is HIDIO ( HIerarchical RL by Discovering Intrinsic Options ) , a hierarchical method that discovers task-agnostic intrinsic options in a self-supervised manner while ∗Denotes equal contribution . Email to jessez @ usc.edu , { haonan.yu , wei.xu } @ horizon.ai †Work done as an intern at Horizon Robotics . learning to schedule them to accomplish environment tasks . The latent option representation is uncovered as the option-conditioned policy is trained , both according to the same self-supervised worker objective . The scheduling of options is simultaneously learned by maximizing environment reward collected by the option-conditioned policy . HIDIO can be easily applied to new sparsereward tasks by simply re-discovering options . We propose and empirically evaluate various instantiations of the option discovery process , comparing the resulting options with respect to their final task performance . We demonstrate that HIDIO is able to efficiently learn and discover diverse options to be utilized for higher task reward with superior sample efficiency compared to other hierarchical methods . 2 PRELIMINARIES . We consider the reinforcement learning ( RL ) problem in a Markov Decision Process ( MDP ) . Let s ∈ RS be the agent state . We use the terms “ state ” and “ observation ” interchangeably to denote the environment input to the agent . A state can be fully or partially observed . Without loss of generality , we assume a continuous action space a ∈ RA for the agent . Let πθ ( a|s ) be the policy distribution with learnable parameters θ , and P ( st+1|st , at ) the transition probability that measures how likely the environment transitions to st+1 given that the agent samples an action by at ∼ πθ ( ·|st ) . After the transition to st+1 , the agent receives a deterministic scalar reward r ( st , at , st+1 ) . The objective of RL is to maximize the sum of discounted rewards with respect to θ : E πθ , P [ ∞∑ t=0 γtr ( st , at , st+1 ) ] ( 1 ) where γ ∈ [ 0 , 1 ] is a discount factor . We will omit P in the expectation for notational simplicity . In the options framework ( Sutton et al. , 1999 ) , the agent can switch between different options during an episode , where an option is translated to a sequence of actions by an option-conditioned policy with a termination condition . A set of options defined over an MDP induces a hierarchy that models temporal abstraction . For a typical two-level hierarchy , a higher-level policy produces options , and the policy at the lower level outputs environment actions conditioned on the proposed options . The expectation in Eq . 1 is taken over policies at both levels . 3 HIERARCHICAL RL BY DISCOVERING INTRINSIC OPTIONS . We now introduce our hierarchical method for solving sparse reward tasks . We assume little prior knowledge about the task structure , except that it can be learned through a hierarchy of two levels . The higher-level policy ( the scheduler πθ ) , is trained to maximize environment reward , while the lower-level policy ( the worker πφ ) is trained in a self-supervised manner to efficiently discover options that are utilized by πθ to accomplish tasks . Importantly , by self-supervision the worker gets access to dense intrinsic rewards regardless of the sparsity of the extrinsic rewards . Without loss of generality , we assume that each episode has a length of T and the scheduler outputs an option every K steps . The scheduled option u ∈ [ −1 , 1 ] D ( where D is a pre-defined dimensionality ) , is a latent representation that will be learned from scratch given the environment task . Modulated by u , the worker executes K steps before the scheduler outputs the next option . Let the time horizon of the scheduler be H = d TK e. Formally , we define Scheduler policy : uh ∼ πθ ( ·|sh,0 ) , 0 ≤ h < H Worker policy : ah , k ∼ πφ ( ·|sh , k , uh ) , 0 ≤ k < K Environment dynamics : sh , k+1 ∼ P ( ·|sh , k , ah , k ) , 0 ≤ h < H , 0 ≤ k < K ( 2 ) where we denote sh , k and ah , k as the k-th state and action respectively , within the h-th option window of length K. Note that given this sampling process , we have sh , K ≡ sh+1,0 , namely , the last state of the current option uh is the initial state of the next option uh+1 . The overall framework of our method is illustrated in Figure 1 . 3.1 LEARNING THE SCHEDULER . Every time the scheduler issues an option uh , it receives an reward Rh computed by accumulating environment rewards over the next K steps . Its objective is : max θ Eπθ [ H−1∑ h=0 βhRh ] , where β = γK and Rh = Eπφ [ K−1∑ k=0 γkr ( sh , k , ah , k , sh , k+1 ) ] ( 3 ) This scheduler objective itself is not a new concept , as similar ones have been adopted by other hRL methods ( Vezhnevets et al. , 2017 ; Nachum et al. , 2018 ; Riedmiller et al. , 2018 ) . One significant difference between our option with that of prior work is that our option u is simply a latent variable ; there is no explicit constraint on what semantics u could represent . In contrast , existing methods usually require their options to reside in a subspace of the state space , to be grounded to the environment , or to have known structures , so that the scheduler can compute rewards and termination conditions for the worker . Note that our latent options can be easily re-trained given a new task . 3.2 LEARNING THE WORKER . The main focus of this paper is to investigate how to effectively learn the worker policy in a selfsupervised manner . Our motivation is that it might be unnecessary to make an option dictate the worker to reach some “ -space ” of goals ( Vezhnevets et al. , 2017 ; Nachum et al. , 2018 ) . As long as the option can be translated to a short sequence of primitive actions , it does not need to be grounded with concrete meanings such as goal reaching . Below we will treat the option as a latent variable that modulates the worker , and propose to learn its latent representation in a hierarchical setting from the environment task . 3.2.1 WORKER OBJECTIVE . We first define a new meta MDP on top of the original task MDP so that for any h , k , and t : 1 ) sh , k : = ( sh,0 , . . . , sh , k ) , 2 ) ah , k : = ( ah,0 , . . . , ah , k ) , 3 ) r ( sh , k , ah , k , sh , k+1 ) : = r ( sh , k , ah , k , sh , k+1 ) , and 4 ) P ( sh , k+1|sh , k , ah , k ) : = P ( sh , k+1|sh , k , ah , k ) . This new MDP equips the worker with historical state and action information since the time ( h , 0 ) when an option hwas scheduled . Specifically , each state sh , k or action ah , k encodes the history from the beginning ( h , 0 ) up to ( h , k ) within the option . In the following , we will call pairs { ah , k , sh , k+1 } option sub-trajectories . The worker policy now takes option sub-trajectories as inputs : ah , k ∼ πφ ( ·|sh , k , ah , k−1 , uh ) , 0 ≤ k < K , whereas the scheduler policy still operates in the original MDP . Denote ∑ h , k ≡ ∑H−1 h=0 ∑K−1 k=0 for simplicity . The worker objective , defined on this new MDP , is to minimize the entropy of the option uh conditioned on the option sub-trajectory { ah , k , sh , k+1 } : max φ E πθ , πφ ∑ h , k log p ( uh|ah , k , sh , k+1 ) ︸ ︷︷ ︸ negative conditional option entropy −β log πφ ( ah , k|sh , k , ah , k−1 , uh ) ︸ ︷︷ ︸ worker policy entropy ( 4 ) where the expectation is over the current πθ and πφ but the maximization is only with respect to φ . Intuitively , the first term suggests that the worker is optimized to confidently identify an option given a sub-trajectory . However , it alone will not guarantee the diversity of options because potentially even very similar sub-trajectories can be classified into different options if the classification model has a high capacity , in which case we say that the resulting sub-trajectory space has a very high “ resolution ” . As a result , the conditional entropy alone might not be able to generate useful options to be exploited by the scheduler for task solving , because the coverage of the sub-trajectory space is poor . To combat this degenerate solution , we add a second term which maximizes the entropy of the worker policy . Intuitively , while the worker generates identifiable sub-trajectories corresponding to a given option , it should act as randomly as possible to separate sub-trajectories of different options , lowering the “ resolution ” of the sub-trajectory space to encourage its coverage . Because directly estimating the posterior p ( uh|ah , k , sh , k+1 ) is intractable , we approximate it with a parameterized posterior log qψ ( uh|ah , k , sh , k+1 ) to obtain a lower bound ( Barber & Agakov , 2003 ) , where qψ is a discriminator to be learned . Then we can maximize this lower bound instead : max φ , ψ E πθ , πφ ∑ h , k log qψ ( uh|ah , k , sh , k+1 ) − β log πφ ( ah , k|sh , k , ah , k−1 , uh ) . ( 5 ) The discriminator qψ is trained by maximizing likelihoods of options given sampled sub-trajectories . The worker πφ is trained via max-entropy RL ( Soft Actor-Critic ( SAC ) ( Haarnoja et al. , 2018 ) ) with the intrinsic reward rloh , k+1 : = log qψ ( · ) − β log πφ ( · ) . β is fixed to 0.01 in our experiments . Note that there are at least four differences between Eq . 5 and the common option discovery objective in either VIC ( Gregor et al. , 2016 ) or DIAYN ( Eysenbach et al. , 2019 ) : 1 . Both VIC and DIAYN assume that a sampled option will last through an entire episode , and the option is always sampled at the beginning of an episode . Thus their option trajectories “ radiate ” from the initial state set . In contrast , our worker policy learns options that initialize every K steps within an episode , and they can have more diverse semantics depending on the various states sh,0 visited by the agent . This is especially helpful for some tasks where new options need to be discovered after the agent reaches unseen areas in later stages of training . 2 . Actions taken by the worker policy under the current option will have consequences on the next option . This is because the final state sh , K of the current option is defined to be the initial state sh+1,0 of the next option . So in general , the worker policy is trained not only to discover diverse options across the current K steps , but also to make the discovery easier in the future steps . In other words , the worker policy needs to solve the credit assignment problem across options , under the expectation of the scheduler policy . 3 . To enable the worker policy to learn from a discriminator that predicts based on option subtrajectories { ah , k , sh , k+1 } instead of solely on individual states sh , k , we have constructed a new meta MDP where each state sh , k encodes history from the beginning ( h , 0 ) up to ( h , k ) within an option h. This new meta MDP is critical , because otherwise one simply can not learn a worker policy from a reward function that is defined by multiple time steps ( sub-trajectories ) since the learning problem is no longer Markovian . 4 . Lastly , thanks to the new MDP , we are able to explore various possible instantiations of the discriminator ( see Section 3.3 ) . As observed in the experiments , individual states are actually not the optimal features for identifying options . These differences constitute the major novelty of our worker objective .

Summary: The paper develops a hierarchical reinforcement learning algorithm and analyzes its behaviour in four robotic manipulation and navigation tasks. The approach is based on a two-level hierarchy, *scheduler* at the top and *worker* at the bottom. This is similar to other approaches in the literature and the algorithm uses many ideas and elements from existing algorithms. However, these ideas and elements are combined in a novel and well-justified manner. The result is an algorithm that yields good results in a range of problems. The experiments are well done. The paper is generally organised well and written clearly. Relevant literature is reviewed well.