Data-efficient Hindsight Off-policy Option Learning

1 INTRODUCTION . Despite deep reinforcement learning ’ s considerable successes in recent years ( Silver et al. , 2017 ; OpenAI et al. , 2018 ; Vinyals et al. , 2019 ) , applications in domains with limited or expensive data have so far been rare . To address this challenge , data efficiency can be improved through additional structure imposed on the solution space . Hierarchical methods , such as the options framework ( Sutton et al. , 1999 ; Precup , 2000 ) , present one approach to integrate different abstractions into the agent . By representing an agent as a combination of low-level and high-level controllers , option policies can support reuse of low-level behaviours and can ultimately accelerate learning . The advantages introduced by the hierarchical control scheme imposed by the options framework are partially balanced by additional complexities , including possible degenerate cases ( Precup , 2000 ; Harb et al. , 2018 ) , trade-offs regarding option length ( Harutyunyan et al. , 2019 ) , and additional stochasticity . Overall , the interaction of algorithm and environment , as exemplified by the factors above , can become increasingly difficult , especially in an off-policy setting ( Precup et al. , 2006 ) . With Hindsight Off-policy Options ( HO2 ) , we present a method for data-efficient , robust off-policy learning of options to partially combat the previously mentioned challenges in option learning . We evaluate against current methods for option learning to demonstrate the importance of off-policy learning for data-efficiency . In addition , we compare HO2 with comparable policy optimization methods for flat policies and mixture policies without temporal abstraction . This allows us to isolate the individual impact of action abstraction ( in mixture and option policies ) and temporal abstraction in the option framework . The algorithm updates the policy via critic-weighted maximum-likelihood ( similar to Abdolmaleki et al . ( 2018b ) ; Wulfmeier et al . ( 2020 ) ) and combines these with an efficient dynamic programming procedure to infer option probabilities along trajectories and update all policy parts via backpropagation through the inference graph ( conceptually related to ( Rabiner , 1989 ; Shiarlis et al. , 2018 ; Smith et al. , 2018 ) ) . Intuitively , the approach can be understood as inferring option and action probabilities for off-policy trajectories in hindsight and maximizing the likelihood of good actions and options by backpropagating through the inference procedure . To stabilize policy updates , HO2 uses adaptive trust-region constraints , demonstrating the importance of robust policy optimization for hierarchical reinforcement learning ( HRL ) in line with recent work ( Zhang & Whiteson , 2019 ) . Rather than conditioning on executed options , the algorithm treats these as unobserved variables , computes the marginalized likelihood , and enables exact gradient computation and renders the algorithm independent of further approximations such as Monte Carlo estimates or continuous relaxation ( Li et al. , 2019 ) . As an additional benefit , the formulation of the inference graph also allows to impose hard constraints on the option termination frequency , thereby regularizing the learned solution without introducing additional weighted loss terms that are dependent on reward scale . We exploit this to further investigate temporal abstraction for learning from scratch and for pre-trained options . Experimentally , we demonstrate that HO2 outperforms existing option learning methods and that both action and temporal abstraction provide strong benefits , particularly in more demanding simulated robot manipulation tasks from raw pixel inputs . Our main contributions include : • a data-efficient off-policy option learning algorithm which outperforms existing option learning methods on common benchmarks . • careful analysis to isolate the impact of action abstraction and temporal abstraction in the option framework by comparing flat policies , mixture policies without temporal abstraction , and finally option policies . Our experiments demonstrate individual improvements from both forms of abstraction in complex 3D robot manipulation tasks from raw pixel inputs , and include ablations of other factors such as trust region constraints and off-policy versus on-policy data . • an intuitive method to further encourage temporal abstraction beyond the core method , using the inference graph to constrain option switches without additional weighted loss terms . We investigate differences between temporal abstraction in learning from scratch and pre-trained options and show that optimizing for temporally abstract behaviour in addition to simply providing the methods for its emergence mostly provides benefits in the context of pre-trained options . 2 PRELIMINARIES . Problem Setup We consider a reinforcement learning setting with an agent operating in a Markov Decision Process ( MDP ) consisting of the state space S , the action space A , and the transition probability p ( st+1|st , at ) of reaching state st+1 from state st when executing action at . The actions at are drawn from the agent ’ s policy π ( at|xt ) , where xt can either refer to the current state st of the agent or , in order to model dependencies on the previous steps , the trajectory until the current step , ht = { st , at−1 , st−1 , ... s0 , a0 } . Jointly , the transition dynamics and policy induce the marginal state visitation distribution p ( st ) . The discount factor γ together with the reward rt = r ( st , at ) gives rise to the expected return , which the agent aims to maximize : J ( π ) = Ep ( st ) , π ( at , st ) [ ∑∞ t=0 γ trt ] . We start by describing mixture policies as an intermediate between flat policies and the option framework ( see Figure 1 ) , which introduce a type of action abstraction via multiple low-level policies . Note that both Gaussian and mixture policies have been trained in prior work via similar policy optimization methods ( Abdolmaleki et al. , 2018a ; Wulfmeier et al. , 2020 ) , which we will extend towards option policies . In particular , we will make use of the connection to isolate the impact of action abstraction and temporal abstraction in the option framework in Section 4.2 . The next paragraphs focus on computing likelihoods of actions and options , which forms the foundation for the proposed critic-weighted maximum likelihood algorithm to learn hierarchical policies . Mixture Policies can be seen as a simplification of the options framework without initiation or termination condition , with resampling of options after every step ( i.e . no dependency between the options of timestep t and t+ 1 in Figure 1 ) . The joint probability of action and option is given as : πθ ( at , ot|st ) = πL ( at|st , ot ) πH ( ot|st ) , with πH ( ot|st ) = πC ( ot|st ) , ( 1 ) where πH and πL respectively represent high-level policy ( which for the mixture is equal to a Categorical distribution πC ) and low-level policy ( components of the resulting mixture distribution ) , and o is the index of the sub-policy or mixture component . 3 METHOD . Option Policies extend mixture policies by incorporating temporal abstraction . We follow the semi-MDP and call-and-return option model ( Sutton et al. , 1999 ) , defining an option as a triple ( I ( st , ot ) , π L ( at|st , ot ) , β ( st , ot ) ) . The initiation condition I describes an option ’ s probability to start in a state and is adapted as I ( st , ot ) = 1∀st ∈ S following ( Bacon et al. , 2017 ; Zhang & Whiteson , 2019 ) . The termination condition bt ∼ β ( st , ot ) denotes a Bernoulli distribution describing the option ’ s probability to terminate in any given state , and the action distribution for a given option is modelled by πL ( at|st , ot ) . Every time the agent observes a state , the current option ’ s termination condition is sampled . If subsequently no option is active , a new option is sampled from the controller πC ( ot|st ) . Finally , we sample from either the continued or new option to generate a new action . The transitions between options can therefore be described by p ( ot|st , ot−1 ) = { 1− β ( st , ot−1 ) + β ( st , ot−1 ) πC ( ot|st ) if ot = ot−1 β ( st , ot−1 ) π C ( ot|st ) otherwise ( 2 ) Equation 3 describes the direct connection between mixture and option policies . In both cases , the low-level policies πL only depend on the current state . However , where mixtures only depend on the current state st for the high-level probabilities πH , options take into account the history ht directly via the previous option as described in Equation 4. πθ ( at , ot|ht ) = πL ( at|st , ot ) πH ( ot|ht ) ( 3 ) Following the graphical model in Figure 1 , the probability of being in component ot at timestep t across a trajectory ht can be determined in a recursive manner based on the option probabilities of the previous timestep . For the first timestep , the probabilities are given by the high-level controller πH ( o0|h0 ) = πC ( o0|s0 ) and for all consecutive steps are computed as follows for M options using the option transition probabilities in Equation 2 . The factorization in Equation 3 and the following exact per-timestep marginalization over options in Equation 4 enable us to efficiently compute the likelihood of actions and options along off-policy trajectories ( instead of using option samples ) to efficiently perform intra-option learning ( Precup , 2000 ) for all options independently of the executed option . π̃H ( ot|ht ) = M∑ ot−1=1 [ p ( ot|st , ot−1 ) πH ( ot−1|ht−1 ) πL ( at−1|st−1 , ot−1 ) ] ( 4 ) In every step , we normalize the distribution following πH ( ot|ht ) = π̃H ( ot|ht ) / ∑M o′t=1 π̃H ( o′t|ht ) . This dynamic programming formulation ( similar to ( Rabiner , 1989 ; Shiarlis et al. , 2018 ; Smith et al. , 2018 ) ) allows the use of automatic differentiation in modern deep learning frameworks ( e.g . ( Abadi et al. , 2016 ) ) to backpropagate through the graph to determine the gradient updates for all policy parameters during the policy improvement step . We compute the probability of the option ot at time t given past states and actions by marginalizing out the previous option . Since ot−1 affects at−1 , we need to take the previous action that was actually chosen into account when performing the marginalization . The method is computationally more efficient than determining updates over all possible sequences of options independently and reduces variance compared to sampling-based approximations with the executed option or new samples during learning . In practice we find that removing the conditioning of component probabilities on actions ( the πL terms in Equation 4 ) improves performance and stability . This is furthermore shown in the ablations for the effects of conditioning on potentially stale actions in off-policy learning in Section 4.3 . We now continue by describing the main policy improvement algorithm , which requires the previously determined option probabilities . The principal three steps are : 1. update the critic following Eq . 5 ; 2. generate an intermediate , non-parametric policy based on the updated critic ( Eq . 6 ) ; 3. update the parametric policy to align to the non-parametric improvement ( Eq . 8 ) . Policy Evaluation In comparison to training mixture policies ( Wulfmeier et al. , 2020 ) , the critic for option policies is a function of s , a , and o since the current option influences the likelihood of future actions and thus rewards . Note that even though we express the policy as a function of the history ht , Q is a function of ot , st , at , since these are sufficient to render the future trajectory independent of the past ( see the graphical model in Figure 1 ) . In the constrained case Q also depends on nt , which we omit here for clarity . We define the TD ( 0 ) objective as min φ L ( φ ) = Est , at , ot∼D [ ( QT −Qφ ( st , at , ot ) ) 2 ] , ( 5 ) where the current states , actions , and options are sampled from the current replay buffer D. For the 1-step target QT = rt + γEst+1 , at+1 , ot+1 [ Q′ ( st+1 , at+1 , ot+1 ) ] , the expectation over the next state is approximated with the sample st+1 from the replay buffer , and we estimate the value by sampling actions and options according to at+1 , ot+1 ∼ πH ( ·|ht+1 ) following Equation 2 . Policy Improvement We follow an Expectation-Maximization procedure similar to ( Wulfmeier et al. , 2020 ; Abdolmaleki et al. , 2018b ) , which first computes an improved non-parametric policy and then updates the parametric policy to match this target . Given Q , all we require to optimize option policies is the possibility to sample from the policy and determine the log-likelihood ( gradient ) of actions and options under the policy given ht . The first step of policy improvement provides us with the non-parametric policy q ( at , ot|ht ) . max q J ( q ) = Eat , ot∼q , st∼D [ Qφ ( st , at , ot ) ] , s.t . Eht∼D [ KL ( q ( ·|ht ) ‖πθ ( ·|ht ) ) ] ≤ E , ( 6 ) where KL ( ·‖· ) denotes the Kullback-Leibler divergence , and E defines a bound on the KL . We can find the solution to the constrained optimization problem in Equation 6 in closed-form and obtain q ( at , ot|ht ) ∝ πθ ( at , ot|ht ) exp ( Qφ ( st , at , ot ) /η ) . ( 7 ) Practically speaking , this step computes samples from the previous policy and weights them based on the corresponding temperature-calibrated values of the critic . The temperature parameter η is computed following the dual of the Lagrangian . The derivation and final form of the dual can be found in Appendix C.1 , Equation 17 . To align parametric and non-parametric policies in the second step , we minimize their KL divergence . θ+ = arg min θ Eht∼D [ KL ( q ( ·|ht ) ‖πθ ( ·|ht ) ) ] , s.t . Eht∼D [ T ( πθ+ ( ·|ht ) ‖πθ ( ·|ht ) ) ] ≤ M ( 8 ) The distance function T in Equation 8 has a trust-region effect and stabilizes learning by constraining the change in the parametric policy . The computed option probabilities from Equation 4 are used in Equation 7 to enable sampling of options as well as Equation 8 to determine and maximize the likelihood of option samples under the policy . We can apply Lagrangian relaxation again and solve the primal as detailed in Appendix C.2 . Finally , we describe the complete pseudo-code for HO2 in Algorithm 1 , where we use target networks for policy π′ and Q-function Q′ to stabilize training . Algorithm 1 Hindsight Off-policy Options input : initial parameters for θ , η and φ , KL regularization parameters , set of trajectories τ while not done do sample trajectories τ from replay buffer // forward pass along sampled trajectories determine component probabilities πH ( ot|ht ) ( Eq . 4 ) sample actions aj and options oj from πθ ( ·|ht ) ( Eq . 3 ) to estimate expectations // compute gradients over batch for policy , Lagrangian multipliers and Q-function δθ ← −∇θ ∑ ht∈τ ∑ j [ exp ( Qφ ( st , aj , oj ) /η ) log πθ ( aj , oj |ht ) ] following Eq . 7 and 8 δη ← ∇ηg ( η ) = ∇ηη + η ∑ ht∈τ log ∑ j [ exp ( Qφ ( st , aj , oj ) /η ) ] following Eq . 17 δφ ← ∇φ ∑ ( st , at , ot ) ∈τ ( Qφ ( st , at , ot ) −QT ) 2 following Eq . 5 update θ , η , φ // apply gradient updates if number of iterations = target update then π′ = πθ , Q′ = Qφ // update target networks for policy π′ and value function Q′ Temporal Consistency Using options over longer sequences can help to reduce the search space and simplify exploration ( Sutton et al. , 1999 ; Harb et al. , 2018 ) . Previous approaches ( e.g . ( Harb et al. , 2018 ) ) rely on additional weighted loss terms which penalize option transitions . We instead introduce a mechanism to explicitly limit the maximum number of switches between options along a trajectory to increase temporal consistency . In comparison to additional loss terms , a parameter for the maximum number of switches can be chosen independently of the reward scale of an environment and provides an intuitive semantic interpretation . Both aspects simplify hyperparameter tuning . The 2D graph for computing option probabilities along a trajectory in Figure 2 is extended with a third dimension representing the number of switches between options . Practically , this means that we are modelling πH ( ot , nt|ht ) where nt represents the number of switches until timestep t. We can marginalize over all numbers of switches to determine option probabilities . In order to encourage option consistency across timesteps , we can instead sum over only a subset of nodes for all n ≤ N with N smaller than the maximal number of switches leading to πH ( ot|ht ) = ∑N nt=0 πH ( ot , nt|ht ) . For the first timestep , only 0 switches are possible and πH ( o0 , n0|h0 ) = 0 everywhere except for n0 = 0 and πH ( o0|n0 = 0 , h0 ) = πC ( o0|s0 ) . For further timesteps , all edges following option terminations β lead to the next step ’ s option probabilities with increased number of switches nt+1 = nt + 1 . All edges following option continuation lead to the probabilities for equal number of switches nt+1 = nt . This results in the computation of the joint distribution for t > 0 in Equation 9 with the option and switch index transitions p ( ot , nt|st , ot−1 , nt−1 ) described in Equation 19 in the Appendix and normalization as πH ( ot , nt|ht ) = π̃H ( ot , nt|ht ) / ∑M o′t=1 ∑L n′t=1 π̃H ( o′t , n ′ t|ht ) . π̃H ( ot , nt|ht ) = M , N∑ ot−1=1 , nt−1=1 p ( ot , nt|st , ot−1 , nt−1 ) πH ( ot−1 , nt−1|ht−1 ) πL ( at−1|st−1 , ot−1 ) ( 9 )

Summary: The paper considers the Hierarchical Reinforcement Learning setting, Options in particular, and proposes an algorithm that allows to learn both the high-level and low-level (option) policies at once, from off-policy samples. An original aspect of the algorithm is that it is easy to constrain the learned policies on how often they terminate an option and start a new one. This prevents the agent from learning tiny options that immediately terminate. It is unclear whether it can also be used to prevent the agent from learning a single big option that does everything.