Policy path programming

1 INTRODUCTION . Reinforcement learning algorithms can leverage internal models of environment dynamics to facilitate the development of good control policies ( Sutton & Barto , 2018 ) . Dynamic programming methods iteratively implement one-step , full-width backups in order to propagate reward information across a state-space representation and then use this information to perform policy updates ( Bellman , 1954 ) . Stochastic approximations of this approach underpin a wide range of model-free reinforcement learning algorithms which can be enhanced by the ability to query samples from an “ internal ” environment model as in the DYNA architecture ( Sutton , 1990 ) . State-space search strategies apply heuristic principles to efficiently sample multi-step paths from internal models and have formed a core component of recent state-of-the-art game playing agents ( Silver et al. , 2016 ) . Model-based policy search ( Deisenroth & Rasmussen , 2011 ; Abdolmaleki et al. , 2015 ) and gradient methods ( Sutton et al. , 1999 ) require sampled paths to approximate policy gradients based on either pure Monte Carlo estimation or by integrating long-run value estimates . All such methods rely on alternating between simulating paths over various horizons and then using this information to improve the policy either directly or indirectly by backing up value estimates and then inferring a policy ( Sutton & Barto , 2018 ; Puterman , 1994 ) . In this study , we introduce policy path programming ( 3P ) which , given an environment model , normatively improves policies in a manner sensitive to the distribution of all future paths without requiring multi-step simulations . In particular , path programming follows the unique trajectory through policy space which iteratively maximizes the expected cumulative reward obtained . We develop 3P for entropy-regularized discounted Markov decision processes ( Levine , 2018 ) . In the entropy-regularized MDP framework , a policy complexity penalty is added to the expected cumulative reward objective ( Levine , 2018 ) ( see Section 2 for details ) . Entropy regularization has several implications which have been investigated previously . The entropy penalty forces policies to be stochastic thereby naturally integrating an exploratory drive into the policy optimization process ( Ahmed et al. , 2018 ) . In particular , the optimal policy can be immediately derived using calculus of variations as a Boltzmann-Gibbs distribution and reveals a path-based consistency law relating optimal value estimates and optimal policy probabilities which can be exploited to form a learning objective ( Nachum et al. , 2017 ) . Furthermore , several studies have successfully used the relative entropy penalty to impose a conservative policy “ trust region ” to constrain policy updates thereby reducing erroneous policy steps due to the high variance in gradient estimation ( Azar et al. , 2012 ; Schulman et al. , 2015 ) . With this setup , we seek to compute this gradient exactly based on a consideration of the distribution of all possible paths that the currently estimated policy will generate . Therefore , we express the MDP objective as a “ sum-over-paths ” and develop our model in this representation . In the sum-over-paths formalism ( Kappen , 2005 ; Theodorou et al. , 2013 ) , the central object of interest is not a state-action pair ( Fig . 1A ) , as is the standard perspective in reinforcement learning in discrete MDPs , but complete state-action sequences or paths ( Fig . 1B ) . The entropy-regularized cumulative reward objective can be re-written in terms of paths and a path policy can be expressed as an assignment of a probability to each path individually ( see Section 3 for details ) . Gradient ascent in the space of policies over paths integrates information over all possible future paths in expectation at every step . 3P is defined as the gradient ascent algorithm which performs policy updates with respect to this path gradient . As a policy iteration method , we show that this is analogous to full-depth , full-width backups . Furthermore , we describe the associated natural path gradient which is fundamentally distinct from previous natural gradient techniques which utilize the asymptotic time limit of local , state-specific , natural policy gradients ( Kakade , 2001 ; Peters et al. , 2005 ) . In Section 2 , we summarize the mathematical framework of entropy-regularized MDPs from the path-based perspective and define our notation . In Section 3 , we derive policy path programming . In Section 4 , we apply the algorithm in state-spaces drawn from a variety of domains and analyze the resulting policy optimization dynamics . We conclude with a discussion in Section 5 . 2 BACKGROUND AND NOTATION . A B Path representation State-action representation We develop the path programming formalism in the context of stochastic environmental dynamics . A state-space X is composed of states x ∈ X and the policy π ( aj |si ) describes the probability of selecting action aj in state si . The reward associated with transitioning from state si to state sk after selecting action aj is denoted R ( si , aj , sk ) ≡ Rijk . Bold-typed notation , s ∈ S , a ∈ A , and u ∈ U denotes sequences of states s ∈ S , actions a ∈ A , and state-action combinations u ≡ ( su , au ) ∈ U respectively . The action set A is the union of the sets of actions available at each state A = ∪si∈SAi . A valid state-action sequence u : = ( . . . , st , at , st+1 , at+1 , . . . ) is referred to as a state-action path . The path probability p ( u ) is defined as the joint distribution over states s and actions a p ( u ) : = ∞∏ t=0 p ( st+1|st , at ) π ( at|st ) = p ( s+1|s , a ) π ( a|s ) ( 1 ) where π ( a|s ) : = ∞∏ t=0 π ( at|st ) , p ( s+1|s , a ) : = ∞∏ t=0 p ( st+1|st , at ) . ( 2 ) The environment dynamics are expressed in the function p ( sk|si , aj ) which denotes the probability of transitioning to state sk after selecting action aj in state si . The MDP objective as a sum-over-paths ( Kappen , 2005 ; Theodorou et al. , 2013 ) is π∗ : = argmax π 〈R ( u ) 〉p R ( u ) = ∞∑ t=0 R ( st , at , st+1 ) ( 3 ) where the angled brackets 〈·〉p denote the expectation operation over the path density p. The form of the MDP objective in Equation 3 expresses a sequential decision-making problem as the determination a single decision but over paths . From this point of view , the max operation in Eqn . 3 is a full-width , full-depth policy iteration which converges in one step . We use the term full-depth because the paths incorporate information over all horizons ( thus deep in time ) and the term full-width because the max operation considers all paths ( Fig . 1D ) . In contrast , policy iteration algorithms use full-width , one-step backups ( Fig . 1C ) . This full-depth , full-width max operation is intractable since it requires a search over all possible paths and so we relax this problem using entropy regularization . In the entropy-regularized reinforcement learning framework ( Levine , 2018 ) , a policy description length penalty for each path u weighted by a temperature parameter τ is added to the MDP objective ( Eqn . 3 ) . The relative entropy regularizer DKL [ π||π0 ] measures policy complexity as the deviation from a prior ( possibly non-uniform ) policy π0 ( Todorov , 2007 ; Kappen et al. , 2012 ; Theodorou et al. , 2013 ) . In this case , the policy description length penalty is −τ log π ( a|s ) π0 ( a|s ) and the resulting entropy-regularized transition rewards Jijk : = J ( si , aj , sk ) , path rewards J ( u ) , and policy objective J [ π ] are then Jijk : = Rijk − τ log πij + τ log π0ij J ( u ) : = ∞∑ t=0 J ( at , st , st+1 ) = R ( u ) − τ logπ ( a|s ) + τ logπ0 ( a|s ) = R ( u ) − τ logp ( u ) + τ logp0 ( u ) J [ π ] = 〈J ( u ) 〉p = −τDKL [ p ( u ) ||p0 ( u ) eτ −1R ( u ) ] . ( 4 ) where we have made use of the compressed notation πij ≡ π ( aj |si ) and p0 ( u ) : = p ( s+1|s , a ) π0 ( a|s ) . From an information-theoretic point of view , the optimal policy π∗ which maximizes J [ π ] gives the best trade-off between maximizing reward and minimizing policy encoding costs . An implication of the description length penalty is that encoding deterministic transitions is infinitely costly [ logπ ( a|s ) → −∞ as π ( a|s ) → 0 ] and therefore the optimal policy will be stochastic . Taking the temperature parameter to zero τ → 0 recovers the standard MDP problem of identifying a deterministic policy in pursuit of maximum expected cumulative reward . In the main text , policy path programming ( 3P ) is developed for entropy-regularized MDPs with stochastic environment dynamics . It is straightforward to derive analogous update equations in the presence of deterministic environmental transitions which correspond to the subset of control problems known as KL-control ( Kappen et al. , 2012 ) or linearly-solvable Markov decision processes ( Todorov , 2007 ) ) . Furthermore , our analysis can be applied to MDPs with absorbing states . Thus , path programming can be applied to a broad class of MDPs . 3 POLICY PATH PROGRAMMING IN DISCRETE MARKOV DECISION PROCESSES . The policy objective function ( Eqn . 4 ) can be re-expressed as J [ π ] = ∑ u∈U ∏ si , sk∈S aj∈Ai ( πijpijk ) nijk ( u ) ∑ si , sk∈S aj∈Ai nijk ( u ) [ Rijk − τ ( log πij − log π0ij ) ] s.t . πij > 0 ∀si ∈ S , aj ∈ Ai , ∑ aj∈A πij = 1 ∀si ∈ S ( 5 ) where we have expressed the objective ( Eqn . 4 ) in terms of counters nijk ( u ) which quantify the number of times that sk is occupied after selecting action aj in state si on path u . We reparametrize the policy parameters πij in terms of natural parameters Aij in an exponential model πij : = eAij ( Nagaoka , 2005 ) . These natural parameters are examples of action preferences in reinforcement learning parlance1 ( Sutton & Barto , 2018 ) and can take any negative real value . Substituting Aij : = log πij , J [ A ] = ∑ u∈S eA·n ( u ) +C·n ( u ) ∑ si , sk∈X aj∈Ai nijk ( u ) ( Rijk − τAij + τA0ij ) eA·n ( u ) = e ∑ si , aj , sk Aijnijk ( u ) = e ∑ si , aj Aijnij ( u ) eC·n ( u ) = e ∑ si , aj , sk Cijknijk ( u ) ( 6 ) where Cijk : = log pijk and A0ij : = log π 0 ij has been analogously substituted , and n is a tensor with components nijk and [ A ] ij : = Aij have been used for the event counters and action preferences respectively . Considering the set of probabilities e ( A+C ) ·n ( u ) parametrized by A as an exponential family ( Nagaoka , 2005 ) , the vector n of transition counters nijk ( u ) constitutes a sufficient statistic for the path u . Given that the policy transition probabilities πij = eAij are drawn from the action preferences Aij ∈ R− via an exponential transformation , we are guaranteed that 0 < πij ≤ 1 for all state-action combinations . In order to ensure that πt always forms a probability distribution at every state , we eliminate a redundant action preference at each state . This is accomplished by defining an arbitrary transition probability at each state in terms of the probabilities of alternative transitions at that state . We index this dependent action preference using an ω subscript as in Aiiω in order to distinguish it from the independent action preferences which will be directly modified during policy optimization . We define iω as the action index of an arbitrary action available in state si . Under the local policy normalization constraint , the action preferences are equivalently constrained via Aiiω = log 1− ∑ aiω 6=aj∈Ai eAij . ( 7 )

Summary: This work proposes a policy iteration algorithm that implements full-depth, full-width backups in contrast to one-step, full-width methods. The authors go over existing algorithms and talks a bit how their proposal conceptually differs in how it performs said backups. They provide a bit of intuition to help explain their algorithm's derivation. Finally, they provide a few experiments showing that their algorithm works.