Dr Jekyll & Mr Hyde: the strange case of off-policy policy updates

1 Introduction . Policy Gradient algorithms in Reinforcement Learning ( RL ) have enjoyed great success both theoretically [ 50 , 43 , 16 , 36 ] and empirically [ 23 , 38 , 37 , 27 ] . Their principle consists in optimizing an objective function J ( the expected discounted return ) through gradient steps , both being formally specified below [ 43 ] : J ( π ) .= E [ ∞∑ t=0 γtRt ∣∣∣∣S0 ∼ p0 , At ∼ π ( ·|St ) , St+1 ∼ p ( ·|St , At ) , Rt ∼ r ( ·|St , At ) ] ( 1 ) ∇θJ ( π ) = ∑ s∈S dπ , γ ( s ) ∑ a∈A qπ ( s , a ) ∇θπ ( a|s ) and θt+1 ←−−−−proj θt + ηt∇θtJ ( πt ) , ( 2 ) using standard Markov Decision Process notations ( recalled in App . A ) , where ηt is the learning rate and dπ , γ ( s ) . = ∑∞ t=0 γ tP ( St = s|π ) the discounted state density , the policies π .= πθ and πt . = πθt are implicitly parametrized by θ for conciseness , and←−−−−proj denotes the projection onto the parameter space . We observe that the update established by the policy gradient theorem is proportional to the state density of the current policy . This is problematic , as it implies that no value improvement can be induced in off-policy states ( i.e . states that are rarely visited ) . As a consequence , ∗Equal contribution . Correspondence to : rolaroch @ microsoft.com . All code available at http : //aka.ms/jnh . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . planning is inefficient in those states to the point of potentially compromising convergence to the optimal policy . To address the planning issue in off-policy states , we generalize the policy gradient theory by considering the more general policy update : U ( θ , d ) . = ∑ s∈S d ( s ) ∑ a∈A qπ ( s , a ) ∇θπ ( a|s ) and θt+1 ←−−−−proj θt + ηtU ( θt , dt ) , ( 3 ) where dt designates the state distribution on which the policy is updated . It does not have to match the distribution induced by π and updates may therefore be off-policy . We note that U ( θ , d ) is not a gradient in the general case , hence the use of the term update . We prove in the direct : πθ . = θ and softmax : πθ ∼̇ exp ( θ ) parametrizations that following the exact policy updates induces a sequence of value functions qt . = qπt that is monotonously increasing , upper bounded , and thereby converges to a q∞ . We then show that the condition “ ∀s ∈ S , ∑ t ηtdt ( s ) =∞ ” on the sequence ( dt ) is necessary and sufficient for q∞ to be optimal . Our result generalizes previous theorems to broader updates and milder assumptions [ 2 , 22 ] . Finally , we significantly improve the existing asymptotic convergence rates . We show that ( qt ) converges to the optimal value i ) exactly in finite time for the direct parametrization ( previously in O ( t−1/2|S||A| ( 1− γ ) −6 ) [ 2 ] ) , and ii ) in O ( t−1|S||A| ( 1− γ ) −2 ) for the softmax parametrization , to be compared with O ( t−1|S|2|A|2 ( 1− γ ) −6 ) [ 22 ] . Building on our theoretical results , we design a novel algorithm : Dr Jekyll and Mr Hyde ( J & H ) . Its principle consists in training two independent policies : a pure exploitation one ( Dr Jekyll ) and a pure exploration one ( Mr Hyde ) , and give control to either one for full trajectories . J & H ’ s independent policies allow to record two separate replay buffers : one on-policy ( Dr Jekyll ’ s ) and one off-policy ( Mr Hyde ’ s ) , and therefore to update J & H ’ s models with any desired mixture of on-policy and off-policy updates . Beyond an algorithm , J & H introduces conditions based on our analysis that actor-critic algorithms should follow to properly plan off-policy . Furthermore , the separation of exploration and exploitation allows to stabilise the training of the exploitation policy while ensuring a full coverage of the state-action space through deep exploration [ 30 ] . We empirically validate our theoretical analysis and algorithmic innovation in both planning and RL settings . In the planning setting where we assume that qt is exactly known , we analyze the impact of the off-policiness of dt and conclude that , while it improves over classic policy gradient , the theoretical sufficient condition does not allow to converge in a reasonable amount of time in very hard planning tasks : we thereby recommend to enforce the stronger condition “ ∀s ∈ S , ∑t ηtdt ( s ) ∈ Ω ( t ) ” . In the reinforcement learning domain , where qt must be estimated from the collected transitions , the off-policiness of J & H ’ s policy updates allows by design to satisfy the recommendation as long as Mr Hyde is able to cover the whole state space . This leads J & H to significantly outperform all competing actor-critic algorithms in the hard planning tasks and to be competitive with gradient updates in simple planning problems . As a proof of concept , we also test J & H in a deep reinforcement learning setting and show that it outperforms various baselines on a simple environment . The paper is organized as follows . Section 2 develops the policy update theory . Section 3 describes J & H and positions it with respect to the literature . Section 4 presents the experiments and their results . Finally , Section 5 concludes the paper . The interested reader may refer to the appendix for the proofs ( App . B ) , the domains ( App . C ) , and the full experiment reports ( App . D , E , and F ) . Finally , the supplementary material contains the code for the experiments : algorithms and environments . 2 Theoretical analysis . First , we recall some background : policy gradient methods depend on a parametrization θ ∈ R|S|×|A| of the policy . Like [ 2 , 22 ] , we will focus on the classic direct and softmax parametrizations : • direct : π ( a|s ) .= θs , a , for which us , a = dt ( s ) qπ ( s , a ) and the projection is on the simplex . • softmax : π ( a|s ) .= exp ( θs , a ) ∑ a′ exp ( θs , a′ ) , for which us , a = dt ( s ) π ( a|s ) ( qπ ( s , a ) − vπ ( s ) ) . For both parametrizations , [ 2 ] proves that following the policy gradient∇θJ ( π ) from Eq . ( 2 ) eventually leads to the optimal policy in finite MDPs under the following assumptions and condition2 : A1 . The model ( p0 , p , r ) is known . A2 . The initial state-distribution covers the full state space : ∀s ∈ S , p0 ( s ) > 0 . C3 . The learning rate is constant : ηt = ( 1−γ ) 3 2γ|A| ( direct ) and ηt = ( 1−γ ) 3 8 ( softmax ) . Those are stringent requirements . A1 implies that the values qπ and state density dπ , γ of the policy are known exactly . A2 is generally not satisfied in standard RL domains . Finally , verifying C3 makes learning slow to the point that it becomes impractical . 2.1 Convergence properties . In this section , we study , under A1 , the convergence properties of the sequence of value functions ( qt ) induced by the update rule defined in Eq . ( 3 ) . We note that U ( θ , d ) is not a gradient in general . Our first theoretical result is the monotonicity of the value function sequence ( qt ) . Theorem 1 ( Monotonicity under the direct and softmax parametrization ) . Under A1 , the sequence of value functions qt . = qπt and vt . = vπt are monotonously increasing . By the monotonous convergence theorem , Thm . 1 directly implies the convergence of ( qt ) : Corollary 1 ( Convergence under the direct and softmax parametrization ) . Under A1 , the sequence of value functions qt uniformly converges : q∞ . = limt→∞ qt . In order to go further and prove convergence to a global optimal value , we need to enforce an additional condition . Indeed , the update U relies multiplicatively on dt which could be equal ( or tend very fast ) to 0 in some states , compromising the policy ’ s ability to reach the optimal value . This argument is not only theoretical , it has been observed by many that policy gradient can get stuck in suboptimal policies , even with entropy regularization . We designed our chain domain to exhibit such behaviour ( see Section 4.1 ) . Next , we show that ( qt ) converges to the optimal value function under some necessary and sufficient condition . Theorem 2 ( Optimality under the direct and softmax parametrization ) . Under A1 , the following condition : C4 . Each state s is updated with weights that sum to infinity over time : ∑∞ t=0 ηtdt ( s ) =∞ , is necessary and sufficient to guarantee that the sequence of value functions ( qt ) converges to optimality : q∞ = q ? . = maxπ∈Π qπ . Proof sketch . In both direct and softmax parametrizations , we assume there exists a state-action pair ( s , a ) advantageous with respect to the state value limits q∞ and v∞ . = limt→∞ vt , that is : adv∞ ( s , a ) . = q∞ ( s , a ) − v∞ ( s ) > 0 . We then prove that in this state s , the policy improvement yielded by the update is lower bounded by a linear function of the update weight ηtdt ( s ) . Both parametrizations require different proof techniques and are dealt with in different theorems . Summing over t , we notice that this lower bounding sum diverges to infinity , which contradicts Corollary 1 . We therefore infer that there can not exist such a state-action pair , which allows us to conclude that no policy improvement is possible , and thus that the values are optimal : q∞ = maxπ∈Π qπ . For the necessity of C4 , we show that the parameter update is upper bounded in both parametrizations by a term linear in the action gap . By choosing the reward function adversarially , we may set it sufficiently small so that the sum of all the gradient steps is insufficient to reach optimality . Thm . 2 is impactful along five dimensions . Practice of on-policy undiscounted updates : ti li i t t : r f - s sac ce c c e a eti li i t t : r f - As Thm . 2 is applicable to any distribution sequence , it allows us in particular to consider the practice of using on-policy undiscounted updates : dt . = dπt,1 . 2An assumption A # is a requirement on the environment or the application setting , while a condition C # is a requirement that may be enforced by a dedicated algorithm in any environment or setting . Thus , it resolves a longstanding gap between the policy gradient theory and the actor-critic algorithm implementations [ 12 , 7 , 51 , 17 , 1 , 19 , 41 ] . This mismatch and its lack of theoretical grounding were identified in [ 45 ] . Later , [ 26 ] proved that the practitioners ’ undiscounted updates are not the gradient of any function and may be strongly biased under state aliasing . Recently , [ 53 ] studied the practical advantage of the undiscounted updates from a representation learning perspective . C4 encompasses both standard policy gradient and the undiscounted update rule : convergence to optimality of both is guaranteed as long as C4 is verified . Our analysis thus shows that the convergence properties of policy gradient and undiscounted updates require the same set of assumptions and conditions . Conversely , it is possible to prove convergence to sub-optimal policies of either when C4 is violated . In other words , Thm . 2 allows to reduce the study of specific algorithms to whether they verify C4 . Experience replay and off-policy updates : i l li t : r r ff- se e ce e a a c a ei l li t : r r ff- It also justifies the use of an experience replay for the actor , a trick also widely used in the literature to distributed the training over several agents [ 24 , 34 , 49 , 14 , 35 ] . Furthermore , while widely overlooked in the literature , we prove that off-policy updates have an even higher impact , since they are necessary to guarantee the convergence to optimality3 . We leverage this discovery to introduce new design principles and a novel algorithm in Section 3 . Policy gradient theory : li i t t : r rc a e eli i t t : r r Next , Thm . 2 generalizes previous optimality convergence results along two axes . The initial state distribution p0 is not required to cover the state space anymore ( aka A2 ) . A2 is unrealistic in most applications and can not be enforced by an algorithm . Relaxing it is an important open problem [ 2 ] , we disprove it in the full class of MDP . However , it might be possible to find a class of MDPs larger than the one verifying A2 where policy gradient also converges . Additionally , the condition on our learning rates is also much more flexible than C3 . Density vs. policy regularization : it . li l i ti : s s r re c e a zait . li l i ti : r r In contrast , C4 can be controlled by a dedicated algorithm . This theoretical result promotes the principle of density-based regularization [ 21 , 32 ] , at the expense of policy-based regularization that can not guarantee that C4 is satisfied and sometimes fails to plan over the whole state space ( our chain domain experiments in Section 4 illustrate it well ) . Towards a generalized policy iteration theorem : li li it ti t : r s r r ra a e e a ze c e a e eli li it ti t : r r r r Finally , by combining Thm . 2 , which analyzes policy improvement , with a thorough theoretical analysis of the policy evaluation step ( discussed in Section 3.1 ) , we see a path towards results describing conditions on both components of generalized policy iteration that guarantee convergence to optimality [ 4 ] . Furthermore , Thm . 2 gives sufficient conditions on the policy improvement step . The generalized policy iteration theorem was conjectured in [ 42 ] but never proved .

Summary: In this paper the authors propose an extension to policy gradient theory which considers policy updates under any state distribution density. Moreover, they present the necessary conditions for convergence to optimality under their new formulation. Subsequently, they implement the ideas supported by their theory and construct a new agent which they call J&H (Dr Jekyll and Dr Hyde). The main idea of this agent is to consider two policies one which is fully exploitative (Jekyll) and one which is completely exploratory (Hyde). This way during training they can combine on-policy and off-policy updates. Finally, they test their agent in a series of simple MDPs and show that their proposed agent is superior at  recovering from converging to a suboptimal policy.