Dr Jekyll & Mr Hyde: the strange case of off-policy policy updates

1 Introduction . Policy Gradient algorithms in Reinforcement Learning ( RL ) have enjoyed great success both theoretically [ 50 , 43 , 16 , 36 ] and empirically [ 23 , 38 , 37 , 27 ] . Their principle consists in optimizing an objective function J ( the expected discounted return ) through gradient steps , both being formally specified below [ 43 ] : J ( π ) .= E [ ∞∑ t=0 γtRt ∣∣∣∣S0 ∼ p0 , At ∼ π ( ·|St ) , St+1 ∼ p ( ·|St , At ) , Rt ∼ r ( ·|St , At ) ] ( 1 ) ∇θJ ( π ) = ∑ s∈S dπ , γ ( s ) ∑ a∈A qπ ( s , a ) ∇θπ ( a|s ) and θt+1 ←−−−−proj θt + ηt∇θtJ ( πt ) , ( 2 ) using standard Markov Decision Process notations ( recalled in App . A ) , where ηt is the learning rate and dπ , γ ( s ) . = ∑∞ t=0 γ tP ( St = s|π ) the discounted state density , the policies π .= πθ and πt . = πθt are implicitly parametrized by θ for conciseness , and←−−−−proj denotes the projection onto the parameter space . We observe that the update established by the policy gradient theorem is proportional to the state density of the current policy . This is problematic , as it implies that no value improvement can be induced in off-policy states ( i.e . states that are rarely visited ) . As a consequence , ∗Equal contribution . Correspondence to : rolaroch @ microsoft.com . All code available at http : //aka.ms/jnh . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . planning is inefficient in those states to the point of potentially compromising convergence to the optimal policy . To address the planning issue in off-policy states , we generalize the policy gradient theory by considering the more general policy update : U ( θ , d ) . = ∑ s∈S d ( s ) ∑ a∈A qπ ( s , a ) ∇θπ ( a|s ) and θt+1 ←−−−−proj θt + ηtU ( θt , dt ) , ( 3 ) where dt designates the state distribution on which the policy is updated . It does not have to match the distribution induced by π and updates may therefore be off-policy . We note that U ( θ , d ) is not a gradient in the general case , hence the use of the term update . We prove in the direct : πθ . = θ and softmax : πθ ∼̇ exp ( θ ) parametrizations that following the exact policy updates induces a sequence of value functions qt . = qπt that is monotonously increasing , upper bounded , and thereby converges to a q∞ . We then show that the condition “ ∀s ∈ S , ∑ t ηtdt ( s ) =∞ ” on the sequence ( dt ) is necessary and sufficient for q∞ to be optimal . Our result generalizes previous theorems to broader updates and milder assumptions [ 2 , 22 ] . Finally , we significantly improve the existing asymptotic convergence rates . We show that ( qt ) converges to the optimal value i ) exactly in finite time for the direct parametrization ( previously in O ( t−1/2|S||A| ( 1− γ ) −6 ) [ 2 ] ) , and ii ) in O ( t−1|S||A| ( 1− γ ) −2 ) for the softmax parametrization , to be compared with O ( t−1|S|2|A|2 ( 1− γ ) −6 ) [ 22 ] . Building on our theoretical results , we design a novel algorithm : Dr Jekyll and Mr Hyde ( J & H ) . Its principle consists in training two independent policies : a pure exploitation one ( Dr Jekyll ) and a pure exploration one ( Mr Hyde ) , and give control to either one for full trajectories . J & H ’ s independent policies allow to record two separate replay buffers : one on-policy ( Dr Jekyll ’ s ) and one off-policy ( Mr Hyde ’ s ) , and therefore to update J & H ’ s models with any desired mixture of on-policy and off-policy updates . Beyond an algorithm , J & H introduces conditions based on our analysis that actor-critic algorithms should follow to properly plan off-policy . Furthermore , the separation of exploration and exploitation allows to stabilise the training of the exploitation policy while ensuring a full coverage of the state-action space through deep exploration [ 30 ] . We empirically validate our theoretical analysis and algorithmic innovation in both planning and RL settings . In the planning setting where we assume that qt is exactly known , we analyze the impact of the off-policiness of dt and conclude that , while it improves over classic policy gradient , the theoretical sufficient condition does not allow to converge in a reasonable amount of time in very hard planning tasks : we thereby recommend to enforce the stronger condition “ ∀s ∈ S , ∑t ηtdt ( s ) ∈ Ω ( t ) ” . In the reinforcement learning domain , where qt must be estimated from the collected transitions , the off-policiness of J & H ’ s policy updates allows by design to satisfy the recommendation as long as Mr Hyde is able to cover the whole state space . This leads J & H to significantly outperform all competing actor-critic algorithms in the hard planning tasks and to be competitive with gradient updates in simple planning problems . As a proof of concept , we also test J & H in a deep reinforcement learning setting and show that it outperforms various baselines on a simple environment . The paper is organized as follows . Section 2 develops the policy update theory . Section 3 describes J & H and positions it with respect to the literature . Section 4 presents the experiments and their results . Finally , Section 5 concludes the paper . The interested reader may refer to the appendix for the proofs ( App . B ) , the domains ( App . C ) , and the full experiment reports ( App . D , E , and F ) . Finally , the supplementary material contains the code for the experiments : algorithms and environments . 2 Theoretical analysis . First , we recall some background : policy gradient methods depend on a parametrization θ ∈ R|S|×|A| of the policy . Like [ 2 , 22 ] , we will focus on the classic direct and softmax parametrizations : • direct : π ( a|s ) .= θs , a , for which us , a = dt ( s ) qπ ( s , a ) and the projection is on the simplex . • softmax : π ( a|s ) .= exp ( θs , a ) ∑ a′ exp ( θs , a′ ) , for which us , a = dt ( s ) π ( a|s ) ( qπ ( s , a ) − vπ ( s ) ) . For both parametrizations , [ 2 ] proves that following the policy gradient∇θJ ( π ) from Eq . ( 2 ) eventually leads to the optimal policy in finite MDPs under the following assumptions and condition2 : A1 . The model ( p0 , p , r ) is known . A2 . The initial state-distribution covers the full state space : ∀s ∈ S , p0 ( s ) > 0 . C3 . The learning rate is constant : ηt = ( 1−γ ) 3 2γ|A| ( direct ) and ηt = ( 1−γ ) 3 8 ( softmax ) . Those are stringent requirements . A1 implies that the values qπ and state density dπ , γ of the policy are known exactly . A2 is generally not satisfied in standard RL domains . Finally , verifying C3 makes learning slow to the point that it becomes impractical . 2.1 Convergence properties . In this section , we study , under A1 , the convergence properties of the sequence of value functions ( qt ) induced by the update rule defined in Eq . ( 3 ) . We note that U ( θ , d ) is not a gradient in general . Our first theoretical result is the monotonicity of the value function sequence ( qt ) . Theorem 1 ( Monotonicity under the direct and softmax parametrization ) . Under A1 , the sequence of value functions qt . = qπt and vt . = vπt are monotonously increasing . By the monotonous convergence theorem , Thm . 1 directly implies the convergence of ( qt ) : Corollary 1 ( Convergence under the direct and softmax parametrization ) . Under A1 , the sequence of value functions qt uniformly converges : q∞ . = limt→∞ qt . In order to go further and prove convergence to a global optimal value , we need to enforce an additional condition . Indeed , the update U relies multiplicatively on dt which could be equal ( or tend very fast ) to 0 in some states , compromising the policy ’ s ability to reach the optimal value . This argument is not only theoretical , it has been observed by many that policy gradient can get stuck in suboptimal policies , even with entropy regularization . We designed our chain domain to exhibit such behaviour ( see Section 4.1 ) . Next , we show that ( qt ) converges to the optimal value function under some necessary and sufficient condition . Theorem 2 ( Optimality under the direct and softmax parametrization ) . Under A1 , the following condition : C4 . Each state s is updated with weights that sum to infinity over time : ∑∞ t=0 ηtdt ( s ) =∞ , is necessary and sufficient to guarantee that the sequence of value functions ( qt ) converges to optimality : q∞ = q ? . = maxπ∈Π qπ . Proof sketch . In both direct and softmax parametrizations , we assume there exists a state-action pair ( s , a ) advantageous with respect to the state value limits q∞ and v∞ . = limt→∞ vt , that is : adv∞ ( s , a ) . = q∞ ( s , a ) − v∞ ( s ) > 0 . We then prove that in this state s , the policy improvement yielded by the update is lower bounded by a linear function of the update weight ηtdt ( s ) . Both parametrizations require different proof techniques and are dealt with in different theorems . Summing over t , we notice that this lower bounding sum diverges to infinity , which contradicts Corollary 1 . We therefore infer that there can not exist such a state-action pair , which allows us to conclude that no policy improvement is possible , and thus that the values are optimal : q∞ = maxπ∈Π qπ . For the necessity of C4 , we show that the parameter update is upper bounded in both parametrizations by a term linear in the action gap . By choosing the reward function adversarially , we may set it sufficiently small so that the sum of all the gradient steps is insufficient to reach optimality . Thm . 2 is impactful along five dimensions . Practice of on-policy undiscounted updates : ti li i t t : r f - s sac ce c c e a eti li i t t : r f - As Thm . 2 is applicable to any distribution sequence , it allows us in particular to consider the practice of using on-policy undiscounted updates : dt . = dπt,1 . 2An assumption A # is a requirement on the environment or the application setting , while a condition C # is a requirement that may be enforced by a dedicated algorithm in any environment or setting . Thus , it resolves a longstanding gap between the policy gradient theory and the actor-critic algorithm implementations [ 12 , 7 , 51 , 17 , 1 , 19 , 41 ] . This mismatch and its lack of theoretical grounding were identified in [ 45 ] . Later , [ 26 ] proved that the practitioners ’ undiscounted updates are not the gradient of any function and may be strongly biased under state aliasing . Recently , [ 53 ] studied the practical advantage of the undiscounted updates from a representation learning perspective . C4 encompasses both standard policy gradient and the undiscounted update rule : convergence to optimality of both is guaranteed as long as C4 is verified . Our analysis thus shows that the convergence properties of policy gradient and undiscounted updates require the same set of assumptions and conditions . Conversely , it is possible to prove convergence to sub-optimal policies of either when C4 is violated . In other words , Thm . 2 allows to reduce the study of specific algorithms to whether they verify C4 . Experience replay and off-policy updates : i l li t : r r ff- se e ce e a a c a ei l li t : r r ff- It also justifies the use of an experience replay for the actor , a trick also widely used in the literature to distributed the training over several agents [ 24 , 34 , 49 , 14 , 35 ] . Furthermore , while widely overlooked in the literature , we prove that off-policy updates have an even higher impact , since they are necessary to guarantee the convergence to optimality3 . We leverage this discovery to introduce new design principles and a novel algorithm in Section 3 . Policy gradient theory : li i t t : r rc a e eli i t t : r r Next , Thm . 2 generalizes previous optimality convergence results along two axes . The initial state distribution p0 is not required to cover the state space anymore ( aka A2 ) . A2 is unrealistic in most applications and can not be enforced by an algorithm . Relaxing it is an important open problem [ 2 ] , we disprove it in the full class of MDP . However , it might be possible to find a class of MDPs larger than the one verifying A2 where policy gradient also converges . Additionally , the condition on our learning rates is also much more flexible than C3 . Density vs. policy regularization : it . li l i ti : s s r re c e a zait . li l i ti : r r In contrast , C4 can be controlled by a dedicated algorithm . This theoretical result promotes the principle of density-based regularization [ 21 , 32 ] , at the expense of policy-based regularization that can not guarantee that C4 is satisfied and sometimes fails to plan over the whole state space ( our chain domain experiments in Section 4 illustrate it well ) . Towards a generalized policy iteration theorem : li li it ti t : r s r r ra a e e a ze c e a e eli li it ti t : r r r r Finally , by combining Thm . 2 , which analyzes policy improvement , with a thorough theoretical analysis of the policy evaluation step ( discussed in Section 3.1 ) , we see a path towards results describing conditions on both components of generalized policy iteration that guarantee convergence to optimality [ 4 ] . Furthermore , Thm . 2 gives sufficient conditions on the policy improvement step . The generalized policy iteration theorem was conjectured in [ 42 ] but never proved .

Summary: The paper extends the policy gradient theorem from purely on-policy gradients to arbitrary state distributions. The authors provide proofs of convergence and (improved) rate of convergence under milder assumptions than the original policy gradient theorem. They propose a simple agent that separately performs on-policy and off-policy updates, and include a carefully-designed toy experiment that elegantly shows their algorithm achieve globally-optimal results when traditional policy gradient approaches fail. They also demonstrate their approach on a simple deep-RL benchmark.