Fast Deterministic Stackelberg Actor-Critic

Most advanced Actor-Critic ( AC ) approaches update the actor and critic concurrently through ( stochastic ) Gradient Descents ( GD ) , which may be trapped into bad local optimality due to the instability of these simultaneous updating schemes . Stackelberg AC learning scheme alleviates these limitations by adding a compensated indirect gradient terms to the GD . However , the indirect gradient terms are time-consuming to calculate , and the convergence rate is also relatively slow . To alleviates these challenges , we find that in the Deterministic Policy Gradient family , by removing the terms that contain Hessian matrices and adopting the block diagonal approximation technique to approximate the remaining inverse matrices , we can construct an approximated Stackelberg AC learning scheme that is easy to compute and fast to converge . Experiments reveal that ours outperform SOTAs in terms of average returns under acceptable training time . 1 INTRODUCTION . Reinforcement learning ( RL ) has been widely adopted for solving various complicated sequential problems such as games ( Mnih et al. , 2013 ) and robotics applications ( Kober et al. , 2013 ) . In particular , the actor-critic ( AC ) approach has achieved remarkable performance in many realworld tasks ( Arulkumaran et al. , 2017 ) . Most AC methods perform stochastic gradient descent on the actor and critic simultaneously . This can be regarded as performing Gradient Descent Ascent ( GDA ) ( Zheng et al. , 2021a ; b ; Wen et al. , 2021 ) , which is known to suffer from convergence to limit cycles or bad locally optimal solutions ( Wang et al. , 2019 ; Jin et al. , 2020 ; Balduzzi et al. , 2018 ) . Moreover , as mentioned in Yang et al . ( 2019 ) , the GDA training scheme is fragile , which could lead to biased critic and may not converge . To address these issues , Zheng et al . ( 2021a ) and Wen et al . ( 2021 ) reformulate the actor-critic problem as a bilevel optimization problem , and introduce a leader-follower ( Stackelberg ) training scheme ( Fiez et al. , 2020 ) . However , this requires the computation of several indirect gradient terms ( including the inverse Hessian matrices ) , which are difficult to compute or accurately approximated . Moreover , empirically , this Stackelberg training scheme converges slower than GDA in the environments where GDA is not trapped into limited cycles or bad local optimums , since the indirect gradient terms pull back the parameters to stay within the best response areas to guarantee that the leader ’ s loss is always small ( Wang et al. , 2019 ) . To reduce the computational complexity , Shewchuk ( 2007 ) introduces the use of conjugate gradient ( CG ) descent , which simplifies the computation of the approximated inverse Hessian to the solving of a linear system . However , to have good performance , a sufficient number of inner iterations is still required to obtain a good approximation of the Hessian matrices . Empirically , its convergence is still not fast . Among these actor-critic methods , the deterministic policy gradient ( DPG ) family ( Silver et al. , 2014 ; Lillicrap et al. , 2016 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) has some advantages over other methods : if we assume the actor to be the follower , and the critic to be the leader , the gradient of the follower can be divided into several terms , some are closely related to the best response function which slows down the training process . Also , they include Hessian matrices which are hard to compute ( We name it as the TD-related term which is formally discussed in Sec . 4 ) . Based on this observation , we propose Fast Stackelberg Deep Deterministic Policy Gradient ( FSDDPG ) , which reduces these TD-related terms to improve the convergence rate and reduce training time . Moreover , together with the block diagonal approximation technique to approximate the remaining matrices , it is possible to further reduce the training complexity from O ( n3m3 ) to O ( n2m2 ) , where n is the number of layers andm is the number of neurons in each layer . Experiments reveal that with our approximated methods outperform SOTAs in terms of average returns under acceptable training time . 2 RELATED WORKS . Actor Critic . The AC methods are widely adopted in solving complicated sequential problems ( Sutton & Barto , 2018 ) . Among these methods , the DPG family are prevalent since they can be both off and on policy and able to be deployed in discrete time and continuous time environments , which are efficient and suit various environments ( Lillicrap et al. , 2016 ) . However , most of the ACs use simultaneously update rule , i.e. , the actor and critic update with the same learning rate , which are not always stable . Empirically , the AC may find better solutions when the critic is updated more frequently than the actor ( Wen et al. , 2021 ) , and we also show it in our experiments in Sec . 6 . Theoretically , most of the stable training methods of GDAs are based on the Two Time Scale Updates ( TTSU ) scheme ( Wu et al. , 2020 ; Dalal et al. , 2018 ) . Although the simultaneously update rule for AC can also be proven convergence ( Fu et al. , 2020 ) , it may not suit all the environments since AC ( GDA ) may trapped into limited cycles and bad local optimality ( Zhang et al. , 2020 ; Wang et al. , 2019 ) . Regarding TTSU , since it requires the learning rate of the follower to be slower than that of the leader , its convergence rate is slow consequently ( Kaledin et al. , 2020 ) . As the AC training schemes can be also represented as a bi-level optimization process , it is naturally to introduce Stackelberg training scheme to AC as the Stackelberg Actor Critic to stabilize training ( Zheng et al. , 2021a ; Wen et al. , 2021 ) . However , the gradient of the follower is time consuming to calculate , and the convergence rate is relatively slow . Thus , in many environments , Stackelberg training may perform worse than vanilla AC under acceptable training time ( Wen et al. , 2021 ) . Gradient-based bi-level optimization . The gradient-based bi-level optimization is used to minimize/maximize a cost function defined in terms of the optimal solution to another cost function ( Sinha et al. , 2017 ) . Currently , the idea of differential Stackelberg ( leader-follower ) game is introduced to convert these problems into the differential ( Stackelberg ) game problems ( Fiez et al. , 2020 ) . To solve the differential Stackelberg game , the Stackelberg gradient descent is introduced by leveraging the implicit function theorem ( Ji et al. , 2021 ; Fiez et al. , 2020 ; Grazzi et al. , 2020 ) , which has already be applied into different areas , e.g. , GAN ( Fiez et al. , 2020 ) , hyper-parameter optimization ( Lorraine et al. , 2020 ) , as well as meta learning ( Rajeswaran et al. , 2019 ) . Approximating ( inverse ) Hessian . Calculating the ( inverse ) Hessian is time-consuming , especially for the deep neural networks ( Martens & Grosse , 2015 ; Botev et al. , 2017 ) . To address this challenge , the conjugate gradient approximation ( Rajeswaran et al. , 2019 ; Mehra , 1969 ; Ghadimi & Wang , 2018 ) , block diagonal approximation ( Martens & Grosse , 2015 ; Botev et al. , 2017 ) , as well as the approximate Newton methods ( Martınez , 2000 ; Xu et al. , 2016 ) are proposed . Here , we choose the block diagonal approximation , since it is easy to be implemented and fast to compute while maintaining a good performance . Another method to approximate Hessian is the Gauss Newton matrix ( Schraudolph , 2002 ) , which is generally used to approximate Hessian for the Mean Squared loss ( MSE ) ( Martens , 2020 ) . Gauss Newton matrix outperforms Hessian matrix for its fast computing , and stability since it is always positive semi-definite ( Chen , 2011 ) . However , it still lacks the obvious justification why dropping the Hessian term helps convergence ( Martens , 2020 ) . 3 PROBLEM FORMULATION . A Markov decision process ( MDP ) can represented by the tuple 〈S , A , R , P ( S|S , A ) , γ〉 , where s ∈ S is the state , a ∈ A is the action , p ( s′|s , a ) is the transition probability , and r ( s , a ) ∈ [ Rmin , Rmax ] is the bounded reward function . ρβ ( s ) is the discounted state distribution of s conducted by a behavior policy β 1. π : S × A → ∆ ( A ) is the policy , where ∆ ( A ) is the simplex of action set A. Qπ is the state-action value function w.r.t . π : Qπ ( s , a ) : = Eπ [ ∑ k γ krk|s , a ] . Here , we parameterize Q and π with w ∈ Ω and θ ∈ Θ , respectively ( Qω and πθ ) , where Ω and Θ are the parameter spaces . Following ( Zheng et al. , 2021a ) , we formulate the actor-critic as a differential Stackelberg game ( Fiez et al. , 2020 ) : maxθ J ( θ , w∗ ( θ ) ) ( 1 ) s.t . w∗ ( θ ) = arg min w L ( θ , w ) , whereJ ( θ , w ) =Es∼ρ ( s ) , a∼π ( ·|s ) [ Qπθw ( s , a ) ] , L ( θ , w ) =Es∼ρ ( s ) , a∼π ( ·|s ) [ ( Qπθw ( s , a ) − ⊥ [ Qπθ ( s , a ) ] ) 2 ] , which is the temporal difference ( TD ) error , and ⊥ ( · ) is the stop-gradient operator to achieve semi-gradient method ( Sutton & Barto , 2018 ) . 1For compactness , we omit the superscript of ρβ ( s ) as ρ ( s ) with a little abuse of notations . Stackelberg learning ( or approximate implicit differentiation ) ( Fiez et al. , 2020 ; Ji et al. , 2021 ) is a framework for solving the bilevel optimization problems ( Ji et al. , 2021 ; Fiez et al. , 2020 ; Grazzi et al. , 2020 ) , and can be used on most AC variants ( e.g. , DDPG ( Lillicrap et al. , 2016 ) , TD3 ( Fujimoto et al. , 2018 ) and SAC ( Haarnoja et al. , 2018 ) ) . Using it on ( 1 ) , we obtain ( Fiez et al. , 2020 ) : θ ← θ + α∂J ( θ , w ∗ ( θ ) ) ∂θ , w ← w − αw ∂L ( θ , w ) ∂w , ( 2 ) where ∂J ( θ , w ∗ ( θ ) ) ∂θ = ∂J ( θ , w ) ∂θ − ( ∂2L ( θ , w ) ∂θ∂w ) ( ∂2L ( θ , w ) ∂w2 ) −1 ∂J ( θ , w ) ∂w , and α , αω are learning rates for the actor and critic , respectively.2 Here , ( ∂2L ( θ , w ) ∂θ∂w ) ( ∂2L ( θ , w ) ∂w2 ) −1 ∂J ( θ , w ) ∂w ( 3 ) is known as the indirect gradient term ( Lorraine et al. , 2020 ) . Empirically , AC with Stackelberg training does not improve the performance of ACs in many environments ( Wen et al. , 2021 ) . This is also confirmed by our experiments in Sec . 6 . This is mainly due to the fact that Stackelberg training converges slowly , and thus it performs worse than vanilla AC when AC does not be trapped into limited cycles and some bad local optimality . Also , inverting the n × n Hessian matrix in ( 3 ) takes O ( n3 ) time , where n is the matrix size and is equal to the number of neural network parameters . Although Conjugate Gradient ( CG ) can be used to reduce the computing complexity ( Zheng et al. , 2021a ) , it still needs sufficient steps for CG to obtain a precise approximation of the Hessian matrix . In this paper , we focus on improving Stackelberg learning in DDPG ( Lillicrap et al. , 2016 ) and TD3 ( Fujimoto et al. , 2018 ) . DDPG and TD3 have been widely adopted for solving complicated continuous control environments ( Wang et al. , 2020 ) . In DPG family , π ( a|s ) degenerates to π ( a|s ) = δ ( a − πθ ( s ) ) , where πθ ( s ) : S → A is the deterministic policy , and δ ( · ) is the Dirac-Delta function . L ( θ , w ) becomes Eρ ( s ) [ ( Qπθw ( s , a ) − ⊥ [ Qπθ ( s , πθ ( s ) ) ] ) 2 ] , where Qπθw ( s , a ) : = Qw ( s , a ) , s.t . a = πθ ( s ) . J ( θ , w ) becomes Es∼ρ ( s ) [ Qπθw ( s , a ) ] . Notice that the practical DPG based methods update the critic by using L̂ ( θ , w ) = Eρ ( s ) , a∼β [ ( Qπθw ( s , a ) − ⊥ [ Qπθ ( s , a ) ] ) 2 ] 3 ( Lillicrap et al. , 2016 ; Fujimoto et al. , 2018 ) . We follow the critic ’ s loss , and the modified update scheme becomes : θ ← θ + α∂J ( θ , w ∗ ( θ ) ) ∂θ , w ← w − αw ∂L̂ ( θ , w ) ∂w , ( 4 ) Comparing with Eq . ( 2 ) , the update scheme for policy θ maintains the same . The main reason is that ∇θL̂ ( θ , w ) = Eρ ( s ) , a∼β [ ∇θ ( Qπθw ( s , a ) −Qπθ ( s , a ) ) 2 ] is not well-defined when a 6= π ( s ) , and thus we still use L ( θ , w ) to update the policy . To help our analysis , we denote algorithms that replace the vanilla policy gradient in Eq . ( 4 ) to DPG in DDPG ( resp . TD3 ) as SDDPG and ( resp . STD3 ) , respectively .

Summary: This paper proposes a method to accelerate Stackelberg Actor-Critic by ignoring the terms that contain Hessian in the indirect gradient term and applying block-diagonal approximation technique to remaining inverse terms. They prove a faster convergence rate of the proposed method to a fixed point. Experiments are conducted to validate the fast convergence and stability.