Fast Deterministic Stackelberg Actor-Critic

Most advanced Actor-Critic ( AC ) approaches update the actor and critic concurrently through ( stochastic ) Gradient Descents ( GD ) , which may be trapped into bad local optimality due to the instability of these simultaneous updating schemes . Stackelberg AC learning scheme alleviates these limitations by adding a compensated indirect gradient terms to the GD . However , the indirect gradient terms are time-consuming to calculate , and the convergence rate is also relatively slow . To alleviates these challenges , we find that in the Deterministic Policy Gradient family , by removing the terms that contain Hessian matrices and adopting the block diagonal approximation technique to approximate the remaining inverse matrices , we can construct an approximated Stackelberg AC learning scheme that is easy to compute and fast to converge . Experiments reveal that ours outperform SOTAs in terms of average returns under acceptable training time . 1 INTRODUCTION . Reinforcement learning ( RL ) has been widely adopted for solving various complicated sequential problems such as games ( Mnih et al. , 2013 ) and robotics applications ( Kober et al. , 2013 ) . In particular , the actor-critic ( AC ) approach has achieved remarkable performance in many realworld tasks ( Arulkumaran et al. , 2017 ) . Most AC methods perform stochastic gradient descent on the actor and critic simultaneously . This can be regarded as performing Gradient Descent Ascent ( GDA ) ( Zheng et al. , 2021a ; b ; Wen et al. , 2021 ) , which is known to suffer from convergence to limit cycles or bad locally optimal solutions ( Wang et al. , 2019 ; Jin et al. , 2020 ; Balduzzi et al. , 2018 ) . Moreover , as mentioned in Yang et al . ( 2019 ) , the GDA training scheme is fragile , which could lead to biased critic and may not converge . To address these issues , Zheng et al . ( 2021a ) and Wen et al . ( 2021 ) reformulate the actor-critic problem as a bilevel optimization problem , and introduce a leader-follower ( Stackelberg ) training scheme ( Fiez et al. , 2020 ) . However , this requires the computation of several indirect gradient terms ( including the inverse Hessian matrices ) , which are difficult to compute or accurately approximated . Moreover , empirically , this Stackelberg training scheme converges slower than GDA in the environments where GDA is not trapped into limited cycles or bad local optimums , since the indirect gradient terms pull back the parameters to stay within the best response areas to guarantee that the leader ’ s loss is always small ( Wang et al. , 2019 ) . To reduce the computational complexity , Shewchuk ( 2007 ) introduces the use of conjugate gradient ( CG ) descent , which simplifies the computation of the approximated inverse Hessian to the solving of a linear system . However , to have good performance , a sufficient number of inner iterations is still required to obtain a good approximation of the Hessian matrices . Empirically , its convergence is still not fast . Among these actor-critic methods , the deterministic policy gradient ( DPG ) family ( Silver et al. , 2014 ; Lillicrap et al. , 2016 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) has some advantages over other methods : if we assume the actor to be the follower , and the critic to be the leader , the gradient of the follower can be divided into several terms , some are closely related to the best response function which slows down the training process . Also , they include Hessian matrices which are hard to compute ( We name it as the TD-related term which is formally discussed in Sec . 4 ) . Based on this observation , we propose Fast Stackelberg Deep Deterministic Policy Gradient ( FSDDPG ) , which reduces these TD-related terms to improve the convergence rate and reduce training time . Moreover , together with the block diagonal approximation technique to approximate the remaining matrices , it is possible to further reduce the training complexity from O ( n3m3 ) to O ( n2m2 ) , where n is the number of layers andm is the number of neurons in each layer . Experiments reveal that with our approximated methods outperform SOTAs in terms of average returns under acceptable training time . 2 RELATED WORKS . Actor Critic . The AC methods are widely adopted in solving complicated sequential problems ( Sutton & Barto , 2018 ) . Among these methods , the DPG family are prevalent since they can be both off and on policy and able to be deployed in discrete time and continuous time environments , which are efficient and suit various environments ( Lillicrap et al. , 2016 ) . However , most of the ACs use simultaneously update rule , i.e. , the actor and critic update with the same learning rate , which are not always stable . Empirically , the AC may find better solutions when the critic is updated more frequently than the actor ( Wen et al. , 2021 ) , and we also show it in our experiments in Sec . 6 . Theoretically , most of the stable training methods of GDAs are based on the Two Time Scale Updates ( TTSU ) scheme ( Wu et al. , 2020 ; Dalal et al. , 2018 ) . Although the simultaneously update rule for AC can also be proven convergence ( Fu et al. , 2020 ) , it may not suit all the environments since AC ( GDA ) may trapped into limited cycles and bad local optimality ( Zhang et al. , 2020 ; Wang et al. , 2019 ) . Regarding TTSU , since it requires the learning rate of the follower to be slower than that of the leader , its convergence rate is slow consequently ( Kaledin et al. , 2020 ) . As the AC training schemes can be also represented as a bi-level optimization process , it is naturally to introduce Stackelberg training scheme to AC as the Stackelberg Actor Critic to stabilize training ( Zheng et al. , 2021a ; Wen et al. , 2021 ) . However , the gradient of the follower is time consuming to calculate , and the convergence rate is relatively slow . Thus , in many environments , Stackelberg training may perform worse than vanilla AC under acceptable training time ( Wen et al. , 2021 ) . Gradient-based bi-level optimization . The gradient-based bi-level optimization is used to minimize/maximize a cost function defined in terms of the optimal solution to another cost function ( Sinha et al. , 2017 ) . Currently , the idea of differential Stackelberg ( leader-follower ) game is introduced to convert these problems into the differential ( Stackelberg ) game problems ( Fiez et al. , 2020 ) . To solve the differential Stackelberg game , the Stackelberg gradient descent is introduced by leveraging the implicit function theorem ( Ji et al. , 2021 ; Fiez et al. , 2020 ; Grazzi et al. , 2020 ) , which has already be applied into different areas , e.g. , GAN ( Fiez et al. , 2020 ) , hyper-parameter optimization ( Lorraine et al. , 2020 ) , as well as meta learning ( Rajeswaran et al. , 2019 ) . Approximating ( inverse ) Hessian . Calculating the ( inverse ) Hessian is time-consuming , especially for the deep neural networks ( Martens & Grosse , 2015 ; Botev et al. , 2017 ) . To address this challenge , the conjugate gradient approximation ( Rajeswaran et al. , 2019 ; Mehra , 1969 ; Ghadimi & Wang , 2018 ) , block diagonal approximation ( Martens & Grosse , 2015 ; Botev et al. , 2017 ) , as well as the approximate Newton methods ( Martınez , 2000 ; Xu et al. , 2016 ) are proposed . Here , we choose the block diagonal approximation , since it is easy to be implemented and fast to compute while maintaining a good performance . Another method to approximate Hessian is the Gauss Newton matrix ( Schraudolph , 2002 ) , which is generally used to approximate Hessian for the Mean Squared loss ( MSE ) ( Martens , 2020 ) . Gauss Newton matrix outperforms Hessian matrix for its fast computing , and stability since it is always positive semi-definite ( Chen , 2011 ) . However , it still lacks the obvious justification why dropping the Hessian term helps convergence ( Martens , 2020 ) . 3 PROBLEM FORMULATION . A Markov decision process ( MDP ) can represented by the tuple 〈S , A , R , P ( S|S , A ) , γ〉 , where s ∈ S is the state , a ∈ A is the action , p ( s′|s , a ) is the transition probability , and r ( s , a ) ∈ [ Rmin , Rmax ] is the bounded reward function . ρβ ( s ) is the discounted state distribution of s conducted by a behavior policy β 1. π : S × A → ∆ ( A ) is the policy , where ∆ ( A ) is the simplex of action set A. Qπ is the state-action value function w.r.t . π : Qπ ( s , a ) : = Eπ [ ∑ k γ krk|s , a ] . Here , we parameterize Q and π with w ∈ Ω and θ ∈ Θ , respectively ( Qω and πθ ) , where Ω and Θ are the parameter spaces . Following ( Zheng et al. , 2021a ) , we formulate the actor-critic as a differential Stackelberg game ( Fiez et al. , 2020 ) : maxθ J ( θ , w∗ ( θ ) ) ( 1 ) s.t . w∗ ( θ ) = arg min w L ( θ , w ) , whereJ ( θ , w ) =Es∼ρ ( s ) , a∼π ( ·|s ) [ Qπθw ( s , a ) ] , L ( θ , w ) =Es∼ρ ( s ) , a∼π ( ·|s ) [ ( Qπθw ( s , a ) − ⊥ [ Qπθ ( s , a ) ] ) 2 ] , which is the temporal difference ( TD ) error , and ⊥ ( · ) is the stop-gradient operator to achieve semi-gradient method ( Sutton & Barto , 2018 ) . 1For compactness , we omit the superscript of ρβ ( s ) as ρ ( s ) with a little abuse of notations . Stackelberg learning ( or approximate implicit differentiation ) ( Fiez et al. , 2020 ; Ji et al. , 2021 ) is a framework for solving the bilevel optimization problems ( Ji et al. , 2021 ; Fiez et al. , 2020 ; Grazzi et al. , 2020 ) , and can be used on most AC variants ( e.g. , DDPG ( Lillicrap et al. , 2016 ) , TD3 ( Fujimoto et al. , 2018 ) and SAC ( Haarnoja et al. , 2018 ) ) . Using it on ( 1 ) , we obtain ( Fiez et al. , 2020 ) : θ ← θ + α∂J ( θ , w ∗ ( θ ) ) ∂θ , w ← w − αw ∂L ( θ , w ) ∂w , ( 2 ) where ∂J ( θ , w ∗ ( θ ) ) ∂θ = ∂J ( θ , w ) ∂θ − ( ∂2L ( θ , w ) ∂θ∂w ) ( ∂2L ( θ , w ) ∂w2 ) −1 ∂J ( θ , w ) ∂w , and α , αω are learning rates for the actor and critic , respectively.2 Here , ( ∂2L ( θ , w ) ∂θ∂w ) ( ∂2L ( θ , w ) ∂w2 ) −1 ∂J ( θ , w ) ∂w ( 3 ) is known as the indirect gradient term ( Lorraine et al. , 2020 ) . Empirically , AC with Stackelberg training does not improve the performance of ACs in many environments ( Wen et al. , 2021 ) . This is also confirmed by our experiments in Sec . 6 . This is mainly due to the fact that Stackelberg training converges slowly , and thus it performs worse than vanilla AC when AC does not be trapped into limited cycles and some bad local optimality . Also , inverting the n × n Hessian matrix in ( 3 ) takes O ( n3 ) time , where n is the matrix size and is equal to the number of neural network parameters . Although Conjugate Gradient ( CG ) can be used to reduce the computing complexity ( Zheng et al. , 2021a ) , it still needs sufficient steps for CG to obtain a precise approximation of the Hessian matrix . In this paper , we focus on improving Stackelberg learning in DDPG ( Lillicrap et al. , 2016 ) and TD3 ( Fujimoto et al. , 2018 ) . DDPG and TD3 have been widely adopted for solving complicated continuous control environments ( Wang et al. , 2020 ) . In DPG family , π ( a|s ) degenerates to π ( a|s ) = δ ( a − πθ ( s ) ) , where πθ ( s ) : S → A is the deterministic policy , and δ ( · ) is the Dirac-Delta function . L ( θ , w ) becomes Eρ ( s ) [ ( Qπθw ( s , a ) − ⊥ [ Qπθ ( s , πθ ( s ) ) ] ) 2 ] , where Qπθw ( s , a ) : = Qw ( s , a ) , s.t . a = πθ ( s ) . J ( θ , w ) becomes Es∼ρ ( s ) [ Qπθw ( s , a ) ] . Notice that the practical DPG based methods update the critic by using L̂ ( θ , w ) = Eρ ( s ) , a∼β [ ( Qπθw ( s , a ) − ⊥ [ Qπθ ( s , a ) ] ) 2 ] 3 ( Lillicrap et al. , 2016 ; Fujimoto et al. , 2018 ) . We follow the critic ’ s loss , and the modified update scheme becomes : θ ← θ + α∂J ( θ , w ∗ ( θ ) ) ∂θ , w ← w − αw ∂L̂ ( θ , w ) ∂w , ( 4 ) Comparing with Eq . ( 2 ) , the update scheme for policy θ maintains the same . The main reason is that ∇θL̂ ( θ , w ) = Eρ ( s ) , a∼β [ ∇θ ( Qπθw ( s , a ) −Qπθ ( s , a ) ) 2 ] is not well-defined when a 6= π ( s ) , and thus we still use L ( θ , w ) to update the policy . To help our analysis , we denote algorithms that replace the vanilla policy gradient in Eq . ( 4 ) to DPG in DDPG ( resp . TD3 ) as SDDPG and ( resp . STD3 ) , respectively .

Summary: This paper improves upon Stackelberg Deep Deterministic Policy Gradients by proposing a set of strategies on how to deal with the Hessian part of the gradient. This should overcome the major limitations of this class of methods, which are the great time complexity and the slow converging rate with respect to the standard actor-critic framework. The authors provide a formal justification on why removing parts of the Hessian and using a block-diagonal approximation is still achieving convergence. Time complexity is analyzed for many variations of the algorithm and the experimental session provides mixed results.