Sample-efficient actor-critic algorithms with an etiquette for zero-sum Markov games

We introduce algorithms based on natural actor-critic and analyze their sample complexity for solving two player zero-sum Markov games in the tabular case . Our results improve the best-known sample complexities of policy gradient/actorcritic methods for convergence to Nash equilibrium in the multi-agent setting . We use the error propagation scheme in approximate dynamic programming , recent advances for global convergence of policy gradient methods , temporal difference learning , and techniques from stochastic primal-dual optimization . Our algorithms feature two stages , requiring agents to agree on an etiquette before starting their interactions , which is feasible for instance in self-play . However , the agents only access to joint reward and joint next state and not to each other ’ s actions or policies . Our complexity results match the best-known results for global convergence of policy gradient algorithms for single agent RL . We provide numerical verification of our methods for a two player bandit environment and a two player game , Alesia . We observe improved empirical performance as compared to the recently proposed optimistic gradient descent-ascent variant for Markov games . 1 INTRODUCTION . We study two-player zero-sum Markov game framework which is a key model with broad applications in competitive reinforcement learning ( RL ) , robust RL , and many others Zhang et al . ( 2019a ; 2021 ) . This framework is introduced by Shapley ( 1953 ) as stochastic games and popularized in RL with Littman ( 1994 ) . In its basic form , two agents with competing interests interact in an environment where the reward and the state transition depend on the actions of both players . Even with this simplicity , such systems achieved impressive success in game-playing and robotics ( Kober et al. , 2013 ; Silver et al. , 2017 ; Mnih et al. , 2015 ; Vinyals et al. , 2019 ; Brown & Sandholm , 2019 ) . While value-based methods offer near-optimal guarantees ( Sidford et al. , 2020 ; Bai et al. , 2020 ; Bai & Jin , 2020 ; Xie et al. , 2020 ; Tian et al. , 2020 ) , policy gradient ( PG ) methods , including actor-critic ( AC ) and their natural counterparts natural PG ( NPG ) ( Kakade , 2001 ) and natural AC ( NAC ) ( Peters & Schaal , 2008 ) , only have limited guarantees , despite their model-free and easy-to-implement structure , flexibility and generality ( Schulman et al. , 2015 ; 2017 ; Wang et al. , 2016 ) . The PG methods ( Kakade , 2001 ; Sutton et al. , 2000 ) directly optimize the value function in the policy space— a non-convex optimization problem even in the basic single agent , tabular setting . Intriguingly , recent results demonstrate globally optimal convergence of PG methods by identifying a hidden convexity structure for single agent RL ( Agarwal et al. , 2020 ; Cen et al. , 2020 ; Mei et al. , 2020 ; Bhandari & Russo , 2019 ; 2021 ; Xu et al. , 2020b ; Lan , 2021 ; Khodadadian et al. , 2021b ; Hong et al. , 2020 ; Xu et al. , 2020a ; Khodadadian et al. , 2021b ) , and multi-agent RL ( MARL ) ( Daskalakis et al. , 2020 ; Wei et al. , 2021 ; Zhao et al. , 2021 ) . The existing results on PG methods for tabular two-player zero-sum Markov games mostly focus on decentralized algorithms with sample complexities Õ ( ϵ−12.5 ) ( Daskalakis et al. , 2020 ) , Õ ( ϵ−8 ) , and even Õ ( ϵ−4 ) , yet with some limitations ( Wei et al. , 2021 ) ; see Section 1.1 for the details . With function approximation , Zhao et al . ( 2021 ) obtains Õ ( ϵ−6 ) sample complexity when given access to unbiased sampling oracles of the value functions . On the other hand , the best-known sample complexity for global optimality for single agent problem is Õ ( ϵ−2 ) in the tabular case ( Lan , 2021 ) . As this complexity is achieved by value-based/modelbased methods in the multi-agent setting ( Sidford et al. , 2020 ; Zhang et al. , 2020 ) , one expects a similar complexity for policy-based methods . Our work precisely bridges this gap and develops policy gradient methods whose performance for MARL is closer to their single agent counterparts . Contributions . We propose algorithms based on natural actor-critic ( NAC ) framework for solving two-player zero-sum Markov games in the tabular case . Our sample complexity results match the best-known ones for global optimality in the single agent setting ( Lan , 2021 ; Khodadadian et al. , 2021b ; Hong et al. , 2020 ; Xu et al. , 2020b ) . In particular , we show that by using inner loops for policy evaluation and a carefully designed algorithm , the sample complexity to get an ϵ-approximate Nash equilibrium , is Õ ( ϵ−2 ) , by assuming a uniform lower bound on the policies . Without this assumption , we show Õ ( ϵ−4 ) complexity.1 Surprisingly , we achieve these results—to our knowledge , for the first time with policy gradient methods—mostly by a careful adaptation of the recent results for policy gradient methods in single agent setting , temporal difference learning , two-stage error propagation framework of policy iteration ( Perolat et al. , 2015 ) , and by employing techniques from stochastic primal-dual optimization . These developments require a careful algorithm design and analysis . In particular , two-stage nature of the algorithm incurs biases between the stages that we have to control carefully . Obtaining Õ ( ϵ−2 ) complexity requires a tighter analysis for both stages of the algorithm , with strict control on the aforementioned bias . Therefore , it requires more advanced techniques and algorithms , inspired from the stochastic primal-dual optimization literature . We explicitly highlight our important new techniques as insights in the sequel . The full proofs are included in the appendices . 1.1 RELATED WORKS . Policy gradient methods . There is growing interest in global convergence of PG methods in the single agent setting . Several works showed convergence rates of natural policy gradient ( NPG ) in the tabular setting by assuming access to exact value function oracle ( Agarwal et al. , 2020 ; Cen et al. , 2020 ; Mei et al. , 2020 ; Bhandari & Russo , 2019 ; 2021 ) or when value functions are estimated from the data ( Shani et al. , 2020 ; Xu et al. , 2020b ; Lan , 2021 ; Khodadadian et al. , 2021b ; Hong et al. , 2020 ; Xu et al. , 2020a ; Khodadadian et al. , 2021b ) . To our knowledge , the best sample complexity for NPG methods with inner loop for policy evaluation ( which we refer to as NAC ) is Õ ( ϵ−2 ) and is due to ( Lan , 2021 ) . For single loop NAC with online policy evaluation , the best sample complexity is Õ ( ϵ−4 ) as obtained in ( Khodadadian et al. , 2021b ; Hong et al. , 2020 ; Xu et al. , 2020b ) ( see ( Khodadadian et al. , 2021a , Table 1 ) ) . For a general overview of results in MARL we refer to Zhang et al . ( 2019a ) . Policy gradient methods for two-player zero-sum Markov games . With the positive results on global convergence of PG methods , translating these results to the competitive MARL has been the goal of many recent works . In particular , independent policy gradient methods with the agents interacting symmetrically has been considered in Daskalakis et al . ( 2020 ) ; Wei et al . ( 2021 ) . The work of Daskalakis et al . ( 2020 ) built on Agarwal et al . ( 2020 ) by using REINFORCE estimator ( Williams , 1992 ) and obtained sample complexity of Õ ( ϵ−12.5 ) for reaching to one-sided Nash equilibrium . The algorithm of Wei et al . ( 2021 ) built on optimistic gradient descent-ascent ( OGDA ) method combined with a running estimate of the value function , obtaining Õ ( ϵ−8 ) sample complexity for finding a policy pair with small duality gap . In addition , Wei et al . ( 2021 ) showed improved complexity Õ ( ϵ−4 ) when restricted to Euclidean projections onto the simplex with metric subregularity assumption . There are two subtleties about this result : First , as pointed out in Daskalakis et al . ( 2020 ) , metric subregularity constant can be arbitrarily small , resulting in degradation of the rate . Second , as also pointed out by Wei et al . ( 2021 ) , this result is limited to Euclidean setting and can not be extended to the NPG with softmax policy update , which requires projection with KL divergence . The algorithm can be seen similar to the gradient ascent algorithm in Agarwal et al . ( 2020 ) . As 1In Appendix E , we design an algorithm based on single loop NAC with the complexity of Õ ( ϵ−4 ) ( and O ( ϵ−7 ) without assuming lower bounded policies ) . Our results on this algorithm is , to our knowledge , the first finite-sample analysis of single loop NAC-based methods for two-player zero-sum Markov games . x , y is the time that it takes to go from state s to state s′ by using policy pair x , y ( Wei et al. , 2021 , Assumption 1 ) . ‡This Õ ( ϵ−4 ) complexity by Wei et al . ( 2021 ) requires using Euclidean projections onto the simplex instead of softmax updates and depends on the metric subregularity constant . Hence it is not applicable to NPG . shown in Agarwal et al . ( 2020 ) for single agent problems , NPG methods have much better convergence properties than Euclidean projected gradient ascent methods . For comparison with the works in Daskalakis et al . ( 2020 ) ; Wei et al . ( 2021 ) , we also refer to Remark 2.1 and Table 1 . Another very related work to ours is by Zhao et al . ( 2021 ) which considered ( i ) tabular setting with exact value functions and ( ii ) online setting with function approximation , also using the error propagation scheme of Perolat et al . ( 2015 ) . Building on Agarwal et al . ( 2020 ) , this work showed Õ ( ϵ−6 ) sample complexity with function approximation , with access to unbiased samples of the value functions . In contrast , we focus on the tabular setting and we do not assume access to unbiased value function oracles . Indeed , lack of unbiased samples for value functions required us to use new insights described in the sequel , to derive the tighter complexities Õ ( ϵ−2 ) and Õ ( ϵ−4 ) . Policy gradient methods for linear quadratic regulator ( LQR ) . For zero-sum LQR , Zhang et al . ( 2019b ) ; Bu et al . ( 2019 ) showed global convergence of PG with exact value function oracles . These methods have a nested structure where one player computes best-response and the other does policy gradient updates . Recently , Zhang et al . ( 2021 ) built on Zhang et al . ( 2019b ) to derive sample complexities when value functions are estimated from data . 2 PRELIMINARIES . Notation . We consider the tabular setting with finite state and action spaces denoted by S , A , B and the discount factor γ < 1 . The policy of the min agent is x and the max agent is y , with action sets A , B , respectively . At state s , both agents take actions independent of each other : a ∼ x ( ·|s ) and b ∼ y ( ·|s ) . Based on the actions , the environment transitions to the next state s′ ∼ P ( ·|s , a , b ) and the agents receive reward |r ( s , a , b ) | ≤ 1 . Given a policy pair x , y , we denote the induced steadystate distribution as ρx , y . Let U denote the uniform distribution for states that we also take as the initial state distribution for simplicity . We denote the probability simplex as ∆ . Given a policy x , we sometimes use the notation xs for x ( ·|s ) in the proofs . We use e ( st ) ∈ R|S| to denote the vector such that e ( s ) = 1 if s = st and e ( s ) = 0 , if s ̸= st. We use the same notation for e ( st , at ) . The value function for state s is defined as V x , y ( s ) = Ex , y [ ∞∑ t=0 γtr ( st , at , bt ) |s0 = s ] , where Ex , y is over random variables st , at , bt for all t ≥ 0 as at ∼ x ( ·|st ) , bt ∼ y ( ·|st ) and st+1 ∼ P ( ·|st , at , bt ) . Similarly , the action value function is defined as Qx , y ( s , a , b ) = Ex , y [ ∑∞ t=0 γ tr ( st , at , bt ) |s0 = s , a0 = a , b0 = b ] . With these definitions , we can state the formal problem . For all s ∈ S , we aim to solve min x ( ·|s ) ∈∆ max y ( ·|s ) ∈∆ V x , y ( s ) . We denote the information needed in algorithms as oracles . We provide the background on NPG , NAC , TD ( 0 ) in Appendix A. Nash equilibrium . We assume the existence of a pair of policies x⋆ , y⋆ that are Nash equilibrium , namely , for all s , V x ⋆ , y ( s ) ≤ V ⋆ ( s ) : = V x⋆ , y⋆ ( s ) ≤ V x , y⋆ ( s ) . We are interested in finding a onesided Nash equilibrium , similar to Daskalakis et al . ( 2020 ) ; Zhao et al . ( 2021 ) ; Zhang et al . ( 2019b ) ; Bu et al . ( 2019 ) . As mentioned in Daskalakis et al . ( 2020 ) , for the other player , one can re-run the algorithm by switching roles to have the guarantee for both players . In particular , for the initial state distribution U , we seek for xout such that Es0∼U [ max y V xout , y ( s0 ) − V ⋆ ( s0 ) ] ≤ ϵ . It is easy to prove that this quantity on the LHS is 0 if and only if xout is a Nash equilibrium . Interaction procedure . We use the interactions of the agents with the environment to estimate the value functions and related oracles for the running of the algorithm . At each interaction , agents have access to ( si , ai , r ( si , ai , bi ) , si+1 ) and ( si , bi , r ( si , ai , bi ) , si+1 ) , respectively . In terms of access of agents , our oracle model is similar to Daskalakis et al . ( 2020 ) ; Wei et al . ( 2021 ) . However , one difference is that we require a game etiquette : Our algorithms have two stages where the agents have to behave differently . As long as this etiquette is respected by the agents ( for example embedded to players in the beginning of the game ) , they do not need further communication . Softmax update rule/NAC . Given Kullback-Leibler divergence KL and action-value function Qxt , xt+1 ( ·|s ) = PKL ( xt ( ·|s ) , Qxt ( s , · ) ) : = arg min x ( ·|s ) ∈∆ ⟨Qxt ( s , · ) , x ( ·|s ) ⟩+KL ( x ( ·|s ) , xt ( ·|s ) ) , ( 1 ) is known as NPG with softmax parameterization ( Agarwal et al. , 2020 , Lemma 5.1 ) . When there is a critic estimating Qxt , along with actor updating xt with NPG , this algorithmic framework is called natural actor-critic , in short , NAC . We focus on KL divergence for simplicity and its wide use . Our developments also hold for more general Bregman divergences as Zhan et al . ( 2021 ) . Assumption 1 . There exists ρ such that , for any policy iterate pair xt , yt , for any state s , it holds that ρxt , yt ( s ) ≥ ρ > 0 , where ρxt , yt is the stationary state distribution induced by the policy pair . Assumption2 . There exist x , y such that , for any policy iterate pair xt , yt , for any state action tuple s , a , b , it holds that xt ( a|s ) ≥ x > 0 , yt ( b|s ) ≥ y > 0 . Our rationale on the assumptions . Assumption 1 and 2 essentially mean positive definiteness of the sampling matrices in policy evaluation ( see eqs . ( 30 ) , ( 34 ) and ( 42 ) ) . To our knowledge , some form of this assumption is required in most of the existing work on temporal difference ( TD ) ( including TD ( 0 ) ) methods for policy evaluation ( Bhandari et al. , 2018 ; Xu et al. , 2020b ; Khodadadian et al. , 2021a ; Lan , 2021 ; Hong et al. , 2020 ; Xu et al. , 2020a ; Wu et al. , 2020 ; Zou et al. , 2019 ) ( see App . A ) . The complexity Õ ( ϵ−2 ) requires Assumption 2 even for single agent problems ( see ( Lan , 2021 , Rem . 1 , Sec . 5.2 ) ) . Remark 2.1 . As summarized in Table 1 , similar assumptions to Assumption1 are used in Daskalakis et al . ( 2020 ) ; Wei et al . ( 2021 ) . In particular , each of these assumptions ensure that all action-state pairs are observed with nonzero probability throughout the game . Moreover , by additionally requiring Assumption 2 , we can obtain the complexity Õ ( ϵ−2 ) , matching the single-agent counterpart . Markovian bias . For simplicity , we assume that we sample from the steady state distribution of a given policy pair . During normal interaction with the environment , this is not the case and we obtain a single stream of data . Hence , TD ( 0 ) update is biased—commonly referred to as the Markovian bias . A large body of literature in the single agent literature showed that the effect of this bias in TD ( 0 ) update is essentially additive and can be handled by assuming uniform mixing of the induced Markov chain ( Wu et al. , 2020 ; Bhandari et al. , 2018 ; Zou et al. , 2019 ; Khodadadian et al. , 2021b ; Xu et al. , 2020b ; a ) . These analyses apply to our policy evaluation routines , extending them to the Markovian setting . For simplicity , we show our techniques with i.i.d . assumption and then illustrate how the extension with Markovian data follows with the uniform mixing assumption in Appendix D. Error propagation for approximate dynamic programming . Perolat et al . ( 2015 ) proposed error propagation analysis for approximate version of generalized policy iteration for zero-sum Markov games ( see Appendix C ) . The authors showed that the following two-stage algorithm converges : • Stage 1 : Given a fixed value function Vk−1 , find the policy pair which is an ϵ-equilibrium . min xs∈∆ max ys∈∆ ∑ a , b x ( a|s ) y ( a|s ) Qk−1 ( s , a , b ) = : xsQsk−1ys , ( 2 ) Algorithm 1 Reflected NAC with a game etiquette and ζ-greedy exploration Require : PKL defined in ( 1 ) in Sec . 2 . Exploration parameter ζ ≥ 0 ( with equality if Assumption 2 holds ) . Subroutines Policy-Eval-V , Policy-Eval-θ , Policy-Eval-ν ( see Alg . 2 , 3 , 4 and note βνn , β θ n , β ω n are potential step sizes for this routine ) . Initial policies x0 , y0 , ȳ0 . xk , yk denote outer , xt , yt denote inner loop ’ s iterate . 1 : for k = 0 , 1 , . . . do 2 : Stage 1 // Approximately solve a matrix game 3 : for t = 0 , 1 , . . . , T − 1 do 4 : [ V̂ xk−1 , V̂ y k−1 ] = [ Policy-Eval-V ( x k−1 , yk−1 , N , βωn ) , Policy-Eval-V ( x k−1 , yk−1 , N , βωn ) ] // Both players compute their own estimations of Vk−1 , denoted as V̂ x and V̂ y 5 : [ θ̂xt+1 , θ̂ y t+1 ] = [ Policy-Eval-θ ( xt , yt , N , V̂ x k−1 , β θ n ) , Policy-Eval-θ ( xt , yt , N , V̂ y k−1 , β θ n ) ] 6 : xt+1 ( ·|s ) = PKL ( xt ( ·|s ) , η ( 2θ̂xt+1 ( s , · ) − θ̂xt ( s , · ) ) ) 7 : yt+1 ( ·|s ) = PKL ( yt ( ·|s ) , −η ( 2θ̂yt+1 ( s , · ) − θ̂ y t ( s , · ) ) ) 8 : Output xk = 1T ∑T t=1 xt . 9 : Stage 2 // Approximately find best response 10 : for t = 0 , 1 , . . . , T − 1 do 11 : ν̂t+1 = Policy-Eval-ν ( xk , ȳt , N , β = βνn ) 12 : ȳt+1 ( ·|s ) = PKL ( ȳt ( ·|s ) , −ην̂t+1 ( s , · ) ) 13 : Output yk = ȳt̂ , where t̂ ∈ [ T ] is selected uniformly at random . Algorithm 2 VN = Policy-Eval-V ( x , y , N , β ) Require : Policy pair x , y , iteration counter N , step size β , initial value function estimate V0 . x̂ ( ·|s ) = ( 1− ζ ) x ( ·|s ) + ζ|A| , ŷ ( ·|s ) = ( 1− ζ ) y ( ·|s ) + ζ |B| . 1 : for n = 0 , 1 , . . . , N − 1 do 2 : Sample sn ∼ ρx̂ , ŷ ( · ) , an ∼ x̂ ( ·|sn ) , bn ∼ ŷ ( ·|sn ) , sn+1 ∼ P ( ·|sn , an , bn ) . 3 : Vn+1 = Vn − βne ( sn ) ( Vn ( sn ) − r ( sn , an , bn ) − γVn ( sn+1 ) ) where Qk−1 ( s , a , b ) = r ( s , a , b ) + γ ∑ s′ P ( s ′|s , a , b ) Vk−1 ( s′ ) . When it is clear from the context , we drop the subscript of Qk−1 . This is a matrix game and is the sample-complexity bottleneck ( Perolat et al. , 2015 ) . Let ϵg denote the accuracy and xk the output of this stage at iteration k : E [ Es∼U [ max ys∈∆ ( xk ) sQsk−1y s − min xs∈∆ max ys∈∆ xsQsk−1y s ] ] = ϵk1 , ( 3 ) where the outer expectation is over the randomness of the algorithm used to generate xk . • Stage 2 : This step finds an approximate best response . The fixed policy ( xk ) , can be viewed as a part of the environment . Denote yk as the approximate best-response computed in this stage , at iteration k. The resulting value function Vk = V x k , yk is fed to stage 1 in the next iteration . Let ϵe be the accuracy for this stage , yk the output of this stage : E [ Es∼U [ max y V x k , y ( s ) − V x k , yk ( s ) ] ] = ϵk2 , ( 4 ) where the outer expectation is over the randomness of the algorithm used to generate yk . Then , Perolat et al . ( 2015 , Theorem 1 ) , Zhao et al . ( 2021 ) show that the following holds ( see also Appendix C ) . E [ Es∼U [ max y∈∆ V x K , y ( s ) − V ⋆ ( s ) ] ] ≤ |S|K 1− γ Õ ( sup k≤K ϵk1 + sup k≤K ϵk2 ) +O ( |S|γK 1− γ ) . ( 5 )

Summary: This paper considers algorithms based on natural actor-critic for solving two player zero-sum Markov games in the tabular case. In particular,  the authors focus on the analysis of the sample complexity for a two-stage algorithm that solves a matrix game and a single agent problem, alternatively and iteratively. By combining and refining recent results for policy gradient methods, this paper manages to match the best-known results for global convergence of policy gradient algorithms for single agent RL.