Sample-efficient actor-critic algorithms with an etiquette for zero-sum Markov games

We introduce algorithms based on natural actor-critic and analyze their sample complexity for solving two player zero-sum Markov games in the tabular case . Our results improve the best-known sample complexities of policy gradient/actorcritic methods for convergence to Nash equilibrium in the multi-agent setting . We use the error propagation scheme in approximate dynamic programming , recent advances for global convergence of policy gradient methods , temporal difference learning , and techniques from stochastic primal-dual optimization . Our algorithms feature two stages , requiring agents to agree on an etiquette before starting their interactions , which is feasible for instance in self-play . However , the agents only access to joint reward and joint next state and not to each other ’ s actions or policies . Our complexity results match the best-known results for global convergence of policy gradient algorithms for single agent RL . We provide numerical verification of our methods for a two player bandit environment and a two player game , Alesia . We observe improved empirical performance as compared to the recently proposed optimistic gradient descent-ascent variant for Markov games . 1 INTRODUCTION . We study two-player zero-sum Markov game framework which is a key model with broad applications in competitive reinforcement learning ( RL ) , robust RL , and many others Zhang et al . ( 2019a ; 2021 ) . This framework is introduced by Shapley ( 1953 ) as stochastic games and popularized in RL with Littman ( 1994 ) . In its basic form , two agents with competing interests interact in an environment where the reward and the state transition depend on the actions of both players . Even with this simplicity , such systems achieved impressive success in game-playing and robotics ( Kober et al. , 2013 ; Silver et al. , 2017 ; Mnih et al. , 2015 ; Vinyals et al. , 2019 ; Brown & Sandholm , 2019 ) . While value-based methods offer near-optimal guarantees ( Sidford et al. , 2020 ; Bai et al. , 2020 ; Bai & Jin , 2020 ; Xie et al. , 2020 ; Tian et al. , 2020 ) , policy gradient ( PG ) methods , including actor-critic ( AC ) and their natural counterparts natural PG ( NPG ) ( Kakade , 2001 ) and natural AC ( NAC ) ( Peters & Schaal , 2008 ) , only have limited guarantees , despite their model-free and easy-to-implement structure , flexibility and generality ( Schulman et al. , 2015 ; 2017 ; Wang et al. , 2016 ) . The PG methods ( Kakade , 2001 ; Sutton et al. , 2000 ) directly optimize the value function in the policy space— a non-convex optimization problem even in the basic single agent , tabular setting . Intriguingly , recent results demonstrate globally optimal convergence of PG methods by identifying a hidden convexity structure for single agent RL ( Agarwal et al. , 2020 ; Cen et al. , 2020 ; Mei et al. , 2020 ; Bhandari & Russo , 2019 ; 2021 ; Xu et al. , 2020b ; Lan , 2021 ; Khodadadian et al. , 2021b ; Hong et al. , 2020 ; Xu et al. , 2020a ; Khodadadian et al. , 2021b ) , and multi-agent RL ( MARL ) ( Daskalakis et al. , 2020 ; Wei et al. , 2021 ; Zhao et al. , 2021 ) . The existing results on PG methods for tabular two-player zero-sum Markov games mostly focus on decentralized algorithms with sample complexities Õ ( ϵ−12.5 ) ( Daskalakis et al. , 2020 ) , Õ ( ϵ−8 ) , and even Õ ( ϵ−4 ) , yet with some limitations ( Wei et al. , 2021 ) ; see Section 1.1 for the details . With function approximation , Zhao et al . ( 2021 ) obtains Õ ( ϵ−6 ) sample complexity when given access to unbiased sampling oracles of the value functions . On the other hand , the best-known sample complexity for global optimality for single agent problem is Õ ( ϵ−2 ) in the tabular case ( Lan , 2021 ) . As this complexity is achieved by value-based/modelbased methods in the multi-agent setting ( Sidford et al. , 2020 ; Zhang et al. , 2020 ) , one expects a similar complexity for policy-based methods . Our work precisely bridges this gap and develops policy gradient methods whose performance for MARL is closer to their single agent counterparts . Contributions . We propose algorithms based on natural actor-critic ( NAC ) framework for solving two-player zero-sum Markov games in the tabular case . Our sample complexity results match the best-known ones for global optimality in the single agent setting ( Lan , 2021 ; Khodadadian et al. , 2021b ; Hong et al. , 2020 ; Xu et al. , 2020b ) . In particular , we show that by using inner loops for policy evaluation and a carefully designed algorithm , the sample complexity to get an ϵ-approximate Nash equilibrium , is Õ ( ϵ−2 ) , by assuming a uniform lower bound on the policies . Without this assumption , we show Õ ( ϵ−4 ) complexity.1 Surprisingly , we achieve these results—to our knowledge , for the first time with policy gradient methods—mostly by a careful adaptation of the recent results for policy gradient methods in single agent setting , temporal difference learning , two-stage error propagation framework of policy iteration ( Perolat et al. , 2015 ) , and by employing techniques from stochastic primal-dual optimization . These developments require a careful algorithm design and analysis . In particular , two-stage nature of the algorithm incurs biases between the stages that we have to control carefully . Obtaining Õ ( ϵ−2 ) complexity requires a tighter analysis for both stages of the algorithm , with strict control on the aforementioned bias . Therefore , it requires more advanced techniques and algorithms , inspired from the stochastic primal-dual optimization literature . We explicitly highlight our important new techniques as insights in the sequel . The full proofs are included in the appendices . 1.1 RELATED WORKS . Policy gradient methods . There is growing interest in global convergence of PG methods in the single agent setting . Several works showed convergence rates of natural policy gradient ( NPG ) in the tabular setting by assuming access to exact value function oracle ( Agarwal et al. , 2020 ; Cen et al. , 2020 ; Mei et al. , 2020 ; Bhandari & Russo , 2019 ; 2021 ) or when value functions are estimated from the data ( Shani et al. , 2020 ; Xu et al. , 2020b ; Lan , 2021 ; Khodadadian et al. , 2021b ; Hong et al. , 2020 ; Xu et al. , 2020a ; Khodadadian et al. , 2021b ) . To our knowledge , the best sample complexity for NPG methods with inner loop for policy evaluation ( which we refer to as NAC ) is Õ ( ϵ−2 ) and is due to ( Lan , 2021 ) . For single loop NAC with online policy evaluation , the best sample complexity is Õ ( ϵ−4 ) as obtained in ( Khodadadian et al. , 2021b ; Hong et al. , 2020 ; Xu et al. , 2020b ) ( see ( Khodadadian et al. , 2021a , Table 1 ) ) . For a general overview of results in MARL we refer to Zhang et al . ( 2019a ) . Policy gradient methods for two-player zero-sum Markov games . With the positive results on global convergence of PG methods , translating these results to the competitive MARL has been the goal of many recent works . In particular , independent policy gradient methods with the agents interacting symmetrically has been considered in Daskalakis et al . ( 2020 ) ; Wei et al . ( 2021 ) . The work of Daskalakis et al . ( 2020 ) built on Agarwal et al . ( 2020 ) by using REINFORCE estimator ( Williams , 1992 ) and obtained sample complexity of Õ ( ϵ−12.5 ) for reaching to one-sided Nash equilibrium . The algorithm of Wei et al . ( 2021 ) built on optimistic gradient descent-ascent ( OGDA ) method combined with a running estimate of the value function , obtaining Õ ( ϵ−8 ) sample complexity for finding a policy pair with small duality gap . In addition , Wei et al . ( 2021 ) showed improved complexity Õ ( ϵ−4 ) when restricted to Euclidean projections onto the simplex with metric subregularity assumption . There are two subtleties about this result : First , as pointed out in Daskalakis et al . ( 2020 ) , metric subregularity constant can be arbitrarily small , resulting in degradation of the rate . Second , as also pointed out by Wei et al . ( 2021 ) , this result is limited to Euclidean setting and can not be extended to the NPG with softmax policy update , which requires projection with KL divergence . The algorithm can be seen similar to the gradient ascent algorithm in Agarwal et al . ( 2020 ) . As 1In Appendix E , we design an algorithm based on single loop NAC with the complexity of Õ ( ϵ−4 ) ( and O ( ϵ−7 ) without assuming lower bounded policies ) . Our results on this algorithm is , to our knowledge , the first finite-sample analysis of single loop NAC-based methods for two-player zero-sum Markov games . x , y is the time that it takes to go from state s to state s′ by using policy pair x , y ( Wei et al. , 2021 , Assumption 1 ) . ‡This Õ ( ϵ−4 ) complexity by Wei et al . ( 2021 ) requires using Euclidean projections onto the simplex instead of softmax updates and depends on the metric subregularity constant . Hence it is not applicable to NPG . shown in Agarwal et al . ( 2020 ) for single agent problems , NPG methods have much better convergence properties than Euclidean projected gradient ascent methods . For comparison with the works in Daskalakis et al . ( 2020 ) ; Wei et al . ( 2021 ) , we also refer to Remark 2.1 and Table 1 . Another very related work to ours is by Zhao et al . ( 2021 ) which considered ( i ) tabular setting with exact value functions and ( ii ) online setting with function approximation , also using the error propagation scheme of Perolat et al . ( 2015 ) . Building on Agarwal et al . ( 2020 ) , this work showed Õ ( ϵ−6 ) sample complexity with function approximation , with access to unbiased samples of the value functions . In contrast , we focus on the tabular setting and we do not assume access to unbiased value function oracles . Indeed , lack of unbiased samples for value functions required us to use new insights described in the sequel , to derive the tighter complexities Õ ( ϵ−2 ) and Õ ( ϵ−4 ) . Policy gradient methods for linear quadratic regulator ( LQR ) . For zero-sum LQR , Zhang et al . ( 2019b ) ; Bu et al . ( 2019 ) showed global convergence of PG with exact value function oracles . These methods have a nested structure where one player computes best-response and the other does policy gradient updates . Recently , Zhang et al . ( 2021 ) built on Zhang et al . ( 2019b ) to derive sample complexities when value functions are estimated from data . 2 PRELIMINARIES . Notation . We consider the tabular setting with finite state and action spaces denoted by S , A , B and the discount factor γ < 1 . The policy of the min agent is x and the max agent is y , with action sets A , B , respectively . At state s , both agents take actions independent of each other : a ∼ x ( ·|s ) and b ∼ y ( ·|s ) . Based on the actions , the environment transitions to the next state s′ ∼ P ( ·|s , a , b ) and the agents receive reward |r ( s , a , b ) | ≤ 1 . Given a policy pair x , y , we denote the induced steadystate distribution as ρx , y . Let U denote the uniform distribution for states that we also take as the initial state distribution for simplicity . We denote the probability simplex as ∆ . Given a policy x , we sometimes use the notation xs for x ( ·|s ) in the proofs . We use e ( st ) ∈ R|S| to denote the vector such that e ( s ) = 1 if s = st and e ( s ) = 0 , if s ̸= st. We use the same notation for e ( st , at ) . The value function for state s is defined as V x , y ( s ) = Ex , y [ ∞∑ t=0 γtr ( st , at , bt ) |s0 = s ] , where Ex , y is over random variables st , at , bt for all t ≥ 0 as at ∼ x ( ·|st ) , bt ∼ y ( ·|st ) and st+1 ∼ P ( ·|st , at , bt ) . Similarly , the action value function is defined as Qx , y ( s , a , b ) = Ex , y [ ∑∞ t=0 γ tr ( st , at , bt ) |s0 = s , a0 = a , b0 = b ] . With these definitions , we can state the formal problem . For all s ∈ S , we aim to solve min x ( ·|s ) ∈∆ max y ( ·|s ) ∈∆ V x , y ( s ) . We denote the information needed in algorithms as oracles . We provide the background on NPG , NAC , TD ( 0 ) in Appendix A. Nash equilibrium . We assume the existence of a pair of policies x⋆ , y⋆ that are Nash equilibrium , namely , for all s , V x ⋆ , y ( s ) ≤ V ⋆ ( s ) : = V x⋆ , y⋆ ( s ) ≤ V x , y⋆ ( s ) . We are interested in finding a onesided Nash equilibrium , similar to Daskalakis et al . ( 2020 ) ; Zhao et al . ( 2021 ) ; Zhang et al . ( 2019b ) ; Bu et al . ( 2019 ) . As mentioned in Daskalakis et al . ( 2020 ) , for the other player , one can re-run the algorithm by switching roles to have the guarantee for both players . In particular , for the initial state distribution U , we seek for xout such that Es0∼U [ max y V xout , y ( s0 ) − V ⋆ ( s0 ) ] ≤ ϵ . It is easy to prove that this quantity on the LHS is 0 if and only if xout is a Nash equilibrium . Interaction procedure . We use the interactions of the agents with the environment to estimate the value functions and related oracles for the running of the algorithm . At each interaction , agents have access to ( si , ai , r ( si , ai , bi ) , si+1 ) and ( si , bi , r ( si , ai , bi ) , si+1 ) , respectively . In terms of access of agents , our oracle model is similar to Daskalakis et al . ( 2020 ) ; Wei et al . ( 2021 ) . However , one difference is that we require a game etiquette : Our algorithms have two stages where the agents have to behave differently . As long as this etiquette is respected by the agents ( for example embedded to players in the beginning of the game ) , they do not need further communication . Softmax update rule/NAC . Given Kullback-Leibler divergence KL and action-value function Qxt , xt+1 ( ·|s ) = PKL ( xt ( ·|s ) , Qxt ( s , · ) ) : = arg min x ( ·|s ) ∈∆ ⟨Qxt ( s , · ) , x ( ·|s ) ⟩+KL ( x ( ·|s ) , xt ( ·|s ) ) , ( 1 ) is known as NPG with softmax parameterization ( Agarwal et al. , 2020 , Lemma 5.1 ) . When there is a critic estimating Qxt , along with actor updating xt with NPG , this algorithmic framework is called natural actor-critic , in short , NAC . We focus on KL divergence for simplicity and its wide use . Our developments also hold for more general Bregman divergences as Zhan et al . ( 2021 ) . Assumption 1 . There exists ρ such that , for any policy iterate pair xt , yt , for any state s , it holds that ρxt , yt ( s ) ≥ ρ > 0 , where ρxt , yt is the stationary state distribution induced by the policy pair . Assumption2 . There exist x , y such that , for any policy iterate pair xt , yt , for any state action tuple s , a , b , it holds that xt ( a|s ) ≥ x > 0 , yt ( b|s ) ≥ y > 0 . Our rationale on the assumptions . Assumption 1 and 2 essentially mean positive definiteness of the sampling matrices in policy evaluation ( see eqs . ( 30 ) , ( 34 ) and ( 42 ) ) . To our knowledge , some form of this assumption is required in most of the existing work on temporal difference ( TD ) ( including TD ( 0 ) ) methods for policy evaluation ( Bhandari et al. , 2018 ; Xu et al. , 2020b ; Khodadadian et al. , 2021a ; Lan , 2021 ; Hong et al. , 2020 ; Xu et al. , 2020a ; Wu et al. , 2020 ; Zou et al. , 2019 ) ( see App . A ) . The complexity Õ ( ϵ−2 ) requires Assumption 2 even for single agent problems ( see ( Lan , 2021 , Rem . 1 , Sec . 5.2 ) ) . Remark 2.1 . As summarized in Table 1 , similar assumptions to Assumption1 are used in Daskalakis et al . ( 2020 ) ; Wei et al . ( 2021 ) . In particular , each of these assumptions ensure that all action-state pairs are observed with nonzero probability throughout the game . Moreover , by additionally requiring Assumption 2 , we can obtain the complexity Õ ( ϵ−2 ) , matching the single-agent counterpart . Markovian bias . For simplicity , we assume that we sample from the steady state distribution of a given policy pair . During normal interaction with the environment , this is not the case and we obtain a single stream of data . Hence , TD ( 0 ) update is biased—commonly referred to as the Markovian bias . A large body of literature in the single agent literature showed that the effect of this bias in TD ( 0 ) update is essentially additive and can be handled by assuming uniform mixing of the induced Markov chain ( Wu et al. , 2020 ; Bhandari et al. , 2018 ; Zou et al. , 2019 ; Khodadadian et al. , 2021b ; Xu et al. , 2020b ; a ) . These analyses apply to our policy evaluation routines , extending them to the Markovian setting . For simplicity , we show our techniques with i.i.d . assumption and then illustrate how the extension with Markovian data follows with the uniform mixing assumption in Appendix D. Error propagation for approximate dynamic programming . Perolat et al . ( 2015 ) proposed error propagation analysis for approximate version of generalized policy iteration for zero-sum Markov games ( see Appendix C ) . The authors showed that the following two-stage algorithm converges : • Stage 1 : Given a fixed value function Vk−1 , find the policy pair which is an ϵ-equilibrium . min xs∈∆ max ys∈∆ ∑ a , b x ( a|s ) y ( a|s ) Qk−1 ( s , a , b ) = : xsQsk−1ys , ( 2 ) Algorithm 1 Reflected NAC with a game etiquette and ζ-greedy exploration Require : PKL defined in ( 1 ) in Sec . 2 . Exploration parameter ζ ≥ 0 ( with equality if Assumption 2 holds ) . Subroutines Policy-Eval-V , Policy-Eval-θ , Policy-Eval-ν ( see Alg . 2 , 3 , 4 and note βνn , β θ n , β ω n are potential step sizes for this routine ) . Initial policies x0 , y0 , ȳ0 . xk , yk denote outer , xt , yt denote inner loop ’ s iterate . 1 : for k = 0 , 1 , . . . do 2 : Stage 1 // Approximately solve a matrix game 3 : for t = 0 , 1 , . . . , T − 1 do 4 : [ V̂ xk−1 , V̂ y k−1 ] = [ Policy-Eval-V ( x k−1 , yk−1 , N , βωn ) , Policy-Eval-V ( x k−1 , yk−1 , N , βωn ) ] // Both players compute their own estimations of Vk−1 , denoted as V̂ x and V̂ y 5 : [ θ̂xt+1 , θ̂ y t+1 ] = [ Policy-Eval-θ ( xt , yt , N , V̂ x k−1 , β θ n ) , Policy-Eval-θ ( xt , yt , N , V̂ y k−1 , β θ n ) ] 6 : xt+1 ( ·|s ) = PKL ( xt ( ·|s ) , η ( 2θ̂xt+1 ( s , · ) − θ̂xt ( s , · ) ) ) 7 : yt+1 ( ·|s ) = PKL ( yt ( ·|s ) , −η ( 2θ̂yt+1 ( s , · ) − θ̂ y t ( s , · ) ) ) 8 : Output xk = 1T ∑T t=1 xt . 9 : Stage 2 // Approximately find best response 10 : for t = 0 , 1 , . . . , T − 1 do 11 : ν̂t+1 = Policy-Eval-ν ( xk , ȳt , N , β = βνn ) 12 : ȳt+1 ( ·|s ) = PKL ( ȳt ( ·|s ) , −ην̂t+1 ( s , · ) ) 13 : Output yk = ȳt̂ , where t̂ ∈ [ T ] is selected uniformly at random . Algorithm 2 VN = Policy-Eval-V ( x , y , N , β ) Require : Policy pair x , y , iteration counter N , step size β , initial value function estimate V0 . x̂ ( ·|s ) = ( 1− ζ ) x ( ·|s ) + ζ|A| , ŷ ( ·|s ) = ( 1− ζ ) y ( ·|s ) + ζ |B| . 1 : for n = 0 , 1 , . . . , N − 1 do 2 : Sample sn ∼ ρx̂ , ŷ ( · ) , an ∼ x̂ ( ·|sn ) , bn ∼ ŷ ( ·|sn ) , sn+1 ∼ P ( ·|sn , an , bn ) . 3 : Vn+1 = Vn − βne ( sn ) ( Vn ( sn ) − r ( sn , an , bn ) − γVn ( sn+1 ) ) where Qk−1 ( s , a , b ) = r ( s , a , b ) + γ ∑ s′ P ( s ′|s , a , b ) Vk−1 ( s′ ) . When it is clear from the context , we drop the subscript of Qk−1 . This is a matrix game and is the sample-complexity bottleneck ( Perolat et al. , 2015 ) . Let ϵg denote the accuracy and xk the output of this stage at iteration k : E [ Es∼U [ max ys∈∆ ( xk ) sQsk−1y s − min xs∈∆ max ys∈∆ xsQsk−1y s ] ] = ϵk1 , ( 3 ) where the outer expectation is over the randomness of the algorithm used to generate xk . • Stage 2 : This step finds an approximate best response . The fixed policy ( xk ) , can be viewed as a part of the environment . Denote yk as the approximate best-response computed in this stage , at iteration k. The resulting value function Vk = V x k , yk is fed to stage 1 in the next iteration . Let ϵe be the accuracy for this stage , yk the output of this stage : E [ Es∼U [ max y V x k , y ( s ) − V x k , yk ( s ) ] ] = ϵk2 , ( 4 ) where the outer expectation is over the randomness of the algorithm used to generate yk . Then , Perolat et al . ( 2015 , Theorem 1 ) , Zhao et al . ( 2021 ) show that the following holds ( see also Appendix C ) . E [ Es∼U [ max y∈∆ V x K , y ( s ) − V ⋆ ( s ) ] ] ≤ |S|K 1− γ Õ ( sup k≤K ϵk1 + sup k≤K ϵk2 ) +O ( |S|γK 1− γ ) . ( 5 )

Summary: This paper studies the sample complexity of learning algorithms in two-player zero-sum tabular Markov games. The authors propose a two-stage algorithm that requires agents to agree on an etiquette, which means the agents have behave differently in two stages. Based on two assumptions on stationary state distribution and action exploration that are different from previous works, the authors show two finite-time convergence bounds, which improve much compared with existing results. The most challenging part of their proof is in bounding the approximation error in the first stage, which approximates the equilibrium of a matrix game. The authors also present some numerical results that can verify the performance of the proposed algorithm.