Decentralized Q-learning in Zero-sum Markov Games

1 Introduction . Reinforcement learning ( RL ) has achieved tremendous successes recently in a wide range of applications , including playing the game of Go [ Silver et al. , 2017 ] , playing video games ( e.g. , Atari [ Mnih et al. , 2015 ] and Starcraft [ Vinyals et al. , 2019 ] ) , robotics [ Lillicrap et al. , 2015 , Kober et al. , 2013 ] , and autonomous driving [ Shalev-Shwartz et al. , 2016 , Sallab et al. , 2017 ] . Most of these applications involve multiple decision-makers , where the agents ’ rewards and the evolution of the system are affected by the joint behaviors of all agents . This setting naturally leads to the problem of multi-agent RL ( MARL ) . In fact , MARL is arguably one key ingredient of large-scale and reliable autonomy , and is significantly more challenging to analyze than single-agent RL . There has been a surging interest recently in both a deeper theoretical and empirical understanding of MARL ; see comprehensive overviews on this topic in Busoniu et al . [ 2008 ] , Zhang et al . [ 2021 ] , Hernandez-Leal et al . [ 2019 ] . The pioneering work that initiated the sub-area of MARL , where the model of Markov/stochastic games [ Shapley , 1953 ] has been considered as a framework , is Littman [ 1994 ] . Since then , there ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . has been a plethora of works on MARL in Markov games ; see a detailed literature review in the supplementary material . These algorithms can be broadly categorized into two types : centralized/coordinated and decentralized/independent ones . For the former type , it is assumed that there exists a central controller for the agents , who can access the agents joint actions and local observations . With full awareness of the game setup , the central controller coordinates the agents to optimize their own policies , and aims to compute an equilibrium . This centralized paradigm is typically suitable for the scenarios when a simulator of the game is accessible [ Silver et al. , 2017 , Vinyals et al. , 2017 ] . Most existing MARL algorithms in Markov games have focused on this paradigm . Nevertheless , in many practical multi-agent learning scenarios , e.g. , multi-robot control [ Wang and de Silva , 2008 ] , urban traffic control [ Kuyer et al. , 2008 ] , as well as economics with rational decisionmakers [ Fudenberg and Levine , 1998 ] , agents make decentralized decisions without a coordinator . Specifically , agents make updates independently with only local observations of their own payoff and action histories , usually in a myopic fashion . Besides its ubiquity in practice , this decentralized paradigm also has the advantage of being scalable , as each agent only cares about her own policy and/or value functions , and the algorithms complexity do not suffer from the exponential dependence on the number of agents . Unfortunately , establishing provably convergent decentralized MARL algorithms is well-known to be challenging ; see the non-convergent cases ( even in the fully cooperative setting ) in Tan [ 1993 ] , Boutilier [ 1996 ] , Claus and Boutilier [ 1998 ] , and see Matignon et al . [ 2012 ] for more empirical evidences . The key challenge is the non-stationarity of the environment from each agent ’ s perspective since all agents are adapting their policies simultaneously and independently . In other words , the opponent is not playing according to a stationary strategy . This non-stationarity issue is in fact one of the core issues in ( decentralized ) MARL [ Busoniu et al. , 2008 , Hernandez-Leal et al. , 2017 ] . Studying when self-interested players can converge to an equilibrium through non-equilibrium adaptation is core question in the related literature of learning in games [ Fudenberg and Levine , 1998 , 2009 ] . For example , simple and stylized learning dynamics , such as fictitious play , are shown to converge to an equilibrium in certain but important classes of games , e.g. , zero-sum [ Robinson , 1951 , Harris , 1998 ] and common-interest [ Monderer and Shapley , 1996 ] , in repeated play of the same game . However , we can not generalize these results to decentralized MARL in Markov games ( also known as stochastic games , introduced by Shapley [ 1953 ] ) , because agents strategies affect not only the immediate reward , as in the repeated play of the same strategic-form game , but also the rewards that will be received in the future . Therefore , the configuration of the induced stage games are not necessarily stationary in Markov games . Contributions . In this paper , we present a provably convergent decentralized MARL learning dynamics2 for zero-sum discounted Markov games over an infinite horizon with minimal information available to agents . Particularly , each agent only has access to her immediate reward and the current state with perfect recall . They do not have access to the immediate reward the opponent receives . They do not know a model of their reward functions and the underlying state transitions probabilities . They are oblivious to the zero-sum structure of the underlying game . They also do not observe the opponent ’ s actions . Indeed , they may even be oblivious to the presence of other agents . Learning dynamics with such minimal information is also referred to as being radically uncoupled or value-based in the literature of learning in games [ Foster and Young , 2006 , Leslie and Collins , 2005 ] . To address the non-stationarity issue , we advocate a two-timescale adaptation of the individual Q-learning , introduced by Leslie and Collins [ 2005 ] and originating in Fudenberg and Levine [ 1998 ] . Particularly , each agent infers the opponent ’ s strategy indirectly through an estimate of the local Q-function ( a function of the opponent ’ s strategy ) and simultaneously forms an estimate of the value function to infer the continuation payoff . The slow update of the value function estimate is natural since agents tend to change their strategies faster than their estimates ( as observed in the evolutionary game theory literature , e.g. , Ely and Yilankaya [ 2001 ] , Sandholm [ 2001 ] ) , but this also helps weakening the dependence between the configuration of the stage games ( specifically the global Q-functions ) and the strategies . We show the almost sure convergence of the learning dynamics to the Nash equilibrium using the stochastic approximation theory , by developing a novel Lyapunov function and identifying the sufficient conditions precisely later in §3 . Our techniques 2To emphasize the difference from many existing MARL algorithms that focus on the computation of Nash equilibrium , we refer to our update rule as learning dynamics , following the literature of learning in games . toward addressing these challenges might be of independent interest . We also verify the convergence of the learning dynamics via numerical examples . To the best of our knowledge , our learning dynamics appears to be one of the first provably convergent decentralized MARL learning dynamics for Markov games that enjoy all the appealing properties below , addressing an important open question in the literature [ Pérolat et al. , 2018 , Daskalakis et al. , 2020 ] . In particular , our learning dynamics – • requires only minimal information available to the agents , i.e. , it is a radically uncoupled learning dynamic , unlike many other MARL algorithm , e.g. , Pérolat et al . [ 2015 ] , Sidford et al . [ 2019 ] , Leslie et al . [ 2020 ] , Sayin et al . [ 2020 ] , Bai and Jin [ 2020 ] , Xie et al . [ 2020 ] , Zhang et al . [ 2020 ] , Liu et al . [ 2020 ] , Shah et al . [ 2020 ] ; • requires no coordination or communication between agents during learning . For example , agents always play the ( smoothed ) best response consistent with their self-interested decisionmaking , contrary to being coordinated to keep playing the same strategy within certain time intervals as in Arslan and Yuksel [ 2017 ] and Wei et al . [ 2021 ] ; • requires no asymmetric update rules and/or stepsizes for the agents unlike existing literature [ Vrieze and Tijs , 1982 ] , [ Bowling and Veloso , 2002 ] , [ Leslie and Collins , 2003 ] , [ Daskalakis et al. , 2020 ] , [ Zhao et al. , 2021 , Guo et al. , 2021 ] . Such an asymmetry implies implicit coordination between agents to decide who follows which update rule or who chooses which stepsize ( and correspondingly who reacts fast or slow ) . Daskalakis et al . [ 2020 ] refers to each agent playing a symmetric role in learning as strongly independent learning . • is both rational and convergent , a desired property for MARL ( independent of whether it is centralized or decentralized ) , e.g. , see Bowling and Veloso [ 2001 ] , Busoniu et al . [ 2008 ] . A MARL algorithm is rational if each agent can converge to best-response , when the opponent plays an ( asymptotically ) stationary strategy ; and it can converge only to an equilibrium when all agents adopt it . A detailed literature review is deferred to the supplementary material due to space limitations . Of particular relevance are two recent works Tian et al . [ 2020 ] and Wei et al . [ 2021 ] studied decentralized setting similar to ours . Tian et al . [ 2020 ] focused on the exploration aspect for finite-horizon settings , and focused on minimizing a weak notion of regret without providing convergence guarantees under self-play.3 Wei et al . [ 2021 ] presented an optimistic variant of the gradient descent-ascent method that shares similar desired properties with our learning dynamics , with a strong guarantee of last-iterate convergence rates . However , the algorithm is delicately designed and different from the common value/policy-based RL update rules , e.g. , Q-learning , as in our work . Moreover , to characterize finite-time convergence , in the model-free setting , the agents need to coordinate to interact multiple steps at each iteration of the algorithm , while our learning dynamics is coordination-free with natural update rules . These two works can thus be viewed as orthogonal to ours . After submitting our paper , we became aware of a concurrent and independent work Guo et al . [ 2021 ] , which also developed a decentralized algorithm for zero-sum Markov games with function approximation and finite-sample guarantees . In contrast to our learning dynamics , the algorithm requires a double-loop update rule , and thus is asymmetric and requires coordination between agents . The assumptions and technical novelties in both works are also fundamentally different . See §A.2 for a detailed comparison . Organization . The rest of the paper is organized as follows . We describe Markov games and our decentralized Q-learning dynamics in §2 . In §3 , we present the assumptions and the convergence results . In §4 , we provide numerical examples . We conclude the paper with some remarks in §5 . The supplementary material includes a detailed literature review and the proofs of technical results . Notations . Superscript denotes player identity . We represent the entries of vectors q ( or matrices Q ) via q [ i ] ( or Q [ i , j ] ) . For two vectors x and y , the inner product is denoted by x · y = xT y . For a finite set A , we denote the probability simplex over A by ∆ ( A ) . 3Note that the same update rule with different stepsize and bonus choices and a certified policy technique , however , can return a non-Markovian approximate Nash equilibrium policy pair in the self-play setting ; see Bai et al . [ 2020 ] for more details . 2 Decentralized Q-learning in Zero-sum Markov Games This section presents a decentralized Q-learning dynamics that does not need access to the opponent ’ s actions and does not need to know the zero-sum structure of the underlying Markov game . To this end , we first start by providing a formal description of Markov games . Consider two players interacting with each other in a common dynamic environment , with totally conflicting objectives . The setting can be described by a two-player zero-sum Markov game , characterized by a tuple 〈S , { Ais } ( i , s ) ∈ { 1,2 } ×S , { ris } ( i , s ) ∈ { 1,2 } ×S , p , γ〉 , where S denotes the set of states , Ais denotes the action set of player i at state s ∈ S , and γ ∈ [ 0 , 1 ) denotes the discount factor . At each interaction round , player i receives a reward according to the function ris : A 1 s ×A2s → R. Since it is a zero-sum game , we have r1s ( a 1 , a2 ) + r2s ( a 1 , a2 ) = 0 for each joint action pair ( a1 , a2 ) ∈ A1s ×A2s . We denote the transition probability from state s to state s′ given a joint action profile ( a1s , a 2 s ) by p ( s′|s , a1s , a2s ) . Let us denote the stationary ( Markov ) strategy of player i by πi : = { πis ∈ ∆ ( Ais ) } s∈S . We define the expected utility of player i under the strategy profile ( π1 , π2 ) as the expected discounted sum of the reward he collects over an infinite horizon U i ( π1 , π2 ) = E { ajk∼πjsk } j∈ { 1,2 } { ∞∑ k=0 γkrisk ( a 1 k , a 2 k ) } , ( 1 ) where { s0 ∼ po , sk+1 ∼ p ( ·|sk , a1k , a2k ) , k ≥ 0 } is a stochastic process describing the evolution of the state over time and po ∈ ∆ ( S ) is the initial state distribution . The expectation is taken with respect to the initial state , randomness induced by state transitions and mixed strategies . A strategy profile ( π1∗ , π 2 ∗ ) is an ε-Nash equilibrium of the Markov game with ε ≥ 0 provided that U1 ( π1∗ , π 2 ∗ ) ≥ U1 ( π1 , π2∗ ) − ε , ∀π1 ( 2a ) U2 ( π1∗ , π 2 ∗ ) ≥ U2 ( π1∗ , π2 ) − ε , ∀π2 . ( 2b ) A Nash equilibrium is an ε-Nash equilibrium with ε = 0 . It is known that such a Nash equilibrium exists for discounted Markov games [ Fink , 1964 , Filar and Vrieze , 2012 ] . Given a strategy profile π : = ( π1 , π2 ) , we define the value function of player i by viπ ( s ) = E { ajk∼πjsk } j∈ { 1,2 } { ris ( a 1 , a2 ) + γ ∑ s′∈S viπ ( s ′ ) p ( s′|s , a1 , a2 ) } , ∀s . ( 3 ) Note that U i ( π1 , π2 ) = E { viπ ( s0 ) | s0 ∼ po } . We also define the Q-function that represents the value obtained for a given state and joint action pair as Qiπ ( s , a 1 , a2 ) = ris ( a 1 , a2 ) + γ ∑ s′∈S viπ ( s ′ ) p ( s′|s , a1 , a2 ) , ∀ ( s , a1 , a2 ) , ( 4 ) as well as the local Q-function for player i as qiπ ( s , a i ) : = Ea−i∼π−is { Qiπ ( s , a 1 , a2 ) } , ∀ ( s , ai ) , ( 5 ) where −i denotes the opponent of player i . By the one-stage deviation principle , we can interpret the interaction between the players at each stage as they are playing an auxiliary stage game , in which the payoff functions are equal to the Q-functions , e.g. , see Shapley [ 1953 ] . However , the Q-functions , and correspondingly the payoff functions in these auxiliary stage games , change with evolving strategies of the players . Therefore , the plethora of existing results for repeated play of the same strategic-form game ( e.g. , see the review Fudenberg and Levine [ 2009 ] ) do not generalize here . To address this challenge , we next introduce our decentralized Q-learning dynamics . 2.1 Decentralized Q-learning Dynamics In our decentralized Q-learning dynamics , minimal information is available to players . In other words , they only have access to the immediate reward received and current state visited with perfect recall . They do not observe the actions taken by the opponent . Correspondingly , they can not form a belief about the opponent ’ s strategy based on the empirical play as in fictitious play [ Fudenberg and Levine , 1998 ] or its variant for stochastic games [ Sayin et al. , 2020 ] . Instead , the players can look for inferring the opponent ’ s strategy , e.g. , by estimating the local Q-function since the local Q-function contains information about the opponent ’ s strategy , as illustrated in Figure 1 . As seen in Figure 1 , however , the local Q-function also depends on the global Q-function while the global Q-function is not necessarily stationary since it depends on the value function , and therefore , depends on the players ’ evolving strategies . An estimate of the global Q function which is slowly evolving would make it relatively stationary compared to the strategies . However , the players can not estimate the global Q-function directly since they do not have access to the opponent ’ s actions . Instead , they estimate the value function while updating it at a slower timescale . The slow update of the value function estimate makes the implicit globalQ-function relatively stationary compared to the strategies . Therefore , the players can use the local Q-function estimate to infer the opponent ’ s strategy . Note that the local Q-function estimate for different actions would get updated asynchronously via the classical Q-learning algorithm since they would be updated only when the associated action is taken , however , the actions are likely to be taken at different frequencies . Instead , here the players update the local Q-function estimate via a learning dynamics inspired from the individual Q-learning . The individual Q-learning , presented by Leslie and Collins [ 2005 ] and originating in Fudenberg and Levine [ 1998 ] , is based on the Q-learning with soft-max exploration while the step sizes are normalized with the probability of the actions taken . This normalization ensures that the estimates for every action get updated at the same learning rate in the expectation . We elaborate further on this after we introduce the precise update of the local Q-function estimate later in this section . Each player i keeps track of { q̂is , k } s∈S and { v̂is , k } s∈S estimating , respectively , the local Q-function and the value function . Player i updates { q̂is , k } s∈S at a faster timescale than { v̂is , k } s∈S . Players also count the number of times each state s is visited ( until the current stage ) , denoted by # s. We assume players know that the reward function takes values in [ −R , R ] for some R ∈ ( 0 , ∞ ) , i.e. , |ris ( a1 , a2 ) | ≤ R for all ( i , s , a1 , a2 ) . Therefore , player i knows that his local Q-function ‖qiπ ( s , · ) ‖∞ ≤ R/ ( 1− γ ) = : D and the value function |viπ ( s ) | ≤ D for any strategy profile π and s ∈ S. Correspondingly , the players initiate these estimates arbitrarily such that ‖q̂is,0‖∞ ≤ D and |v̂is,0| ≤ D , for all s. Let sk denote the current state at stage k. Player i takes his action aik ∈ Aisk , drawn from the smoothed best response Br i sk ( q̂isk , k , τ # sk ) ∈ ∆ ( A i s ) , which depends on a temperature parameter τ # s > 0 associated with # s. We define Br i s ( q , τ ) : = argmax µ∈∆ ( Ais ) { µ · q + τνis ( µ ) } , ( 6 ) Table 1 : Decentralized Q-learning dynamics in Markov games Require : Keep track of { q̂is , k , v̂is , k , # s } s∈S . 1 : Observe the current state sk . 2 : Update the entry of the local Q-function estimate for the previous state sk−1 and local action aik−1 according to q̂isk−1 , k [ a i k−1 ] = q̂ i sk−1 , k−1 [ a i k−1 ] + ᾱ i k−1 ( rik−1 + γv̂ i sk , k−1 − q̂ i sk−1 , k−1 [ a i k−1 ] ) , where ᾱik−1 = min { 1 , α # sk−1 πik−1 [ a i k−1 ] } , and q̂is , k [ a i ] = q̂is , k−1 [ a i ] for all ( s , ai ) 6= ( sk−1 , aik−1 ) . 3 : Increment the counter only for the current state by one . 4 : Take action aik ∼ πik where πik = argmax µ∈∆ ( Aisk ) { µ · q̂isk , k + τ # skν i sk ( µ ) } . 5 : Collect the local reward rik and update the value function estimate for the current state according to v̂isk , k+1 = v̂ i sk , k + β # sk [ πik · q̂isk , k − v̂ i sk , k ] . On the other hand , v̂is , k+1 = v̂ i s , k for all s 6= sk . where νis is a smooth and strictly concave function whose gradient is unbounded at the boundary of the simplex ∆ ( Ais ) [ Fudenberg and Levine , 1998 ] . The temperature parameter τ > 0 controls the amount of perturbation on the smoothed best response . The smooth perturbation ensures that there exists a unique maximizer πik : = Br i sk ( q̂isk , k , τ # sk ) . Choosing ν i s ( µ ) : = − ∑ ai∈Ais µ [ ai ] logµ [ ai ] results in the explicit characterization : πik [ a i ] = exp ( q̂isk , k [ a i ] /τ # sk ) ∑ ãi exp ( q̂isk , k [ ã i ] /τ # sk ) > 0 . ( 7 ) We let player i update his local Q-function estimate ’ s entry associated with the current state and local action pair ( sk , aik ) towards the reward received plus the discounted continuation payoff estimate . To this end , we include v̂isk+1 , k as an unbiased estimate of the continuation payoff obtained by looking one stage ahead , as in the classical Q-learning introduced by Watkins and Dayan [ 1992 ] . Due to the one-stage look ahead , the update for the current state and local action can take place just after the game visits the next state . The update of q̂isk , k is given by q̂isk , k+1 [ a i k ] = q̂ i sk , k [ aik ] + ᾱ i k ( rik + γv̂ i sk+1 , k − q̂isk , k [ a i k ] ) , ( 8 ) where ᾱik ∈ ( 0 , 1 ] is defined by ᾱik : = min { 1 , α # sk πik [ a i k ] } , with { αc } c > 0 a step size sequence , and rik ∈ [ −R , R ] denotes the immediate reward of player i at stage k. There is no update on others , i.e. , q̂is , k+1 [ a i ] = q̂is , k [ a i ] for all ( s , ai ) 6= ( sk , aik ) . Inspired by the approach in Leslie and Collins [ 2005 ] , the normalization addresses the asynchronous update of the entries of the local Q-function estimate and ensures that every entry of the local Q-function estimate is updated at the same rate in expectation . We will show this explicitly in the proof of main theorem in the supplementary material . Simultaneous to updating the local Q-function estimate , player i updates his value function estimate v̂isk , k towards π i k · q̂isk , k corresponding to the expected value of the current state . However , the player uses a different step size { βc } c > 0 and updates v̂isk , k according to v̂isk , k+1 = v̂ i sk , k + β # sk [ πik · q̂isk , k − v̂ i sk , k ] . ( 9 ) For other states s 6= sk , there is no update on the value function estimate , i.e. , v̂is , k+1 = v̂is , k . To sum up , player i follows the learning dynamics in Table 1 . We emphasize that this dynamic is radically uncoupled since each player ’ s update rule does not depend on the opponent ’ s payoffs or actions . In the next section , we study its convergence properties .

Summary: This paper presents a fully decentralized Q-learning algorithm for Markov games.  The individual ("local") Q-function is tracked separately from a state-value function, with updates to the Q-function taken according to a TD-error computed using the state-value function (of the next state).  Crucially, these two function are learned at differing timescales, making the action-reward value for each state "appear" stationary.  The algorithm is demonstrated to converge to Nash equilibrium (or the neighborhood of a Nash equilibrium, depending on the strength of assumptions made) in zero-sum Markov games when both players use it, both theoretically and numerically.