Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games

1 INTRODUCTION . In reinforcement learning ( RL ) ( Sutton and Barto , 2018 ) , an agent learns to make decisions that minimize its expected total cost through sequential interactions with the environment . Multi-agent reinforcement learning ( MARL ) ( Shoham et al. , 2003 ; 2007 ; Busoniu et al. , 2008 ) aims to extend RL to sequential decision-making problems involving multiple agents . In a non-cooperative game , we are interested in the Nash equilibrium ( Nash , 1951 ) , which is a joint policy of all the agents such that each agent can not decrease its expected total cost by unilaterally deviating from its Nash policy . The Nash equilibrium plays a critical role in understanding the social dynamics of self-interested agents ( Ash , 2000 ; Axtell , 2002 ) and constructing the optimal policy of a particular agent via fictitious selfplay ( Bowling and Veloso , 2000 ; Ganzfried and Sandholm , 2009 ) . With the recent development in deep learning ( LeCun et al. , 2015 ) , MARL with function approximation achieves tremendous empirical successes in applications , including Go ( Silver et al. , 2016 ; 2017 ) , Poker ( Heinrich and Silver , 2016 ; Moravčı́k et al. , 2017 ) , Star Craft ( Vinyals et al. , 2019 ) , Dota ( OpenAI , 2018 ) , autonomous driving ( Shalev-Shwartz et al. , 2016 ) , multi-robotic systems ( Yang and Gu , 2004 ) , and solving social dilemmas ( de Cote et al. , 2006 ; Leibo et al. , 2017 ; Hughes et al. , 2018 ) . However , since the capacity of the joint state and action spaces grows exponentially in the number of agents , such MARL approaches become computationally intractable when the number of agents is large , which is common in real-world applications ( Sandholm , 2010 ; Calderone , 2017 ; Wang et al. , 2017a ) . Mean-field game is proposed by Huang et al . ( 2003 ; 2006 ) ; Lasry and Lions ( 2006a ; b ; 2007 ) with the idea of utilizing mean-field approximation to model the strategic interactions within a large population . In a mean-field game , each agent has the same cost function and state transition , which depend on the other agents only through their aggregated effect . As a result , the optimal policy of each agent depends solely on its own state and the aggregated effect of the population , and such an optimal policy is symmetric across all the agents . Moreover , if the aggregated effect of the population corresponds to the Nash equilibrium , then the optimal policy of each agent jointly constitutes a Nash equilibrium . Although such a Nash equilibrium corresponds to an infinite number of agents , it well approximates the Nash equilibrium for a sufficiently large number of agents ( Bensoussan et al. , 2016 ) . Also , as the aggregated effect of the population abstracts away the strategic interactions between individual agents , it circumvents the computational intractability of the MARL approaches that do not exploit symmetry . However , most existing work on mean-field games focuses on characterizing the existence and uniqueness of the Nash equilibrium rather than designing provably efficient algorithms . In particular , most existing work considers the continuous-time setting , which requires solving a pair of Hamilton-Jacobi-Bellman ( HJB ) and Fokker-Planck ( FP ) equations , whereas the discrete-time setting is more common in practice , e.g. , in the aforementioned applications . Moreover , most existing approaches , including the ones based on solving the HJB and FP equations , require knowing the model of dynamics ( Bardi and Priuli , 2014 ) , or having the access to a simulator , which generates the next state given any state-action pair and aggregated effect of the population ( Guo et al. , 2019 ) , which is often unavailable in practice . To address these challenges , we develop an efficient model-free RL approach to mean-field game , which provably attains the Nash equilibrium . In particular , we focus on discrete-time mean-field games with linear state transitions and quadratic cost functions , where the aggregated effect of the population is quantified by the mean-field state . Such games capture the fundamental difficulties of general mean-field games and well approximates a variety of real-world systems such as power grids ( Minciardi and Sacile , 2011 ) , swarm robots ( Fang , 2014 ; Araki et al. , 2017 ; Doerr et al. , 2018 ) , and financial systems ( Zhou and Li , 2000 ; Huang and Li , 2018 ) . In detail , based on the Nash certainty equivalence ( NCE ) principle ( Huang et al. , 2006 ; 2007 ) , we propose a mean-field actor-critic algorithm which , at each iteration , given the mean-field state µ , approximately attains the optimal policy π∗µ of each agent , and then updates the mean-field state µ assuming that all the agents follow π∗µ . We parametrize the actor and critic by linear and quadratic functions , respectively , and prove that such a parameterization encompasses the optimal policy of each agent . Specifically , we update the actor parameter using policy gradient ( Sutton et al. , 2000 ) and natural policy gradient ( Kakade , 2002 ; Peters and Schaal , 2008 ; Bhatnagar et al. , 2009 ) and update the critic parameter using primal-dual gradient temporal difference ( Sutton et al. , 2009a ; b ) . In particular , we prove that given the mean-field state µ , the sequence of policies generated by the actor converges linearly to the optimal policy π∗µ . Moreover , when alternatingly update the policy and mean-field state , we prove that the sequence of policies and its corresponding sequence of mean-field states converge to the unique Nash equilibrium at a linear rate . Our approach can be interpreted from both “ passive ” and “ active ” perspectives : ( i ) Assuming that each self-interested agent employs the single-agent actorcritic algorithm , the policy of each agent converges to the unique Nash policy , which characterizes the social dynamics of a large population of model-free RL agents . ( ii ) For a particular agent , our approach serves as a fictitious self-play method for it to find its Nash policy , assuming the other agents give their best responses . To the best of our knowledge , our work establishes the first efficient model-free RL approach with function approximation that provably attains the Nash equilibrium of a discrete-time mean-field game . As a byproduct , we also show that the sequence of policies generated by the single-agent actor-critic algorithm converges at a linear rate to the optimal policy of a linear-quadratic regulator ( LQR ) problem in the presence of drift , which may be of independent interest . Related Work . Mean-field game is first introduced in Huang et al . ( 2003 ; 2006 ) ; Lasry and Lions ( 2006a ; b ; 2007 ) . In the last decade , there is growing interest in understanding continuous-time mean-field games . See , e.g. , Guéant et al . ( 2011 ) ; Bensoussan et al . ( 2013 ) ; Gomes et al . ( 2014 ) ; Carmona and Delarue ( 2013 ; 2018 ) and the references therein . Due to their simple structures , continuous-time linear-quadratic mean-field games are extensively studied under various model assumptions . See Li and Zhang ( 2008 ) ; Bardi ( 2011 ) ; Wang and Zhang ( 2012 ) ; Bardi and Priuli ( 2014 ) ; Huang et al . ( 2016a ; b ) ; Bensoussan et al . ( 2016 ; 2017 ) ; Caines and Kizilkale ( 2017 ) ; Huang and Huang ( 2017 ) ; Moon and Başar ( 2018 ) ; Huang and Zhou ( 2019 ) for examples of this line of work . Meanwhile , the literature on discrete-time linear-quadratic mean-field games remains relatively scarce . Most of this line of work focuses on characterizing the existence of a Nash equilibrium and the behavior of such a Nash equilibrium when the number of agents goes to infinity ( Gomes et al. , 2010 ; Tembine and Huang , 2011 ; Moon and Başar , 2014 ; Biswas , 2015 ; Saldi et al. , 2018a ; b ; 2019 ) . See also Yang et al . ( 2018a ) , which applies maximum entropy inverse RL ( Ziebart et al. , 2008 ) to infer the cost function and social dynamics of discrete-time mean-field games with finite state and action spaces . Our work is most related to Guo et al . ( 2019 ) , where they propose a mean-field Q-learning algorithm ( Watkins and Dayan , 1992 ) for discrete-time mean-field games with finite state and action spaces . Such an algorithm requires the access to a simulator , which , given any state-action pair and mean-field state , outputs the next state . In contrast , both our state and action spaces are infinite , and we do not require such a simulator but only observations of trajectories under given mean-field state . Correspondingly , we study the mean-field actor-critic algorithm with linear function approximation , whereas their algorithm is tailored to the tabular setting . Also , our work is closely related to Mguni et al . ( 2018 ) , which focuses on a more restrictive setting where the state transition does not involve the mean-field state . In such a setting , mean-field games are potential games , which is , however , not true in more general settings ( Li et al. , 2017 ; Briani and Cardaliaguet , 2018 ) . In comparison , we allow the state transition to depend on the mean-field state . Meanwhile , they propose a fictitious self-play method based on the single-agent actor-critic algorithm and establishes its asymptotic convergence . However , their proof of convergence relies on the assumption that the single-agent actor-critic algorithm converges to the optimal policy , which is unverified therein . In addition , our work is related to Jayakumar and Aditya ( 2019 ) , where the proposed algorithm is only shown to converge asymptotically to a stationary point of the mean-field game . Our work also extends the line of work on finding the Nash equilibria of Markov games using MARL . Due to the computational intractability introduced by the large number of agents , such a line of work focuses on finite-agent Markov games ( Littman , 1994 ; 2001 ; Hu and Wellman , 1998 ; Bowling , 2001 ; Lagoudakis and Parr , 2002 ; Hu and Wellman , 2003 ; Conitzer and Sandholm , 2007 ; Perolat et al. , 2015 ; Pérolat et al. , 2016b ; a ; 2018 ; Wei et al. , 2017 ; Zhang et al. , 2018 ; Zou et al. , 2019 ; Casgrain et al. , 2019 ) . See also Shoham et al . ( 2003 ; 2007 ) ; Busoniu et al . ( 2008 ) ; Li ( 2018 ) for detailed surveys . Our work is related to Yang et al . ( 2018b ) , where they combine the mean-field approximation of actions ( rather than states ) and Nash Q-learning ( Hu and Wellman , 2003 ) to study general-sum Markov games with a large number of agents . However , the Nash Q-learning algorithm is only applicable to finite state and action spaces , and its convergence is established under rather strong assumptions . Also , when the number of agents goes to infinity , their approach yields a variant of tabular Q-learning , which is different from our mean-field actor-critic algorithm . For policy optimization , based on the policy gradient theorem , Sutton et al . ( 2000 ) ; Konda and Tsitsiklis ( 2000 ) propose the actor-critic algorithm , which is later generalized to the natural actor-critic algorithm ( Peters and Schaal , 2008 ; Bhatnagar et al. , 2009 ) . Most existing results on the convergence of actor-critic algorithms are based on stochastic approximation using ordinary differential equations ( Bhatnagar et al. , 2009 ; Castro and Meir , 2010 ; Konda and Tsitsiklis , 2000 ; Maei , 2018 ) , which are asymptotic in nature . For policy evaluation , the convergence of primal-dual gradient temporal difference is studied in Liu et al . ( 2015 ) ; Du et al . ( 2017 ) ; Wang et al . ( 2017b ) ; Yu ( 2017 ) ; Wai et al . ( 2018 ) . However , this line of work assumes that the feature mapping is bounded , which is not the case in our setting . Thus , the existing convergence results are not applicable to analyzing the critic update in our setting . To handle the unbounded feature mapping , we utilize a truncation argument , which requires more delicate analysis . Finally , our work extends the line of work that studies model-free RL for LQR . For example , Bradtke ( 1993 ) ; Bradtke et al . ( 1994 ) show that policy iteration converges to the optimal policy , Tu and Recht ( 2017 ) ; Dean et al . ( 2017 ) study the sample complexity of least-squares temporal-difference for policy evaluation . More recently , Fazel et al . ( 2018 ) ; Malik et al . ( 2018 ) ; Tu and Recht ( 2018 ) show that the policy gradient algorithm converges at a linear rate to the optimal policy . See as also Hardt et al . ( 2016 ) ; Dean et al . ( 2018 ) for more in this line of work . Our work is also closely related to Yang et al . ( 2019 ) , where they show that the sequence of policies generated by the natural actorcritic algorithm enjoys a linear rate of convergence to the optimal policy . Compared with this work , when fixing the mean-field state , we use the actor-critic algorithm to study LQR in the presence of drift , which introduces significant difficulties in the analysis . As we show in §3 , the drift causes the optimal policy to have an additional intercept , which makes the state- and action-value functions more complicated . Notations . We denote by ‖M‖∗ the spectral norm , ρ ( M ) the spectral radius , σmin ( M ) the minimum singular value , and σmax ( M ) the maximum singular value of a matrix M . We use ‖α‖2 to represent the ` 2-norm of a vector α , and ( α ) j i to denote the sub-vector ( αi , αi+1 , . . . , αj ) > , where αk is the k-th entry of the vector α . For scalars a1 , . . . , an , we denote by poly ( a1 , . . . , an ) the polynomial of a1 , . . . , an , and this polynomial may vary from line to line . We use [ n ] to denote the set { 1 , 2 , . . . , n } for any n ∈ N .

Summary: This paper considers the problem of model-free reinforcement learning in discrete-time, linear-quadratic Markovian mean-field games with ergodic costs. The authors begin by establishing the existence and uniqueness of the Nash equilibrium in such a setting, and then proposes an mean-field actor-critic algorithm with linear function approximation. The proposed algorithm is shown to converge linearly with high probability under certain standard conditions.