Learning Graphon Mean Field Games and Approximate Nash Equilibria

1 INTRODUCTION . Today , reinforcement learning ( RL ) finds application in various application areas such as robotics ( Kober et al. , 2013 ) , autonomous driving ( Kiran et al. , 2021 ) or navigation of stratospheric balloons ( Bellemare et al. , 2020 ) as a method to realize effective sequential decision-making in complex problems . RL remains a very active research area , and there remain many challenges in multi-agent reinforcement learning ( MARL ) as a generalization of RL such as learning goals , non-stationarity and scalability of algorithms ( Zhang et al. , 2021 ) . Nonetheless , potential applications for MARL are manifold and include e.g . teams of unmanned aerial vehicles ( Tožička et al. , 2018 ; Pham et al. , 2018 ) or video games ( Berner et al. , 2019 ; Vinyals et al. , 2017 ) . While the domain of MARL is somewhat empirically successful , problems quickly become intractable as the number of agents grows , and methods in MARL typically miss a theoretical foundation . A recent tractable approach to handling the scalability problem in MARL are competitive mean field games and cooperative mean field control ( Gu et al. , 2021 ) . Instead of considering generic multi agent Markov games , one considers many agents under the weak interaction principle , i.e . each agent alone has a negligible influence on all other agents . This class of models naturally contains a large number of real world scenarios and can find application e.g . in analysis of power network resilience ( Bagagiolo & Bauso , 2014 ) , smart heating ( Kizilkale & Malhame , 2014 ) , edge computing ( Banez et al. , 2019 ) or flocking ( Perrin et al. , 2021b ) . See also Djehiche et al . ( 2017 ) for a review of other engineering applications . Mean field games . Mean field games ( MFGs ) were first popularized in the independent seminal works of Huang et al . ( 2006 ) and Lasry & Lions ( 2007 ) for the setting of differential games with diffusion-type dynamics given by stochastic differential equations . See also Guéant et al . ( 2011 ) ; Bensoussan et al . ( 2013 ) for a review . Since then , extensions have been manifold and include e.g . discrete-time ( Saldi et al. , 2018 ) , partial observability ( Saldi et al. , 2019 ) , major-minor formulations ( Nourian & Caines , 2013 ) and many more . In the learning community , there has been immense recent interest in finding and analyzing solution methods for mean field equilibria ( Cardaliaguet & Hadikhanloo , 2017 ; Mguni et al. , 2018 ; Guo et al. , 2019 ; Subramanian & Mahajan , 2019 ; Elie et al. , 2020 ; Cui & Koeppl , 2021 ; Pasztor et al. , 2021 ; Perolat et al. , 2021 ; Perrin et al. , 2021a ) , solving the inverse reinforcement learning problem ( Yang et al. , 2018a ) or applying related approximations directly to MARL ( Yang et al. , 2018b ) . In contrast to our work , Yang et al . ( 2018a ) consider states instead of agents on a graph , while Yang et al . ( 2018b ) requires restrictive assumptions and only considers averages instead of distributions of the neighbors . Recently , focus increased also on the cooperative case of mean field control ( Carmona et al. , 2019b ; Mondal et al. , 2021 ) , for which dynamic programming holds on an enlarged state space , resulting in a high-dimensional Markov decision process ( Pham & Wei , 2018 ; Motte & Pham , 2019 ; Gu et al. , 2020 ; Cui et al. , 2021 ) . Mean field systems on graphs . For mean field systems on dense graphs , prior work mostly considers mean field systems without control ( Vizuete et al. , 2020 ) or time-dynamics , i.e . the static case ( Parise & Ozdaglar , 2019 ; Carmona et al. , 2019a ) . To the best of our knowledge , Gao & Caines ( 2017 ) and Caines & Huang ( 2019 ) are the first to consider general continuous-time diffusion-type graphon mean field systems with control , the latter proposing many clusters of agents as well as proving an approximate Nash property as the number of clusters and agents grows . There have since been efforts to control cooperative graphon mean field systems with diffusive linear dynamics using spectral methods ( Gao & Caines , 2019a ; b ) . On the other hand , Bayraktar et al . ( 2020 ) ; Bet et al . ( 2020 ) consider large non-clustered systems in a continuous-time diffusion-type setting without control , while Aurell et al . ( 2021b ) and Aurell et al . ( 2021a ) consider continuous-time linear-quadratic systems and continuous-time jump processes respectively . To the best of our knowledge , only Vasal et al . ( 2021 ) have considered solving and formulating a graphon mean field game in discrete time , though requiring analytic computation of an infinite-dimensional value function defined over all mean fields and thus being inapplicable to arbitrary problems in a black-box , learning manner . In contrast , we give a general learning scheme and also provide extensive theoretical analysis of our algorithms and ( slightly different ) model . Finally , for sparse graphs there exist preliminary results ( Gkogkas & Kuehn , 2020 ; Lacker & Soret , 2020 ) , though the setting remains to be developed . Our contribution . In this work , we propose a dense graph limit extension of MFGs in discrete time , combining graphon mean field systems with mean field games . More specifically , we consider limits of many-agent systems with discrete-time graph-based dynamics and weak neighbor interactions . In contrast to prior works , we consider one of the first general discrete-time formulations as well as its controlled case , which is a natural setting for many problems that are inherently discrete in time or to be controlled digitally at discrete decision times . Our contribution can be summarized as : ( i ) formulating one of the first general discrete-time graphon MFG frameworks for approximating otherwise intractable large dense graph games ; ( ii ) providing an extensive theoretical analysis of existence and approximation properties in such systems ; ( iii ) providing general learning schemes for finding graphon mean field equilibria , and ( iv ) empirically evaluating our proposed approach with verification of theoretical results in the finite N -agent graph system , finding plausible approximate Nash equilibria for otherwise infeasible large dense graph games with many agents . 2 DENSE GRAPH MEAN FIELD GAMES . In the following , we will give a dense graph N -agent model as well as its corresponding mean field system , where agents are affected only by the overall state distribution of all neighbors , as visualized in Figure 1 . As a result of the law of large numbers , this distribution will become deterministic – the mean field – as N →∞ . We begin with graph-theoretical preliminaries , see also Lovász ( 2012 ) for a review . The study of dense large graph limits deals with the limiting representation of adjacency matrices called graphons . Define I : = [ 0 , 1 ] and W0 as the space of all bounded , symmetric and measurable functions ( graphons ) W ∈ W0 , W : I × I → R bounded by 0 ≤ W ≤ 1 . For any simple graph G = ( { 1 , . . . , N } , E ) , we define its step-graphon a.e . uniquely by WG ( x , y ) = ∑ i , j∈ { 1 , ... , N } 1 ( i , j ) ∈E · 1x∈ ( i−1N , iN ] · 1y∈ ( j−1N , jN ] , ( 1 ) see e.g . Figure 1 . We equipW0 with the cut ( semi- ) norm ‖·‖ and cut ( pseudo- ) metric δ ‖W‖ : = sup S , T ∣∣∣∣ ∫ S×T W ( x , y ) dxdy ∣∣∣∣ , δ ( W , W ′ ) : = infϕ ‖W −W ′ ϕ‖ , ( 2 ) for graphons W , W ′ ∈ W0 and W ′ϕ ( x , y ) : = W ′ ( ϕ ( x ) , ϕ ( y ) ) , where the supremum is over all measurable subsets S , T ⊆ I and the infimum is over measure-preserving bijections ϕ : I → I . To provide motivation , note that convergence in δ is equivalent to e.g . convergence of probabilities of locally encountering any fixed subgraph by randomly sampling a subset of nodes . Many such properties of graph sequences ( GN ) N∈N converging to some graphon W ∈ W0 can then be described by W , and we point to Lovász ( 2012 ) for details . In this work , we will primarily use the analytical fact that for converging graphon sequences ‖WGN −W‖ → 0 , we equivalently have ‖WGN −W‖L∞→L1 = sup ‖g‖∞≤1 ∫ I ∣∣∣∣ ∫ I ( WGN ( α , β ) −W ( α , β ) ) g ( β ) dβ ∣∣∣∣ dα→ 0 ( 3 ) under the operator norm of operators L∞ → L1 , see e.g . Lovász ( 2012 ) , Lemma 8.11 . By Lovász ( 2012 ) , Theorem 11.59 , the above is equivalent to convergence in the cut metric δ ( WGN , W ) → 0 up to relabeling . In the following , we will therefore assume sequences of simple graphs GN = ( VN , EN ) with vertices VN = { 1 , . . . , N } , edge sets EN , edge indicator variables ξNi , j : = 1 ( i , j ) ∈EN for all nodes i , j ∈ VN , and associated step graphons WN converging in cut norm . Assumption 1 . The sequence of step-graphons ( WN ) N∈N converges in cut norm ‖·‖ or equivalently in operator norm ‖·‖L∞→L1 as N →∞ to some graphon W ∈ W0 , i.e . ‖WN −W‖ → 0 , ‖WN −W‖L∞→L1 → 0 . ( 4 ) Next , we define W -random graphs to consist of vertices VN : = { 1 , . . . , N } with adjacency matrices ξN generated by sampling graphon indices αi uniformly from I and edges ξNi , j ∼ Bernoulli ( W ( αi , αj ) ) for all vertices i , j ∈ VN . For experiments , by Lovász ( 2012 ) , Lemma 10.16 , we can thereby generate a.s. converging graph sequences by sampling W -random graphs for any fixed graphon W ∈ W0 . In principle , one could also consider arbitrary graph generating processes whenever a valid relabeling function ϕ is known . In our work , the usage of graphons enables us to find mean field systems on dense graphs and to extend the expressiveness of classical MFGs . As examples , we will use the limiting graphons of uniform attachment , ranked attachment and p-Erdős–Rényi ( ER ) random graphs given by Wunif ( x , y ) = 1−max ( x , y ) , Wrank ( x , y ) = 1− xy and Wer ( x , y ) = p respectively ( Borgs et al. , 2011 ; Lovász , 2012 ) , each of which exhibits different node connectivities as shown in Figure 2 . Finite agent graph game . For simplicity of analysis , we consider finite state and action spaces X , U as well as times T : = { 0 , 1 , . . . , T − 1 } . On a metric space A , define the spaces of all Borel probability measures P ( A ) and all Borel measures B1 ( A ) bounded by 1 , equipped with the L1 norm . For simplified notation , we denote both a measure ν and its probability mass function by ν ( · ) . Define the space of policies Π : = P ( U ) T ×X , i.e . agents apply Markovian feedback policies πi = ( πit ) t∈T ∈ Π that act on local state information . This allows for the definition of weakly interacting agent state and action random variables Xi0 ∼ µ0 , U it ∼ πit ( · | Xit ) , Xit+1 ∼ P ( · | Xit , U it , Git ) , ∀t ∈ T , ∀i ∈ VN ( 5 ) under some transition kernel P : X ×U×B1 ( X ) → P ( X ) , where the empirical neighborhood mean field Git of agent i is defined as the B1 ( X ) -valued ( unnormalized ) neighborhood state distribution Git : = 1 N ∑ j∈VN ξNi , jδXjt , ( 6 ) where δ is the Dirac measure , i.e . each agent affects each other at most negligibly with factor 1/N . Finally , for each agent i we define separate , competitive objectives JNi ( π 1 , . . . , πN ) : = E [ T−1∑ t=0 r ( Xit , U i t , Git ) ] ( 7 ) to be maximized over πi , where r : X × U × B1 ( X ) → R is an arbitrary reward function . Remark 1 . We can also consider infinite horizons , J̃Ni ( π1 , . . . , πN ) ≡ E [ ∑∞ t=0 γ tr ( Xit , U i t , Git ) ] with all results but Theorem 4 holding . One may also extend to state-action distributions , heterogeneous starting conditions and time-dependent r , P , though we avoid this for expositional simplicity . Remark 2 . Note that it is straightforward to extend to heterogeneous agents by modelling agent types as part of the agent state , see also e.g . Mondal et al . ( 2021 ) . It is only required to model agent states in a unified manner , which does not imply that there can be no heterogeneity . With this , we can give a typical notion of Nash equilibria as found e.g . in Saldi et al . ( 2018 ) . However , under graph convergence in Assumption 1 , it is always possible for a finite number of nodes to have an arbitrary neighborhood differing from the graphon as N → ∞ . Thus , it is impossible to show approximate optimality for all nodes and only possible to show for an increasingly large fraction 1 − p → 1 of nodes . For this reason , we slightly weaken the notion of Nash equilibria by restricting to a fraction 1− p of agents , as e.g . considered in ( Carmona , 2004 ; Elie et al. , 2020 ) . Definition 1 . An ( ε , p ) -Markov-Nash equilibrium ( almost Markov-Nash equilibrium ) for ε , p > 0 is defined as a tuple of policies ( π1 , . . . , πN ) ∈ ΠN such that for any i ∈ WN , we have JNi ( π 1 , . . . , πN ) ≥ sup π∈Π JNi ( π 1 , . . . , πi−1 , π , πi+1 , . . . , πN ) − ε , ( 8 ) for some setWN ⊆ VN containing at least b ( 1− p ) Nc agents , i.e . |WN | ≥ b ( 1− p ) Nc . The minimal such ε > 0 for any fixed policy tuple ( and typically p = 0 ) is also called its exploitability . Whilst we ordain ε-optimality only for a fraction 1− p of agents , if the fraction p is negligible , it will have negligible impact on other agents as a result of the weak interaction property . Thus , the solution will remain approximately optimal for almost all agents for sufficiently small p regardless of the behavior of that fraction p of agents . In the following , we will give a limiting system that shall provide ( ε , p ) -Markov-Nash equilibria with ε , p→ 0 as N →∞ . Graphon mean field game . The formal N → ∞ limit of the N -agent game constitutes its graphon mean field game ( GMFG ) , which shall be rigorously justified in Section 3 . We define the space of measurable state marginal ensemblesMt : = P ( X ) I and measurable mean field ensembles M : = P ( X ) T ×I , in the sense that α 7→ µαt ( x ) is measurable for any µ ∈ M , t ∈ T , x ∈ X . Similarly , we define the space of measurable policy ensembles Π ⊆ ΠI , i.e . with measurable α 7→ παt ( u | x ) for any π ∈ Π , t ∈ T , x ∈ X , u ∈ U . In the GMFG , we will consider infinitely many agents α ∈ I instead of the finitely many i ∈ VN . As a result , we will have infinitely many policies πα ∈ Π – one for each agent α – through some measurable policy ensemble π ∈ Π . We again define state and action random variables Xα0 ∼ µ0 , Uαt ∼ παt ( · | Xαt ) , Xαt+1 ∼ P ( · | Xαt , Uαt , Gαt ) , ∀ ( α , t ) ∈ I × T ( 9 ) where we introduce the ( now deterministic ) B1 ( X ) -valued neighborhood mean field of agents α as Gαt : = ∫ I W ( α , β ) µβt dβ ( 10 ) for some deterministic µ ∈M . Under fixed π ∈ Π , µαt should be understood as the law of Xαt , µαt ≡ L ( Xαt ) . Finally , define the maximization objective of agent α over πα for fixed µ ∈M as Jµα ( π α ) ≡ E [ T−1∑ t=0 r ( Xαt , U α t , Gαt ) ] . ( 11 ) To formulate the limiting version of Nash equilibria , we define a map Ψ : Π→M mapping from a policy ensemble π ∈ Π to the corresponding generated mean field ensemble µ = Ψ ( π ) ∈M by µα0 ≡ µ0 , µαt+1 ( x′ ) ≡ ∑ x∈X µαt ( x ) ∑ u∈U παt ( u | x ) P ( x′ | x , u , Gαt ) , ∀α ∈ I ( 12 ) where integrability in ( 10 ) holds by induction , and note how then µαt = L ( Xαt ) . Similarly , let Φ : M → 2Π map from a mean field ensemble µ to the set of optimal policy ensembles π characterized by πα ∈ arg maxπ∈Π Jµα ( π ) for all α ∈ I , which is particularly fulfilled if παt ( u | x ) > 0 =⇒ u ∈ arg maxu′∈U Qµα ( t , x , u′ ) for all α ∈ I , t ∈ T , x ∈ X , u ∈ U , where Qµα is the optimal action value function under fixed µ ∈M following the Bellman equation Qµα ( t , x , u ) = r ( x , u , Gαt ) + ∑ x′∈X P ( x′ | x , u , Gαt ) arg max u′∈U Qµα ( t+ 1 , x ′ , u′ ) ( 13 ) with Qµα ( T , x , u ) ≡ 0 and generally time-dependent , see Puterman ( 2014 ) for a review . We can now define the GMFG version of Nash equilibria as policy ensembles π generating mean field ensembles µ under which they are optimal , as µαt = L ( Xαt ) if all agents α ∈ I follow πα . Definition 2 . A Graphon Mean Field Equilibrium ( GMFE ) is a pair ( π , µ ) ∈ Π ×M such that π ∈ Φ ( µ ) and µ = Ψ ( π ) .

Summary: This paper studies Markov games with large number of agents. The authors start by proposing a novel discrete-time formulation for graphon mean field games. They show that the mean field game exists, and that finite graph games converge to the mean field game when the number of agents goes to infinity. They also provide an algorithm that learns an approximate Nash equilibria.