Title: The Best of Both Worlds in Network Population Games: Reaching Consensus and Convergence to Equilibrium
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-Agent Learning

Consensus Formation

Smooth Fictitious Play

Network Game

Population Game
Abstract: Reaching consensus and convergence to equilibrium are two major challenges of multi-agent systems. Although each has attracted significant attention, relatively few studies address both challenges at the same time. This paper examines the connection between the notions of consensus and equilibrium in a multi-agent system where multiple interacting sub-populations coexist. We argue that consensus can be seen as an intricate component of intra-population stability, whereas equilibrium can be seen as encoding inter-population stability. We show that smooth fictitious play, a well-known learning model in game theory, can achieve both consensus and convergence to equilibrium in diverse multi-agent settings. Moreover, we show that the consensus formation process plays a crucial role in the seminal thorny problem of equilibrium selection in multi-agent learning.

The Best of Both Worlds in Network Population Games: Reaching Consensus & Convergence to Equilibrium

Abstract

Reaching consensus and convergence to equilibrium are two major challenges of multi-agent systems. Although each has attracted significant attention, relatively few studies address both challenges at the same time. This paper examines the connection between the notions of consensus and equilibrium in a multi-agent system where multiple interacting sub-populations coexist. We argue that consensus can be seen as an intricate component of intra-population stability, whereas equilibrium can be seen as encoding inter-population stability. We show that smooth fictitious play, a well-known learning model in game theory, can achieve both consensus and convergence to equilibrium in diverse multi-agent settings. Moreover, we show that the consensus formation process plays a crucial role in the seminal thorny problem of equilibrium selection in multi-agent learning.

1 Introduction

Two fundamental problems of multi-agent systems are consensus formation [26, 3, 12, 30, 5, 28, 86, 68] and learning in games [48, 33, 66, 58, 57, 70, 71, 13, 55]. The former (sometimes also referred to as cooperative control, flocking, synchronization, and social norms) aims to understand how a decentralized system of agents, which starts off in an initially disordered state, is able to develop shared coordinated beliefs, opinions, or behaviors [7]. The latter (also known as multi-agent learning) examines whether, how and what type of equilibrium arises as the result of the long-run learning processes of individual agents [33]. Each problem has attracted significant interest, yet, these two problems are largely studied in isolation.

Is there a unified way to think about consensus formation and learning in games? We argue that consensus and equilibrium , the fundamental notions of these two fields, can be both understood as stability concepts in a multi-agent system where there co-exist multiple interacting sub-populations (or populations hereafter). Consensus can be seen as an intricate component of intra -population stability that reflects whether each population is internally cohesive, i.e., a population is stable if all the agents belonging to this population achieve consensus and develop the same belief, opinion, or behavior. In contrast, game-theoretic (e.g., Nash) equilibrium can be seen at a more macro level as encoding inter -population stability, i.e., the stability between different populations is achieved by alignment to their population-wise incentives. Namely, a stable system of populations is characterized by the internal cohesion of each population as well as the game-theoretic equilibration between different populations. The merits of this way of thinking are twofold. It presents a non-trivial connection

between consensus and equilibrium, bringing them together under the same umbrella of stability concepts for a system of populations. Moreover, it distinguishes between intra-population stability and inter-population stability so that the emerging phenomena that it can describe are significantly richer, and that it allows for important distinctions that would have otherwise been impossible.

In the rich literature on consensus formation, the idea of building up a connection to game-theoretic equilibration is not entirely new. Classic mathematical models, such as the Degroot model [27] and the Friedkin-Johnsen model [31], showed that consensus of opinions can be reached if individuals form their opinions taking a weighted average of their own opinions and the opinions of others. In seminal work, Bindel et al. [15] showed that such a repeated averaging process is equivalent to a best-response play in the game with a quadratic reward function measuring the distance between individuals' opinions, leading to interest in understanding consensus from the perspective of learning in games [38, 23, 2, 14, 36, 10, 9]. In parallel, Marden et al. [62] demonstrated how the cooperative control problem of consensus can be formulated as better reply with inertia dynamics in potential games, while Young [89] proved the emergence of social norms using evolutionary game theory. These notable advances, however, typically assumed the reward structure to encode an incentive to align individual agents' opinions or behaivors with those of others. Little is formally known about consensus formation in the absence of such a coordinative reward structure, or whether such a coordinative reward structure is a prerequisite for achieving both consensus formation and gametheoretic equilibration.

In the literature on multi-agent learning, the concept of consensus has been less explored compared to the abundant study of convergent or non-convergent learning behaviors in games [13, 79, 66, 81, 77, 6, 57, 16]. We argue that consensus is of crucial importance to multi-agent learning-it ensures not only intra-population stability when there co-exist multiple interacting populations but also complete predictability and computational efficiency in more general settings. Imagine the case that for a system of agents, the average probability ( ¯ x A and ¯ x B ) of choosing between two strategies ( A and B ) is shown to equilibrate at ¯ x A = ¯ x B = 0 . 5 . If consensus is reached, it is completely predictable that every agent plays A and B with equal probability 0 . 5 . However, if consensus is not reached, there are many possibilities for agents' behaviors, to name a few-it is possible (i) that exactly half of the agents play A with probability 1 and the other half plays B with probability 1 , and (ii) that some agents play A and B with equal probability 0 . 5 whereas the rest constantly switches between these two strategies. The unpredictability in agents' behaviors can be problematic for system design, especially in safety-critical scenarios, where a system designer ought to rule out the possibility of getting the system to evolve into some (e.g., unsafe) states [74, 76, 73, 91]. From the perspective of computational efficiency, consensus is usually a desirable property and is embedded in the mean field game literature to simplify the characterization of equilibrium-by exploiting the homogeneity of the population, it suffices to understand the behavior of a single representative agent facing the mean field [55, 54, 51, 71, 42, 87, 1].

Putting these together motivates our central questions:

Are there natural multi-agent learning models that can achieve the best of both worldsreaching consensus as well as convergence to equilibrium-in diverse settings?

How does the consensus formation process affect equilibrium selection in multi-agent learning?

Our Model. We provide an affirmative answer to the first question, and our solution issmooth fictitious play (SFP) [32]-a well-known belief-based learning model in game theory. In this paper, we consider a network of populations , where each vertex represents a population (or continuum) of agents and each edge denotes that agents from connected populations are paired up to play 2-player subgames. We call this type of game a network population game, 1 as it has a close connection to the network (or graphical polymatrix) games [52] and the population games [75]. By SFP, each agent forms beliefs about the strategies of agents of the connected populations based on the average

PredictedBeliefDistribution

EmpiricalBeliefDistribution

VarianceofBeliefs

& Distance to the Equilibrium (S,S)

0.010

Variance of Beliefs

0.6

Distance to the Equilibrium

Distance to the Equilibrium

0.5

Variance of Beliefs

Probability density

20

Probability density

20

t=45

t=45

0.005

t=35

t=35

10

=25

10

t=25

t=15

Time step

t=15

Time

t=5

t=5

+0

0

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.000

0

20

40

60

80

100

0.0

Beliefs about the Other’s Playing H

Beliefs about the Other's Playing H

Time step

PredictedBeliefDistribution

EmpiricalBeliefDistribution

VarianceofBeliefs

&DistancetotheEquilibrium(H,H)

0.0006

0.3

→ Distance to the Equilibrium

Distance to the Equilibrium

Variance of Beliefs

Probability density

Probability density

0.2

40

t=45

40

t=45

0.0003

20

t=25

t=35

20

t=25

t=35

0.1

t=15

t=15

t=5

t=5

0

0.00

0.25

0.50

0.75

001

0

00°0

0.25

0.50

0.75

1.00

0.0000

0

20

40

60

80

100

0.0

Beliefs about the Other’s Playing H

Beliefs about the Other's Playing H

Time step

(b) The initial beliefs are distributed according to Beta (280 , 120) for both populations. The variance of initial beliefs is ≈ 0 . 0005 and the mean of initial beliefs is 0 . 7 .

plays of those populations, and responds with an aggregated smoothed best response. To allow for a meaningful study of consensus formation, we assume that agents within each population have different beliefs and different behaviors in the initial state. In other words, our model is a heterogeneous belief model [35], unlike the homogeneous belief models extensively studied in the SFP literature [8, 82, 11, 32, 44, 47, 46].

Our Approach & Key Results. To see whether a consensus is reached, one needs a measure of it. A natural approach is to assume the system state to be a probability measure (or distribution) over the set of beliefs (referred to as the belief distribution). This notion of a system state allows us to quantify how much agents reach a consensus by examining the variance of the belief distribution. Intuitively, a value of zero for the variance means that agents have the same belief and have achieved consensus. However, analyzing the evolution of the belief distribution requires new techniques beyond the classic approach of stochastic approximation [8, 11, 35, 44, 47]. We address this challenge by establishing a system of nonlinear partial differential equationscontinuity equations 2 -to describe the continuous-time change in the belief distribution (Proposition 1). This approach allows us to analyze the evolution of the variance without deriving an analytical form of the belief distribution. We show that for any network population games, the variance of beliefs decreases to zero at a quadratic rate with time in every population (Theorem 1). This means that even if the 2-player subgames that agents (of different populations) play are not coordinative but zero-sum, consensus will still be reached. Put differently, a coordinative reward structure is not a prerequisite for achieving consensus.

The next step is to examine the convergence to equilibrium. Our finding that the variance of beliefs decreases to zero suggests that the asymptotic behavior of a population that starts with different beliefs can be understood by studying that of a single representative agent. Inspired by this, we temporarily override the assumption of the heterogeneous belief model and assume a single representative agent for each population, resulting in the classic form of network games with each vertex representing a single agent. Under this assumption, we use Lyapunov arguments to prove the convergence of SFP to quantal response equilibria (QRE) 3 in two classes of network games that capture network competition and network coordination. Specifically, we show that for a weighted zero-sum network game, SFP converges to a unique QRE even if the underlying game has many distinct Nash equilibria (Theorem 2). We also show that for an exact potential network game that exhibits a star structure, SFP converges to the set of QRE (Theorem 3). We then use the techniques of asymptotically autonomous dynamical systems to show that the convergence to QRE holds for network population games as well (Theorem 4 and Theorem 5). Therefore, SFP converges to QRE in all weighted zero-sum network (population) games as well as all exact potential star-graph (population) games.

At this point, we have shown that SFP can achieve consensus as well as convergence to equilibrium in a wide range of network population games. Specifically, consensus is always reached, regardless of the specific setting. In the language of stability concepts, by SFP, the intra-population stability is always guaranteed, but the inter-population stability may require additional assumptions about the incentive nature as well as the network topology. This may lead to an error-prone intuition that the consensus formation process is orthogonal to the equilibrium selection process and thus has little effect on it. This is indeed not true. We illustrate this through an example of a 2-population stag hunt game, where different variances of initial beliefs result in qualitatively different longterm system behaviors. As we show in Figure 1, even though the initial mean belief is the same, the consensus formation process starting with a low level of initial disagreement (a small value for initial variance) results in the risk dominant equilibrium ( H,H ) , while starting with a high level of initial disagreement (a large value for initial variance) leads to the payoff dominant equilibrium ( S, S ) . We provide additional evidence covering all possible initial mean beliefs under the same 2-population stag-hunt game setting in Figure 2 (Section 6). Figure 2 shows that increasing the variance of initial beliefs from 0 to 0 . 02 , 0 . 05 , and 0 . 1 expands the region of attraction of the payoff dominant equilibrium ( S, S ) , allowing a wider range of initial mean beliefs to approach it. Thus, in the case of network coordination, consensus formation plays a crucial role in the seminal thorny problem of equilibrium selection, and a larger initial variance can promote convergence to the equilibrium ( S, S ) . In the case of network competition, convergence to a unique QRE is formally guaranteed (Theorem 4); we empirically verify this in 5-population asymmetric matching pennies games with many distinct Nash equilibria (Section 1 of the Appendix).

Other Related Work. SFP is a well-studied model that converges in various classes of games, including most 2-player-2-action games [32, 44, 11, 47], n-player potential game s[82, 45], and zero-sum and identical-interest stochastic games [8, 78, 59]. However, SFP in network (population) games has not been previously explored. Our research extends the findings of Fudenberg and Takahashi [35] (who showed that agents with different beliefs in a 2-population setting will converge to the same belief) to the network settings, but we demonstrate that the variance of initial beliefs plays a crucial role in equilibrium selection, which has not been found in [35]. On the other hand, networked multi-agent learning is a current frontier in AI research [90, 61, 43, 21, 17, 4, 25]. Previous studies have shown the non-convergent behavior of replicator dynamics in (weighted) zero-sum network games [72, 18, 66, 79], the convergence of smooth Q-learning in weighted potential [56] and weighted zero-sum network games [57], and the convergence of fictitious play in zero-sum network games [29]. This paper provides new convergence results for SFP in both network games and network population games, covering a wide range of generic 2-population-2-action settings, star-graph settings that admit an exact potential, and all weighted zero-sum settings. Finally, the evolution of probability distributions over initial conditions or strategies has also received increasing interest [22, 53, 49, 50, 24, 60]. Continuity equations have been used as a tool to show that under Cross learning, agents with different mixed strategies in a population game will not converge to the same mixed strategy [53], and to also describe the more complicated Q-learning dynamics in various settings [49, 50, 24, 60]. This paper formally shows that the probability distribution over

initial conditions can eventually degenerate to a point mass, and leveraging on this, presents a novel technique for proving the convergence of learning dynamics. 4

2 Preliminaries

̸

A network population game Γ = ( N, ( V, E ) , ( S i , ω i ) ∀ i ∈ V , ( A ij ) ( i,j ) ∈ E ) consists of a multi-agent system N distributed over a connected graph ( V, E ) , where V = { 1 , ..., n } is the set of vertices each represents a population (continuum) of agents, and E is the set of pairs, ( i, j ) , of population i = j ∈ V . For each population i ∈ V , agents of this population has a finite set S i of pure strategies (or actions) with generic elements s i ∈ S i . Agents may also use mixed strategies (or choice distributions). For an arbitrary agent k in population i , its mixed strategy is a vector x i ( k ) ∈ ∆ i , where ∆ i is the simplex in R | S i | such that ∑ s i ∈ S i x is i ( k ) = 1 and x is i ( k ) ≥ 0 , ∀ s i ∈ S i . Each edge ( i, j ) ∈ E defines a series of two-player subgames between populations i and j . We denote the payoff matrices for agents of population i and j in these two-player subgames by A ij ∈ R | S i |×| S j | and A ji ∈ R | S j |×| S i | , respectively. For every time step, each agent chooses a (mixed or pure) strategy and plays that strategy in all two-player subgames with agents from the connected (or neighbor) populations. Let x = ( x i , { x j } ( i,j ) ∈ E ) be a mixed strategy profile, where x i (or x j ) denotes a generic mixed strategy in population i (or j ). Given the mixed strategy profile x , the expected payoff of using x i in the game Γ is

Smooth (or stochastic) fictitious play (SFP) is a belief-based model for learning in games. By SFP, agents act as if they are Bayesian such that they assume that the strategies of other agents are drawn from some fixed but unknown distribution. Given a game Γ , consider an arbitrary agent k in a population i ∈ V . Let V i = { j ∈ V : ( i, j ) ∈ E } be the set of neighbor populations. Agent k maintains a weight κ i js j ( k ) for each opponent strategy s j ∈ S j of each neighbor population j ∈ V i . For every time step t , agents refine these weights based on the average play of each neighbor population, i.e., for each opponent strategy s j ∈ S j , j ∈ V i ,

where ¯ x js j is the mean probability of playing strategy s j in population j . 5 For simplicity, we assume the initial sum of weights ∑ s j ∈ S j κ i js j ( k, 0) to be the same for every agent in the system and denote this initial sum by λ . Based on the weights, agent k forms a belief about the neighbor population j such that each opponent strategy s j is played with probability

Let µ i j ( k ) be the vector of beliefs with the s j -th element equals µ i js j ( k ) . Given a game Γ , agent k 's expected payoff of using a pure strategy s i ∈ S i in response to the aggregate beliefs µ i j ( k ) , ∀ j ∈ V i about neighbor populations is

where e s i is a unit vector with the s i -th element equal 1 . By convention [33, 34, 44], we suppose that agents' payoffs are perturbed and that agents play smooth best responses maximizing their perturbed

payoffs. Agent k 's perturbed payoff of using a mixed strategy x i ( k ) ∈ ∆ i is defined as

where ϵ > 0 , and the function v is strictly concave and its gradient becomes arbitrarily large near the boundary of the simplex ∆ i [44]. 6 For the function v , a typical choice is v ( x i ( k )) = -1 β ∑ s i ∈ S i x is i ( k ) ln( x is i ( k )) . Under this form, the maximization of the perturbed payoff π i yields a unique choice distribution x i ( k ) from the logit choice rule [45], that is,

where β is the temperature (or the degree of rationality). For clarity of our presentation, we choose to focus on the logit choice rule in this paper; however, all our results readily generalize to any function v satisfying the above two standard assumptions. In the following, we usually drop the time index t and agent index k in the bracket (depending on the context) for notational convenience.

3 Belief Dynamics in Network Population Games

Observe that for an arbitrary agent k of population i , its belief µ i j ( k ) about population j is in the simplex ∆ j = { µ i j ( k ) ∈ R | S j | | ∑ s j ∈ S j µ i js j ( k ) = 1 , µ i js j ( k ) ≥ 0 , ∀ s j ∈ S j } . We assume that the system state is characterized by a Borel probability measure P defined on the state space ∆ = ∏ i ∈ V ∆ i . Given µ i ∈ ∆ i , we write the marginal probability density function as p ( µ i , t ) . Note that p ( µ i , t ) represents the density of agents having the belief µ i about population i throughout the system ; it does not represent the density of agents of population i that have the belief µ i , nor the density of agents having the belief µ i about population i in a certain population. Define µ = { µ i } i ∈ V ∈ ∆ . Since agents maintain separate beliefs about different neighbor populations, the joint probability density function p ( µ , t ) can be factorized, i.e., p ( µ , t ) = ∏ i ∈ V p ( µ i , t ) . We make the following assumption for the initial marginal density functions.

Assumption 1. At time t = 0 , for each population i ∈ V , the marginal density function p ( µ i , t ) is continuously differentiable and has zero mass at the boundary of the simplex ∆ i .

This assumption excludes the setting that a population of agents starts off with the same belief as well as the setting that agents start off with extreme beliefs. Under this mild condition, we derive a partial differential equation to describe the evolution of the belief distribution.

Proposition 1. The continuous-time dynamic of the marginal density function p ( µ i , t ) for each population i ∈ V is described by a partial differential equation

where ∇· is the divergence operator and ¯ x i is the mean mixed strategy with each s i -th element

The partial differential equation in Equation 7 is also known as a continuity equation in the study of transport phenomena (e.g., of mass or energy) in a physical system. We remark that continuity equations, tracking the evolution of a probability distribution, do not allow for general solutions. However, continuity equations provide a tractable way to examine the evolution of the moments of the probability distribution, without obtaining an analytical form of that distribution. The analytical form for the dynamic of the variance of beliefs (the second moment of the belief distribution) is given as follows.

Theorem 1. The dynamic of the variance of beliefs µ i about each population i ∈ V is governed by an ordinary differential equation such that for each strategy s i ,

At given time t , Var ( µ is i ) = ( λ +1 λ + t +1 ) 2 σ 2 ( µ is i ) , where σ 2 ( µ is i ) is the initial variance. Thus, the variance Var ( µ is i ) decays to zero at a quadratic rate with time.

Note that the above theorem makes no assumption about the 2-player subgames agents play, the network structure of populations, or even the initial belief distribution. Therefore, it shows that for any network of populations, the variance of beliefs about each population always asymptotically tends to zero. As an immediate consequence of this theorem, the asymptotic distribution of the beliefs is as follows:

Corollary 1. The probability density function p ( µ i , t ) for every population i ∈ V will eventually evolve into a Dirac delta function, implying that in the long run agents reach a consensus and play the same mixed strategy within each population.

The intuition of Theorem 1 and Crollary 1 is that under SFP, agents revise their beliefs about other populations based on the average play of those populations; consequently, the observations are the same within an agent population, and thus it is only a matter of time that the agent population will develop the same belief and play the same strategy. Once consensus is reached, agents within a population will not develop different beliefs afterwards, as ensured by Theorem 1. In this sense, the agent population is internally cohesive and achieves intra-population stability.

However, it does not mean that different agent populations will eventually play the same strategy. This is because agents' strategies are the smooth best responses to their aggregated beliefs about their own set of neighbor populations (which generally vary depending on the network topology). This naturally raises the question of how different agent populations align with their population-wise incentives (which can be of a competitive nature). This also sets our paper apart from previous work that studies game-theoretic equilibration in the context of consensus formation [15, 62, 38, 23, 2, 14, 36, 10, 9], in which consensus is achieved as the result of agents' incentives to align their opinions or behaiviors to those of others (i.e., the incentives are of a coordinative nature).

Before diving into the convergence results, we take a moment to analyze the mean belief (first moment) dynamic. Intuitively, the mean belief represents the average system state. The following proposition provides an analytical form of the mean belief dynamic.

Proposition 2. The mean belief dynamic for each population i ∈ V is governed by an ordinary differential equation such that for each strategy s i ,

where f s i ( { µ } j ∈ V i ) is the logit choice rule (Equation 6) applied to strategy s i ∈ S i , and Var ( µ js j ) is the variance of belief µ js j in the entire system.

Weremark that the mean belief dynamic, to be exact, should be under the joint influence of infinitely many moments of the belief distribution. However, the higher the order of the moments, the less of the influence of the moments, and by convention, the effects of the third and higher moments can be assumed negligible [39, 41, 64]. 7 Thus, in this proposition, the mean belief dynamic depends on the current mean belief and also on the current variance of beliefs.

Recall that Theorem 1 shows that the variance of beliefs will eventually decrease to zero. As a result, the second term of the mean belief dynamic (Equation 10) will asymptotically tend to zero. This suggests that the asymptotic mean belief dynamic of a population that starts with different beliefs in the initial state might be understood by studying the belief dynamic of a single representative agent.

Intuitively, if each population can be represented by a single agent, then the concerned network population games will effectively degenerate into the classic form of network games with each vertex representing a single agent. Inspired by this, in the next section, we temporarily focus on SFP in the classic network games.

4 Convergence of Smooth Fictitious Play in Classic Network Games

In a classic network game (or network game for short), the only difference with respect to the network population games is that now each vertex represents a single agent. We derive the belief dynamic of each vertex (or agent) in the following proposition.

Proposition 3. In a network game, the belief dynamic of each agent is governed by an ordinary differential equation such that for each agent i and each strategy s i ,

It follows from this proposition that the equilibrium state should satisfy the following property.

Proposition 4. In a network game, the equilibrium state should satisfy that x ∗ i = µ ∗ i for every strategy s i ∈ S i and every population i ∈ V . Thus, the equilibrium state corresponds to the solution to the system of equations

Such an equilibrium state coincides with the Quantal Response Equilibria [65] of the network game.

By QRE, in this paper, we refer to their canonical form also referred to as logit QRE in the literature [40]. We remark that QRE [65] is a game theoretic solution concept under bounded rationality, and is an extension of the standard Nash equilibrium which allows for errors in choice.

Weighted zero-sum network games capture the case of network competition, and are the generalization of 2-player zero-sum games to the network settings [19]. A network game is weighted zero-sum if there exist positive constants (or weights) ω 1 , . . . , ω n such that

where r i ( x ) is the expected payoff of using a mixed strategy x i given the mixed strategy profile x = ( x i , { x j } ( i,j ) ∈ E ) . If subgames along every edge ( i, j ) ∈ E are 2-player zero-sum, i.e., x ⊤ i A ij x j + x ⊤ j A ji x i = 0 , then the network game is zero-sum; however, the converse is not generally true. As opposed to 2-player zero-sum games that admit a unique value at equilibrium [85], the Nash equilibrium payoffs in weighted zero-sum network games need not be unique, and it generally allows for infinitely many Nash equilibria. In the following theorem, we show that regardless of the number of Nash equilibria in a network game, the belief dynamic is guaranteed to converge to a unique QRE.

Theorem 2. For any weighted zero-sum network game, the belief dynamic converge to a unique QRE which is globally asymptotically stable.

Exact potential network games capture the case of network coordination. A network game admits an exact potential if for each edge ( i, j ) ∈ E , the payoff matrices of the two-player subgame satisfy A ij = A ⊤ ji [21, 20]. Pure or mixed Nash equilibria in network coordination are complex. As reported in recent works [20, 17, 4], finding a pure Nash equilibrium is PLS-complete. Hence, learning in the general case of network coordination is difficult and generally requires some conditions [67, 69]. In the following theorem, we show the convergence result in some cases of network coordination.

Theorem 3. For any exact potential network game that exhibits a star structure, each orbit of the belief dynamic converges to the set of QRE.

We remark that we employ the Lyapunov approach (i.e., establishing the existence of the Lyapunov function) to prove the above convergence results. As a result, the convergence to equilibrium is equivalent to the network of agents being stable in the sense of Lyapunov.

Basins of Attraction of (S,S) and (H,H）under Different Variances of Initial Beliefs

o²(μ1H) = 0

²(μ1H)=0.02

²(μ1H)=0.05

(S,S)

H

Initial μ2H

Initial μ2H

Initial μ2H

0.8

0.8

0.8

0.8

Initial

0.6

0.6

0.6

0.6

(H,H)

0.6

0.8

1

0.6

0.8

1

0.6

0.8

1

0.6

0.8

1

Initial μ1H

Initialp1H

Initialp1H

Initial p1H

5 Convergence of Smooth Fictitious Play in Network Population Games

We shall now formalize that the convergence results in network games can be seamlessly carried over to network population games with different beliefs in the initial state, and establish the interpopulation stability in network population games. Recall that the mean belief dynamic (Proposition 2) depends on the current mean belief as well as the current variance of beliefs, with the variance decreasing over time (Theorem 1). Leveraging on the definition of asymptotically autonomous dynamical systems [63], we establish the following lemma.

Lemma 1. In a network population game, for every population i ∈ V , the mean belief dynamic (Equation 10) is asymptotically autonomous with the limit equation being d µ i dt = x i -µ i , which after time-reparmeterization is equivalent to the belief dynamic in a network game (Equation 11).

Thieme [84] provided the following seminal result that formally connects the asymptotic behavior of an asymptotically autonomous dynamical system to that of its associated limit equation.

Lemma 2 (Thieme [84] Theorem 4.2) . Given a metric space ( X,d ) , let the solution flows of an asymptotically autonomous system and its limit equation be ϕ and Θ , respectively. Assume that the equilibria of Θ are isolated compact Θ -invariant subsets of X . The ω -Θ -limit set of any precompact Θ -orbit contains a Θ -equilibrium. The point ( s, x ) , s ≥ t 0 , x ∈ X , have a pre-compact ϕ -orbit. Then the following alternative holds: 1) ϕ ( t, s, x ) → e, t → ∞ , for some Θ -equilibrium e, and 2) the ω -ϕ -limit set of ( s, x ) contains finitely many Θ -equilibria which are chained to each other in a cyclic way.

Combining the above lemmas and the convergence results under the setting of classic network games, the convergence results of SFP in network population games are readily deducible:

Theorem 4. For any weighted zero-sum network population game, the belief (as well as the choice distribution) of every individual converges to a unique QRE.

Theorem 5. For any exact potential network population game that exhibits a star structure, the belief (as well as the choice distribution) of every individual converges to the set of QRE.

Note that under the 2-population settings, there is only one possible graph structure-one edge connecting the two populations. Hofbauer and Hopkins [44] have shown that SFP converges in all 2-player weighted potential games, and observed that almost all 2-player-2-action games are either 2-player weighted potential games or 2-player weighted zero-sum games. Therefore, combining their results and our aforementioned results, SFP converges to the set of QRE under almost all the 2-population-2-action settings (including all the 2-population-2-action weighted potential settings and all the 2-population-2-action weighted zero-sum settings).

6 Experiments: Equilibrium Selection in Two-Population Stag Hunt Games

In a two-population stag hunt game, agents from two populations are paired up to choose between two actions H (hare) and S (stag). If both players hunt a hare, they both receive a reward of 1 . If

both players hunt a stag, they both receive a reward of 4 . If one hunts a hare and the other hunts a stag, the former receives a reward of 2 whereas the latter receives nothing. There are two pure strategy Nash equilibria in each subgame: the risk dominant equilibrium ( H,H ) and the payoff dominant equilibrium ( S, S ) , which naturally raises the problem of equilibrium selection.

Figure 1 showed that given the same initial mean belief, changing the variances of initial beliefs can result in different limit behaviors. In Figure 2, we systematically studied the effect of the variance of initial beliefs by visualizing how it affects the regions of attraction to different equilibria. To be more specific, we numerically solved the mean belief dynamics for a wide range of initial mean beliefs, given different variances of initial beliefs, and colored the regions of attraction. We observed in Figure 2 that as the variance of initial beliefs increases (from the left to right panel), a wider range of initial mean beliefs results in the convergence to the QRE that approximates the payoff dominant equilibrium ( S, S ) . Hence, consensus is inevitable, but the initial disagreement still provides an approach to equilibrium selection and helps select the highly desirable equilibrium.

Note that in the case of network competition, the convergence to a unique QRE is formally guaranteed; we empirically verified this through a 5-population asymmetric matching pennies game presented in Appendix A.

7 Conclusions

This paper presents a formal treatment for SFP in network population games where each population of agents start off with different beliefs and strategies. Representing the system state with a distribution over beliefs, we show that consensus is always reached, and that agents eventually pointwise converge to the same belief, regardless of specific settings. In addition, we establish the convergence of SFP to Quantal Response Equilibria in general competitive network population games as well as exact potential network population games with star structure. Empirically, we show that although the initial belief heterogeneity vanishes in the limit, it plays a crucial role in equilibrium selection and helps select highly desirable equilibria.

Acknowledgment

Shuyue Hu would like to thank Ms. Die Hu for her assistance in generating Figure 1 and the corresponding video presented in this paper. This research is supported in part by the National Research Foundation, Singapore and DSO National Laboratories under its AI Singapore Program (AISG Award No: AISG2-RP-2020-016), NRF2019-NRFANR095 ALIAS grant, NRF 2018 Fellowship NRF-NRFF2018-07, grant PIESGP-AI-2020-01, AME Programmatic Fund (Grant No.A20H6b0151) from A*STAR and Provost's Chair Professorship grant RGEPPV2101.

References

Andrea Angiuli, Nils Detering, Jean-Pierre Fouque, Mathieu Lauriere, and Jimin Lin. Reinforcement learning algorithm for mixed mean field control games. arXiv preprint arXiv:2205.02330 , 2022.

Vincenzo Auletta, Ioannis Caragiannis, Diodato Ferraioli, Clemente Galdi, and Giuseppe Persiano. Generalized discrete preference games. In IJCAI , volume 16, pages 53-59, 2016.

Vincenzo Auletta, Angelo Fanelli, and Diodato Ferraioli. Consensus in opinion formation processes in fully evolving environments. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 6022-6029, 2019.

Yakov Babichenko and Aviad Rubinstein. Settling the complexity of nash equilibrium in congestion games. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 1426-1437, 2021.

Rafael Bailo, Mattia Bongini, José A Carrillo, and Dante Kalise. Optimal consensus control of the cucker-smale model. IFAC-PapersOnLine , 51(13):1-6, 2018.

D Balduzzi, S Racaniere, J Martens, J Foerster, K Tuyls, and T Graepel. The mechanics of n-player differentiable games. In ICML , volume 80, pages 363-372. JMLR. org, 2018.

Andrea Baronchelli. The emergence of consensus: a primer. Royal Society open science , 5(2):172189, 2018.

Lucas Baudin and Rida Laraki. Smooth fictitious play in stochastic games with perturbed payoffs and unknown transitions. In Advances in Neural Information Processing Systems .

Dario Bauso and Mark Cannon. Consensus in opinion dynamics as a repeated game. Automatica , 90:204-211, 2018.

Dario Bauso, Hamidou Tembine, and Tamer Basar. Opinion dynamics in social networks through mean-field games. SIAM Journal on Control and Optimization , 54(6):3225-3257, 2016.

Michel Benaım and Morris W Hirsch. Mixed equilibria and dynamical systems arising from fictitious play in perturbed games. Games and Economic Behavior , 29(1-2):36-72, 1999.

Jon Atli Benediktsson and Philip H Swain. Consensus theoretic classification methods. IEEE transactions on Systems, Man, and Cybernetics , 22(4):688-704, 1992.

Martino Bernasconi, Federico Cacciamani, Simone Fioravanti, Nicola Gatti, and Francesco Trovo. The evolutionary dynamics of soft-max policy gradient in games. In Proceedings of the AAAI Conference on Artificial Intelligence , 2022.

Kshipra Bhawalkar, Sreenivas Gollapudi, and Kamesh Munagala. Coevolutionary opinion formation games. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing , pages 41-50, 2013.

David Bindel, Jon Kleinberg, and Sigal Oren. How bad is forming your own opinion? In 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science , pages 57-66. IEEE, 2011.

Daan Bloembergen, Karl Tuyls, Daniel Hennes, and Michael Kaisers. Evolutionary dynamics of multi-agent learning: A survey. Journal of Artificial Intelligence Research , 53:659-697, 2015.

Shant Boodaghians, Rucha Kulkarni, and Ruta Mehta. Smoothed efficient algorithms and reductions for network coordination games. arXiv preprint arXiv:1809.02280 , 2018.

Victor Boone and Georgios Piliouras. From darwin to poincaré and von neumann: Recurrence and cycles in evolutionary and algorithmic game theory. In International Conference on Web and Internet Economics , pages 85-99. Springer, 2019.

Yang Cai, Ozan Candogan, Constantinos Daskalakis, and Christos Papadimitriou. Zerosum polymatrix games: A generalization of minmax. Mathematics of Operations Research , 41(2):648-655, 2016.

Yang Cai and Constantinos Daskalakis. On minmax theorems for multiplayer games. In Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete algorithms , pages 217-234. SIAM, 2011.

Yang Cai et al. The complexity of Nash equilibria in multiplayer zero-sum games and coordination games . PhD thesis, Massachusetts Institute of Technology, 2010.

Yun Kuen Cheung, Georgios Piliouras, and Yixin Tao. The evolution of uncertainty of learning in games. In International Conference on Learning Representations .

Flavio Chierichetti, Jon Kleinberg, and Sigal Oren. On discrete preferences and coordination. In Proceedings of the fourteenth ACM conference on Electronic commerce , pages 233-250, 2013.

Chen Chu, Yong Li, Jinzhuo Liu, Shuyue Hu, Xuelong Li, and Zhen Wang. A formal model for multiagent q-learning dynamics on regular graphs. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI , pages 194-200, 2022.

Aleksander Czechowski and Georgios Piliouras. Poincar \ ' { e } -bendixson limit sets in multiagent learning. 2022.

Abir De, Isabel Valera, Niloy Ganguly, Sourangshu Bhattacharya, and Manuel Gomez Rodriguez. Learning and forecasting opinion dynamics in social networks. Advances in neural information processing systems , 29, 2016.

Morris H DeGroot. Reaching a consensus. Journal of the American Statistical association , 69(345):118-121, 1974.

Yucheng Dong, Quanbo Zha, Hengjie Zhang, Gang Kou, Hamido Fujita, Francisco Chiclana, and Enrique Herrera-Viedma. Consensus reaching in social network group decision making: Research paradigms and challenges. Knowledge-Based Systems , 162:3-13, 2018.

Christian Ewerhart and Kremena Valkanova. Fictitious play in networks. Games and Economic Behavior , 123:182-206, 2020.

Dimitris Fotakis, Dimitris Palyvos-Giannas, and Stratis Skoulakis. Opinion dynamics with local interactions. In IJCAI , pages 279-285, 2016.

Noah E Friedkin and Eugene C Johnsen. Social influence and opinions. Journal of Mathematical Sociology , 15(3-4):193-206, 1990.

Drew Fudenberg and David M Kreps. Learning mixed equilibria. Games and economic behavior , 5(3):320-367, 1993.

Drew Fudenberg and David K Levine. The theory of learning in games , volume 2. MIT press, 1998.

Drew Fudenberg and David K Levine. Conditional universal consistency. Games and Economic Behavior , 29(1-2):104-130, 1999.

Drew Fudenberg and Satoru Takahashi. Heterogeneous beliefs and local information in stochastic fictitious play. Games and Economic Behavior , 71(1):100-120, 2011.

Jason Gaitonde, Jon Kleinberg, and Éva Tardos. Polarization in geometric opinion dynamics. In Proceedings of the 22nd ACM Conference on Economics and Computation , pages 499-519, 2021.

Sriram Ganapathi Subramanian, Pascal Poupart, Matthew E Taylor, and Nidhi Hegde. Multi type mean field reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems , pages 411-419, 2020.

Javad Ghaderi and Rayadurgam Srikant. Opinion dynamics in social networks with stubborn agents: Equilibrium and convergence rate. Automatica , 50(12):3209-3215, 2014.

Colin S Gillespie. Moment-closure approximations for mass-action models. IET systems biology , 3(1):52-58, 2009.

Jacob K Goeree, Charles A Holt, and Thomas R Palfrey. Quantal response equilibrium. In Quantal Response Equilibrium . Princeton University Press, 2016.

Leo A Goodman. Population growth of the sexes. Biometrics , 9(2):212-225, 1953.

Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. Learning mean-field games. In Advances in Neural Information Processing Systems , pages 4967-4977, 2019.

Shubham Gupta, Rishi Hazra, and Ambedkar Dukkipati. Networked multi-agent reinforcement learning with emergent communication. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems , pages 1858-1860, 2020.

Josef Hofbauer and Ed Hopkins. Learning in perturbed asymmetric games. Games and Economic Behavior , 52(1):133-152, 2005.

Josef Hofbauer and William H Sandholm. On the global convergence of stochastic fictitious play. Econometrica , 70(6):2265-2294, 2002.

Josef Hofbauer and William H Sandholm. Evolution in games with randomly disturbed payoffs. Journal of economic theory , 132(1):47-69, 2007.

Ed Hopkins. A note on best response dynamics. Games and Economic Behavior , 29(1-2):138150, 1999.

Junling Hu, Michael P Wellman, et al. Multiagent reinforcement learning: theoretical framework and an algorithm. In ICML , volume 98, pages 242-250. Citeseer, 1998.

Shuyue Hu, Chin-wing Leung, and Ho-fung Leung. Modelling the dynamics of multiagent qlearning in repeated symmetric games: a mean field theoretic approach. In Advances in Neural Information Processing Systems , pages 12102-12112, 2019.

Shuyue Hu, Chin-Wing Leung, Ho-fung Leung, and Harold Soh. The dynamics of q-learning in population games: A physics-inspired continuity equation model. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems , page 615-623, 2022.

Minyi Huang, Roland P Malhamé, Peter E Caines, et al. Large population stochastic dynamic games: closed-loop mckean-vlasov systems and the nash certainty equivalence principle. Communications in Information & Systems , 6(3):221-252, 2006.

Elena Janovskaja. Equilibrium points in polymatrix games. Lithuanian Mathematical Journal , 8(2):381-384, 1968.

Ratul Lahkar and Robert M Seymour. Reinforcement learning in population games. Games and Economic Behavior , 80:10-38, 2013.

Jean-Michel Lasry and Pierre-Louis Lions. Mean field games. Japanese journal of mathematics , 2(1):229-260, 2007.

Mathieu Laurière, Sarah Perrin, Matthieu Geist, and Olivier Pietquin. Learning mean field games: A survey. arXiv preprint arXiv:2205.12944 , 2022.

Stefanos Leonardos and Georgios Piliouras. Exploration-exploitation in multi-agent learning: Catastrophe theory meets game theory. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11263-11271, 2021.

Stefanos Leonardos, Georgios Piliouras, and Kelly Spendlove. Exploration-exploitation in multi-agent competition: Convergence with bounded rationality. Advances in Neural Information Processing Systems , 34, 2021.

Stefanos Leonardos, Iosif Sakos, Costas Courcoubetis, and Georgios Piliouras. Catastrophe by design in population games: Destabilizing wasteful locked-in technologies. In Proceedings of the 16th International Conference on Web and Internet Economics , pages 7-11, 2020.

David S Leslie, Steven Perkins, and Zibo Xu. Best-response dynamics in zero-sum stochastic games. Journal of Economic Theory , 189:105095, 2020.

Chin-wing Leung, Shuyue Hu, and Ho-fung Leung. Modelling the dynamics of multi-agent q-learning: The stochastic effects of local interaction and incomplete information.

Yiwei Liu, Jiamou Liu, Kaibin Wan, Zhan Qin, Zijian Zhang, Bakhadyr Khoussainov, and Liehuang Zhu. From local to global norm emergence: Dissolving self-reinforcing substructures with incremental social instruments. In International Conference on Machine Learning , pages 6871-6881. PMLR, 2021.

Jason R Marden, Gürdal Arslan, and Jeff S Shamma. Connections between cooperative control and potential games illustrated on the consensus problem. In 2007 European Control Conference (ECC) , pages 4604-4611. IEEE, 2007.

L Markus. Asymptotically autonomous differential systems. contributions to the theory of nonlinear oscillations iii (s. lefschetz, ed.), 17-29. Annals of Mathematics Studies , 36, 1956.

Timothy I Matis and Ivan G Guardiola. Achieving moment closure through cumulant neglect. The Mathematica Journal , 12:12-2, 2010.

Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games. Games and economic behavior , 10(1):6-38, 1995.

Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 2703-2717. SIAM, 2018.

Sai Ganesh Nagarajan, David Balduzzi, and Georgios Piliouras. From chaos to order: Symmetry and conservation laws in game dynamics. In International Conference on Machine Learning , pages 7186-7196. PMLR, 2020.

Angelia Nedic, Asuman Ozdaglar, and Pablo A Parrilo. Constrained consensus and optimization in multi-agent networks. IEEE Transactions on Automatic Control , 55(4):922-938, 2010.

Gerasimos Palaiopanos, Ioannis Panageas, and Georgios Piliouras. Multiplicative weights update with constant step-size in congestion games: Convergence, limit cycles and chaos. Advances in Neural Information Processing Systems , 30, 2017.

Julien Perolat, Remi Munos, Jean-Baptiste Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder, et al. From poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization. In International Conference on Machine Learning , pages 8525-8535. PMLR, 2021.

Sarah Perrin, Julien Pérolat, Mathieu Laurière, Matthieu Geist, Romuald Elie, and Olivier Pietquin. Fictitious play for mean field games: Continuous time analysis and applications. Advances in Neural Information Processing Systems , 33:13199-13213, 2020.

Georgios Piliouras and Jeff S Shamma. Optimization despite chaos: Convex relaxations to complex limit sets via poincaré recurrence. In Proceedings of the twenty-fifth annual ACMSIAM symposium on Discrete algorithms , pages 861-873. SIAM, 2014.

Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang, Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich. Rmix: Learning risk-sensitive policies for cooperative reinforcement learning agents. Advances in Neural Information Processing Systems , 34:23049-23062, 2021.

D Sai Koti Reddy, Amrita Saha, Srikanth G Tamilselvam, Priyanka Agrawal, and Pankaj Dayama. Risk averse reinforcement learning for mixed multi-agent environments. In Proceedings of the 18th international conference on autonomous agents and multiagent systems , pages 2171-2173, 2019.

William H Sandholm. Population games and evolutionary dynamics . MIT press, 2010.

Arnab Sarker, Peter Fisher, Joseph E. Gaudio, and Anuradha M. Annaswamy. Accurate parameter estimation for safety-critical systems with unmodeled dynamics. Artificial Intelligence , 316:103857, 2023.

Yuzuru Sato, Eizo Akiyama, and J Doyne Farmer. Chaos in learning a simple two-person game. Proceedings of the National Academy of Sciences , 99(7):4748-4751, 2002.

Muhammed O Sayin, Kaiqing Zhang, and Asuman Ozdaglar. Fictitious play in markov games with single controller. In Proceedings of the 23rd ACM Conference on Economics and Computation , pages 919-936, 2022.

Stratis Skoulakis, Tanner Fiez, Ryann Sim, Georgios Piliouras, and Lillian Ratliff. Evolutionary game theory squared: Evolving agents in endogenously evolving zero-sum games. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11343-11351, 2021.

Gordon Dennis Smith. Numerical solution of partial differential equations: finite difference methods . Clarendon Press, 1985.

Sriram Srinivasan, Marc Lanctot, Vinicius Zambaldi, Julien Pérolat, Karl Tuyls, Rémi Munos, and Michael Bowling. Actor-critic policy optimization in partially observable multiagent environments. Advances in neural information processing systems , 31, 2018.

Brian Swenson and H Vincent Poor. Smooth fictitious play in n × 2 potential games. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers , pages 1739-1743. IEEE, 2019.

György Szabó and Gabor Fath. Evolutionary games on graphs. Physics reports , 446(4-6):97216, 2007.

Horst R Thieme. Convergence results and a poincaré-bendixson trichotomy for asymptotically autonomous differential equations. Journal of mathematical biology , 30(7):755-763, 1992.

J v. Neumann. Zur theorie der gesellschaftsspiele. Mathematische annalen , 100(1):295-320, 1928.

Tushar Vaidya, Carlos Murguia, and Georgios Piliouras. Learning agents in financial markets: Consensus dynamics on volatility. In AAMAS , pages 2106-2108, 2018.

Qiaomin Xie, Zhuoran Yang, Zhaoran Wang, and Andreea Minca. Provable fictitious play for general mean-field games. arXiv preprint arXiv:2010.04211 , 2020.

Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In International Conference on Machine Learning , pages 5567-5576, 2018.

H Peyton Young. The evolution of social norms. economics , 7(1):359-387, 2015.

Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning , pages 5872-5881. PMLR, 2018.

Andrey Zhitnikov and Vadim Indelman. Simplified risk-aware decision making with beliefdependent rewards in partially observable domains. Artificial Intelligence , 312:103775, 2022.

A Experiments on Network Competition

We have shown in Theorem 5 (of the main paper) that SFP converges to a unique QRE in any weighted zero-sum network population game even if there are multiple Nash equilibria underlying that game. In the following, we corroborate this by providing empirical evidence in agent-based simulations with different belief initialization.

Game Description. Consider a five-population asymmetric matching pennies game [56], where the network structure is a line (depicted in Figure 1). Each agent has two actions { H,T } . Agents in populations 1 and 5 do not learn; they always play strategies H and T , respectively. For agents in populations 2 to 4 , they receive +1 if they match the strategy of the opponent in the next population, and receive -1 if they mismatch. On the contrary, they receive +1 if they mismatch the strategy of the opponent in the previous population, and receive -1 if they match. Hence, this game has infinitely many Nash equilibria of the form: agents in populations 2 and 4 play strategy T , whereas agents in population 3 are indifferent between strategies H and T .

Experimental Setups. In this game, agents in each population form two beliefs (one for the previous population and one for the next population). We are mainly interested in the strategies of population 3 , as the Nash equilibria differ in the strategies in population 3 . Thus, we let the initial beliefs about populations 1, 3 and 5 remain unchanged across different cases, and vary population 3 's initial beliefs about populations 2 and 4. The initial beliefs about populations 1, 3 and 5, denoted by µ 1 H , µ 3 H and µ 5 H , are distributed according to the distributions Beta (20 , 10) , Beta (6 , 4) , and Beta (10 , 5) , respectively. The initial beliefs about populations are given in the legends of Figure 2. In all cases, the initial sum of weights λ = 10 and the temperature β = 10 . Note that µ iT = 1 -µ iH for all populations i = 1 , 2 , 3 , 4 , 5 . We run 100 simulation runs for each initialization, and each simulation run consists of 1 , 000 agents in each population.

Results. As shown in Figure 4, given differential initialization of beliefs, agents in population 3 converge to the same equilibrium where they all take strategy H with probability 0 . 5 . Therefore, even when the underlying zero-sum game has many Nash equilibria, SFP with different initial belief heterogeneity selects a unique equilibrium, addressing the problem of equilibrium selection.

-1

+1

-1

+1

-1

+1

Population1

Population2

Population3

Population

14

Population5

H

{H, T}

{H, T}

{H, T}

T

Match

Match

Match

Match

Probability ofPlayingH inPopulation 3

Vary theinitalMean,Fix theinitialVariance

1

1

B(5,10),B(8,2)

B(10,20),B(16,4)

B(5,10),B(2,8)

B(2.5,5),B(4,1)

B(10,5),B(8,2)

B(50,100),B(80,20)

HC

HC

0.5

0.5

0

0

100

200

300

400

100

200

300

400

Time t

Time t

B Proof of Proposition 1

It follows from Equation 2 and Equation 3 of the main paper that the change in µ i j ( k, t ) between two discrete time steps is

Lemma 3. Under Assumption 1 of the main paper, for an arbitrary agent k in population i , its belief µ i j ( k, t ) about a neighbor population j will never reach the extreme belief, i.e., the probability density for the boundary of the simplex ∆ i will remain zero.

Proof. Assumption 1 ensures that ¯ x j (0) is in the interior of the simplex ∆ j . Moreover, the logit choice function (Equation 5 in the main paper) also ensures that ¯ x j ( t ) stays in the interior of ∆ j afterwards for a finite temperature β . Hence, from Equation 14, one can see that µ i j ( k, t ) for every time step t will stay in the interior of ∆ j .

In the following, for notation convenience, we sometimes drop the agent index k and the time index t depending on the context. Consider a population i . We rewrite the change in the beliefs about this population as follows.

Suppose that the amount of time that passes between two successive time steps is δ ∈ (0 , 1] . We rewrite the above equation as

Next, we consider a test function θ ( µ i ) . Define

Applying Taylor series for θ ( µ i ( t + δ )) at µ i ( t ) , we obtain

where H denotes the Hessian matrix. Hence, the expectation E [ θ ( µ i ( t + δ ))] is

Moving the term E [ θ ( µ i ( t ))] to the left hand side and dividing both sides by δ , we recover the quantity Y , i.e.,

Taking the limit of Y with δ → 0 , the contribution of the second term on the right hand side vanishes, yielding

Apply integration by parts. We obtain

where we have leveraged that the probability mass p ( µ i , t ) at the boundary ∂ ∆ i remains zero as a result of Lemma 1. On the other hand, according to the definition of Y ,

Therefore, we have the equality

As θ is a test function, this leads to

Rearranging the terms, we obtain Equation 7 of the main paper. By the definition of expectation given a probability distribution, it is straightforward to obtain Equation 8 of the main paper. Q.E.D.

C Proof of Theorem 1

Without loss of generality, we consider the variance of the belief µ is i about strategy s i of population i . Note that

Consider the first term on the right hand side. We apply the Leibniz rule to interchange differentiation and integration, and then substitute ∂p ( µ i ,t ) ∂t with Equation 8 in the main paper.

E

d

[(

µ

is dt

where γ := -1 λ + t +1 . Applying integration by parts to the first term in Equation 32 yields ∫ ( µ is i ) 2 ∑ s i ∈ S i ∂ µ is i p ( µ i , t ) (¯ x is i -µ is i ) d µ i = -∫ ( µ is i ) 2 p ( µ i , t )   ∑ s ′ i ∈ S i ∂ µ is ′ i (¯ x is ′ i -µ is ′ i )   + p ( µ i , t ) ∂ µ is i [ ( µ is i ) 2 (¯ x is i -µ is i ) ] d µ i (33)

)

2

]

Hence, we have

i

where we have leveraged that the probability mass at the boundary remains zero (Lemma 1). Combining the above two equations, we obtain

Next, we consider the second term in Equation 28. By Lemma 4, we have

Combining Equations 37 and 38, the dynamics of the variance is

Q.E.D.

D Proof of Proposition 2

Lemma 4. The dynamics of the mean belief ¯ µ i about each population i ∈ V is governed by a differential equation

Proof. The time derivative of the mean belief about strategy s i is

We apply the Leibniz rule to interchange differentiation and integration, and then substitute ∂p ( µ i ,t ) ∂t with Equation 8 in the main paper.

where γ := -1 λ + t +1 . Apply integration by parts to the first term in Equation 48.

where we have leveraged that the probability mass at the boundary remains zero. Hence, it follows from Equation 48 that

We repeat the mean probability ¯ x is i , which has been given in Equation 8 in the main paper, as follows:

Applying the Taylor expansion to approximate this function at the mean belief ¯ µ , we have

where H denotes the Hessian matrix. Hence, we can rewrite Equation 55 as

Observe that in Equation 59, the second and the third term can be canceled out. Moreover, for any two neighbor populations j, k ∈ V i , the beliefs µ j , µ k about these two populations are updated separately and independently. Hence, the covariance of these beliefs are zero. We apply the moment closure approximation [64, 39] with the second order and obtain

Hence, substituting ¯ x is i in Lemma 4 with the above approximation, we have the mean belief dynamics

Q.E.D.

E Proof of Proposition 3

It follows from Equation 2 and Equation 3 of the main paper that the change in beliefs between two successive time steps is as follows.

Suppose that the amount of time that passes between two successive time steps is δ ∈ (0 , 1] . We rewrite the above equation as

Move the term µ i ( t ) to the right hand side and divide both sides by δ ,

Assume that the amount of time δ between two successive time steps goes to zero. we have

Q.E.D.

F Proof of Proposition 4

It is straightforward to see that

Denote the equilibrium points of the system dynamics, which satisfies the above equation, by ( x ∗ i , µ ∗ i ) for each population i . By the logit choice rule, we have

Leveraging that x ∗ i = µ ∗ i , ∀ i ∈ V at equilibrium, we can replace µ ∗ j with x ∗ j . Q.E.D.

G Proof of Theorem 2

Consider an agent i in a classic network game. The set of neighbors is V i , the set of beliefs about the neighbors is { µ j } j ∈ V i , and the choice distribution is x i . Given a classic network game, the expected payoff is given by x ⊤ i ∑ ( i,j ) ∈ E A ij µ j . Define a perturbed payoff function

where v ( x i ) = -1 β ∑ s i ∈ S i x is i ln( x is i ) . Under this form of v ( x i ) , the maximization of π i yields the choice distribution x i from the logit choice function [33]. Based on this, we establish the following lemma.

Lemma 5. For a choice distribution x i of SFP in a network game,

Proof. This lemma immediately follows from the fact that the maximization of π i will yield the choice distribution x i from the logit choice function [33].

The belief dynamics of an agent can be simplified after time-reparameterization.

Lemma 6. Given τ = ln λ + t +1 λ +1 , the belief dynamics of homogeneous systems (given in Equation 11 in the main paper) is equivalent to

Proof. From τ = ln λ + t +1 λ +1 , we have

By the chain rule, for each dimension s i ,

Next, we define the Lyapunov function L as

where { ω i } i ∈ V is the set of positive weights defined in the weighted zero-sum Γ . The function L is non-negative because for every i ∈ V , x i maximizes the function π i . When for every i ∈ V , x i = µ i , the function L reaches the minimum value 0 .

Rewrite L as

We observe that π i ( x i , { µ j } j ∈ V i ) is convex in µ j , j ∈ V i by Danskin's theorem, and -v ( µ i ) is strictly convex in µ i . Moreover, by the weighted zero-sum property given in Equation 2 in the main paper, we have

since µ i ∈ ∆ i , µ j ∈ ∆ j for every i, j ∈ V. Therefore, the function L is a strictly convex function and attains its minimum value 0 at a unique point x i = µ i , ∀ i ∈ V.

Consider the function L i . Its time derivative is

Note that the partial derivative ∂ x i π i equals 0 by Lemma 5. Thus, we can rewrite this as

where from Equation 81 to 82, we apply Lemma 5 to substitute ∑ j ∈ V i ( A ij µ j ) ⊤ with -∂ x i v ( x i ) . Hence, summing over all the populations, the time derivative of L is

The summation in the second line is equivalent to

By the weighted zero-sum property given in Equation 2 in the main paper, this summation equals 0 , yielding

Note that the function v is strictly concave such that its second derivative is negative definite. By this property, ˙ L ≤ 0 with equality only if x i = µ i , ∀ i ∈ V , which corresponds to the QRE. Therefore, L is a strict Lyapunov function, and the global asymptotic stability of the QRE follows. Q.E.D.

H Proof of Theorem 3

Consider a root agent j of a star structure. Its set of leaf (neighbor) agents is V j , the set of beliefs about the leaf agents is { µ i } i ∈ V j , and the choice distribution is x j . Given the game Γ , the expected payoff is x ⊤ j ∑ i ∈ V j A ji µ i . Define a perturbed payoff function

where v ( x j ) = -1 β ∑ s j ∈ S j x js j ln( x js j ) . Under this form of v ( x j ) , the maximization of π j yields the choice distribution x j from the logit choice function [33].

Consider a leaf agent i of the root agent j . It has only one neighbor, which is population j . Thus, given the game Γ , the expected payoff is x ⊤ i A ij µ j . Define a perturbed payoff function

where v ( x i ) = -1 β ∑ s i ∈ S i x is i ln( x is i ) . Similarly, the maximization of π i yields the choice distribution x i from the logit choice function [33]. Based on this, we establish the following lemma.

Lemma 7. For choice distributions of SFP in a network game with a star structure,

Proof. This lemma immediately follows from the fact that the maximization of π j and π i , respectively, yield the choice distributions x j and x i from the logit choice function [33].

Let R⊂ V be the set of all root agents. We define

Consider the function L j . Its time derivative ˙ L j is

Since we have ( A ij µ j ) ⊤ = µ ⊤ j A ⊤ ij = µ ⊤ j A ji , applying Lemma 7, we can substitute ∑ i ∈ V j ( A ji µ i ) ⊤ with -∂ x j v ( x j ) , and µ ⊤ j A ji with -∂ x i v ( x i ) , yielding

Note that the function v is strictly concave such that its second derivative is negative definite. By this property, ˙ L j ≥ 0 with equality only if x i = µ i , ∀ i ∈ V j and x j = µ j . Thus, the time derivative of the function L , i.e., ˙ L = ∑ j ∈R ˙ L j ≥ 0 with equality only if x i = µ i , ∀ i ∈ V j , x j = µ j , ∀ j ∈ R . Q.E.D.

I Proof of Lemma 1

Definition 1. A nonautonomous system of differential equations in R n

is said to be asymptotically autonomous with limit equation

if f ( t, x ) → g ( x ) , t →∞ , where the convergence is uniform on each compact subset of R n . Conventionally, the solution flow of Eq. 96 is called the asymptotically autonomous semiflow (denoted by ϕ ) and the solution flow of Eq. 97 is called the limit semiflow (denoted by Θ ).

We first time-reparameterize the mean belief dynamics of heterogeneous systems. Assume τ = ln λ + t +1 λ +1 . By the chain rule and Equation 61, for each dimension s i ,

Observe that exp( -2 τ ) decays to zero exponentially fast and that both σ 2 ( µ js j ) and ∂ 2 f s i (¯ µ ) ( ∂µ js j ) 2 are bounded for every µ in the simplex ∏ j ∈ V i ∆ j . Hence, Equation 101 converges locally and uniformly to the following equation:

Note that x is i = f s i (¯ µ ) for a single representative agent, and thus the above equation is algebraically equivalent to the limit equation in Lemma 1 of the main paper. Q.E.D.

J Numerical Methods, Source Code, and Computing Resource

Numerical Method for the PDE model. Only limited types of PDEs allow analytic solutions. Hence, we numerically solve the PDE using the finite difference method [80]. The theoretical predictions in Figure 1 of the main paper are generated using the finite difference method given a specific initial setting (the initial sum of weights is λ = 10 , the temperature is β = 10 , the initial belief distribution is specified in the caption of the figures).

Source Code and Computing Resource. The source code for reproducing our main experiments is included in the supplementary material (available at https://sites.google.com/view/shuyue-hu ). The Matlab script finitedifference.m numerically solves our continuity equation model. Figure 1 of the main paper is generated using the Origins Lab based on the numerical solution of the continuity equations. The Matlab script regionofattraction.m visualizes the region of attraction of different equilibria in stag hunt games, which are depicted in Figure 2. The Python scripts simulation(staghunt).py and simulation(matchingpennies).py correspond to the agent-based simulations in two-population stag hunt games and five-population asymmetric matching pennies games, respectively. We use a laptop (CPU: AMD Ryzen 7 5800H) to run all the experiments.

Peer reviews:
comments: SUMMARY:
This paper examines the connection between the notions of consensus and equilibrium in a multi-agent system where multiple interacting sub-populations coexist. They argue that consensus can be seen as an intricate component of intra-population stability, whereas equilibrium can be seen as encoding inter-population stability. They show that smooth fictitious play can achieve both consensus and convergence to equilibrium in diverse multi-agent settings.

STRENGTHS:
Excellent paper that brings together the concepts of consensus and equilibrium. Strong theory, interesting experiments.

WEAKNESSES:
1) don't forget the conclusion in the final version of the paper. 
2) It is clear that in the long run, due to the strong law of large numbers, FP is such that agents within a population will form the same beliefs. Therefore, the point of view taken by the authors is to study a representative agent. I say: what could be interesting is the transient regime where agents beliefs have not yet converged. There, one could use the central limit theorem, and look at the interplay between, on the one side, consensus that has not been reached yet, and the convergence of populations to equilibria. You could also look at it with 2 different learning rates (one for intra, one for inter populations, I suggest you explore connections with this https://arxiv.org/pdf/2205.02330.pdf)

QUESTIONS:
please comment on 2) above.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors defined a network population game, which is a multipartite network game where each partite is a population and agents in the same population do not interact with each other and only interact with agents from other populations. In this way, each population can be easily abstracted into a \"super-agent\", and each population has separate beliefs about different neighbor populations. \n\nThe authors show that when the agents perform interactions using the network population model and adopt a smooth fictitious play dynamic, the population's belief will gradually have lower variance and the mean belief will reach a quantal response equilibrium (QRE) in both weighted zero-sum games and exact potential network games.\n\nSTRENGTHS:\n1. The intention of this paper is good, which tries to address both the consensus and the convergence of multi-agent learning systems\n2. The overall flow of this paper is easy to follow, and the authors conduct both theoretical and numerical studies\n\nWEAKNESSES:\n1. The justification for using the network population game is not sufficient. When this specific game model fits real-world problems, especially related to belief updates. For now, the purpose of this assumption seems like making the convergence and part much easier to study. \n2. The reason for using the smooth fictitious play dynamics is also not sufficiently justified. The reason for including the \\nu penalty term in the utility function should be provided (e.g., saying this is an entropy-based cost and why this is reasonable), and whether it is necessary for the convergence and consensus study should be elaborated. \n3. No elaboration on how the \\epsilon term and the A_ij influence the consensus and convergence\n4. Lack of discussion on the limitations\n\nQUESTIONS:\n1. What is the intuition behind the perturbed payoff in Eqn (5)? Is this cost term a necessity for consensus and convergence?\n2. Line 203 says \"Agents maintain separate beliefs about different neighbor populations\", where is this previously justified and why is this reasonable?\n\nLIMITATIONS:\nThe authors did not discuss the limitations of this work in the paper.\n\nI'm not sure if and how the population size of each population will influence the consensus and equilibrium outcome and whether there will be any fairness issues based on this. For now, I will not flag ethics review issues since I can't identify or exclude them. \n\nIt will be nice for the authors to add discussions on the limitations, at least in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper combines reaching consensus and convergence to equilibrium for network population games. Consider a network whose vertices correspond to a population. Edges between vertices (or populations) represent two-player sub-games between each pair of agents in these neighboring populations. The authors specifically focus on smoothed fictitious play for these two-player sub-games while agents seek to reach consensus in their beliefs about agents' policies in neighboring populations. In that sense, the approach is analogous to (or motivated by) the anonymous random matching interpretation of fictitious play dynamics to justify the myopic nature of the agents [Fudenberg and Kreps, Learning mixed equilibria. Games and Economic Behavior, 1993]. In particular, consider (large) populations of agents in each player role. Each period, all agents are matched to play the game and are told only to play in their own match. Agents are unlikely to play their current opponent again for a long time, even unlikely to play anyone who played anyone who played her. So, if the population size is large enough compared to the discount factor, it is not worth sacrificing current payoff to influence an opponent’s future play. In these populations, agents share their belief. The consensus dynamics presented serve this purpose. Therefore, the results are expected to hold even though I have not checked the proofs in detail.\n\nSTRENGTHS:\n- Convergence of SFP dynamics in weighted zero-sum network games and exact potential network games with star structure.\n\nWEAKNESSES:\n- There is no motivating example for the network population game formulation.\n- Results for potential network games are presented only for star structure.\n\nQUESTIONS:\n- Can you provide a motivating example justifying the network population model in practice (specifically the 2-player sub-games)?\n- What is the reason to restrict the network structure to star structure only in potential network games?\n\nLIMITATIONS:\nI have not identified any discussion about the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the connection between the notions of consensus and equilibrium in a multi-agent system where multiple interacting sub-populations coexist, and it aims at answering below two central research questions in network (population) games scenario: \n     [1] Are there natural multi-agent learning models that can achieve the best of both worlds—reaching consensus as well as convergence to equilibrium—in diverse settings?\n     [2] How does the consensus formation process affect equilibrium selection in multi-agent learning?\n\nThe authors argue that consensus and equilibrium, the fundamental notions of these two fields, can be both understood as stability concepts in a multi-agent system where there co-exist multiple interacting sub-populations. In particular, consensus can be seen as an intricate component of intra-population stability, whereas equilibrium can be seen as encoding inter-population stability. The authors show that SFP (smooth fictitious play) algorithm can achieve consensus as well as convergence to equilibrium in a wide range of network population games (and unlike previous literature, here a coordinative reward structure is not a prerequisite for achieving\nconsensus). They also empirically shows that consensus formation process plays a crucial role in the seminal thorny problem of equilibrium's election in multi-agent learning (e.g., starts with the same initial mean belief, a large variance of initial beliefs results in a more desired equilibrium). Experiments were conducted for the scenario of Equilibrium Selection in Two-Population Stag Hunt Games.\n\nSTRENGTHS:\n1. The paper is organized and presented well and clearly, I found the manuscript is reader friendly.\n\n2. This work extend existing literature in multiple directions and presented a couple of nontrivial novel theoretical results, which is quite beneficial to the research community. E.g., SFP in network (population) games has not been previously explored until this work, unifying consensus formation and learning in multi-agent games, etc., proving consensus without assuming a coordinative reward structure, etc..\n\n3. There are quite some helpful/valuable elaborations/explanations about comparing this work with relevant literature and discussing the differences and advantages.\n\nWEAKNESSES:\n1. I'd like to see some discussions about the future research directions, and how this work could inspire/benefit other future research.\n\n2. Regarding the figure 1 about impacts of variances of initial beliefs, it seems to be just plot of one example setting, and the empirical conclusion is not very convincing to me and I'd like to see some more clarification/elaboration/justification. For example, consider such a scenario where the starting mean belief is already the optimal value of a final desired steady state/equilibrium, it seems that if the staring variance is 0 (extreme case of small variance), then convergence to optimal results is already achieved, which is obviously better than a larger initial variance with the same starting mean belief, which is a counter example of the empirical conclusion of the paper saying that a larger initial variance (given same starting mean belief) is preferred. I would like to see either some rigorous mathematical proofs about such conclusion, or very comprehensive empirical studies on this before drawing the conclusions on this. \n\n3. In page 4, it was mentioned that \"This paper formally shows that the probability distribution over initial conditions can eventually degenerate to a point mass, and leveraging on this, presents a novel technique for proving the convergence of learning dynamics.\" It would be good to see some more elaborations on how this \"novel technique\" could be used for proving the convergence of learning dynamics in other relevant problems.\n\nQUESTIONS:\nPlease see my questions/comments/suggestions in above section when talking about weakness.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}