SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The Best of Both Worlds in Network Population Games: Reaching Consensus & Convergence to Equilibrium
Shuyue Hu
Shanghai Artificial Intelligence Laboratory
hushuyue@pjlab.org.cn
Harold Soh
harold@comp.nus.edu.sg
Reaching consensus and convergence to equilibrium are two major challenges of multi-agent systems. Although each has attracted significant attention, relatively few studies address both challenges at the same time. This paper examines the connection between the notions of consensus and equilibrium in a multi-agent system where multiple interacting sub-populations coexist. We argue that consensus can be seen as an intricate component of intra-population stability, whereas equilibrium can be seen as encoding inter-population stability. We show that smooth fictitious play, a well-known learning model in game theory, can achieve both consensus and convergence to equilibrium in diverse multi-agent settings. Moreover, we show that the consensus formation process plays a crucial role in the seminal thorny problem of equilibrium selection in multi-agent learning.
Two fundamental problems of multi-agent systems are consensus formation [26, 3, 12, 30, 5, 28, 86, 68] and learning in games [48, 33, 66, 58, 57, 70, 71, 13, 55]. The former (sometimes also referred to as cooperative control, flocking, synchronization, and social norms) aims to understand how a decentralized system of agents, which starts off in an initially disordered state, is able to develop shared coordinated beliefs, opinions, or behaviors [7]. The latter (also known as multi-agent learning) examines whether, how and what type of equilibrium arises as the result of the long-run learning processes of individual agents [33]. Each problem has attracted significant interest, yet, these two problems are largely studied in isolation.
Is there a unified way to think about consensus formation and learning in games? We argue that consensus and equilibrium, the fundamental notions of these two fields, can be both understood as stability concepts in a multi-agent system where there co-exist multiple interacting sub-populations (or populations hereafter). Consensus can be seen as an intricate component of intra-population stability that reflects whether each population is internally cohesive, i.e., a population is stable if all the agents belonging to this population achieve consensus and develop the same belief, opinion, or behavior. In contrast, game-theoretic (e.g., Nash) equilibrium can be seen at a more macro level as encoding inter-population stability, i.e., the stability between different populations is achieved by alignment to their population-wise incentives. Namely, a stable system of populations is characterized by the internal cohesion of each population as well as the game-theoretic equilibration between different populations. The merits of this way of thinking are twofold. It presents a non-trivial connection
between consensus and equilibrium, bringing them together under the same umbrella of stability concepts for a system of populations. Moreover, it distinguishes between intra-population stability and inter-population stability so that the emerging phenomena that it can describe are significantly richer, and that it allows for important distinctions that would have otherwise been impossible.
In the rich literature on consensus formation, the idea of building up a connection to game-theoretic equilibration is not entirely new. Classic mathematical models, such as the Degroot model [27] and the Friedkin-Johnsen model [31], showed that consensus of opinions can be reached if individuals form their opinions taking a weighted average of their own opinions and the opinions of others. In seminal work, Bindel et al. [15] showed that such a repeated averaging process is equivalent to a best-response play in the game with a quadratic reward function measuring the distance between individuals' opinions, leading to interest in understanding consensus from the perspective of learning in games [38, 23, 2, 14, 36, 10, 9]. In parallel, Marden et al. [62] demonstrated how the cooperative control problem of consensus can be formulated as better reply with inertia dynamics in potential games, while Young [89] proved the emergence of social norms using evolutionary game theory. These notable advances, however, typically assumed the reward structure to encode an incentive to align individual agents' opinions or behaviors with those of others. Little is formally known about consensus formation in the absence of such a coordinative reward structure, or whether such a coordinative reward structure is a prerequisite for achieving both consensus formation and game-theoretic equilibration.
In the literature on multi-agent learning, the concept of consensus has been less explored compared to the abundant study of convergent or non-convergent learning behaviors in games [13, 79, 66, 81, 77, 6, 57, 16]. We argue that consensus is of crucial importance to multi-agent learning—it ensures not only intra-population stability when there co-exist multiple interacting populations but also complete predictability and computational efficiency in more general settings. Imagine the case that for a system of agents, the average probability  $(\bar{x}_A$  and  $\bar{x}_B)$  of choosing between two strategies  $(A$  and  $B)$  is shown to equilibrate at  $\bar{x}_A = \bar{x}_B = 0.5$ . If consensus is reached, it is completely predictable that every agent plays  $A$  and  $B$  with equal probability 0.5. However, if consensus is not reached, there are many possibilities for agents' behaviors, to name a few—it is possible (i) that exactly half of the agents play  $A$  with probability 1 and the other half plays  $B$  with probability 1, and (ii) that some agents play  $A$  and  $B$  with equal probability 0.5 whereas the rest constantly switches between these two strategies. The unpredictability in agents' behaviors can be problematic for system design, especially in safety-critical scenarios, where a system designer ought to rule out the possibility of getting the system to evolve into some (e.g., unsafe) states [74, 76, 73, 91]. From the perspective of computational efficiency, consensus is usually a desirable property and is embedded in the mean field game literature to simplify the characterization of equilibrium—by exploiting the homogeneity of the population, it suffices to understand the behavior of a single representative agent facing the mean field [55, 54, 51, 71, 42, 87, 1].
Putting these together motivates our central questions:
- Are there natural multi-agent learning models that can achieve the best of both worlds—reaching consensus as well as convergence to equilibrium—in diverse settings?
- How does the consensus formation process affect equilibrium selection in multi-agent learning?
Our Model. We provide an affirmative answer to the first question, and our solution is—smooth fictitious play (SFP) [32]—a well-known belief-based learning model in game theory. In this paper, we consider a network of populations, where each vertex represents a population (or continuum) of agents and each edge denotes that agents from connected populations are paired up to play 2-player subgames. We call this type of game a network population game, $^{1}$  as it has a close connection to the network (or graphical polymatrix) games [52] and the population games [75]. By SFP, each agent forms beliefs about the strategies of agents of the connected populations based on the average
<sup>1</sup>A good survey for network population games and variants thereof is [83]. A prototypical example is the men-women interactions that give rise to a 2-population Battle of the Sexes game, which corresponds to a single-edge network in our case. Similarly, one can think of an ecosystem with a graph of predator-prey interactions; the fact that there is no self-interaction within a population captures that there is no in-species cannibalism. Another example is the multi-army combat setting; combatants of the same army do not face each other.
(a) The initial beliefs are distributed according to  $\mathrm{Beta}(14,6)$  for both populations. The variance of initial beliefs is 0.01 and the mean of initial beliefs is 0.7.
(b) The initial beliefs are distributed according to  $\mathrm{Beta}(280,120)$  for both populations. The variance of initial beliefs is  $\approx 0.0005$  and the mean of initial beliefs is 0.7.
Figure 1: System evolution in 2-population stag hunt games. Over time, the belief distributions become more concentrated (left panel: the predictions of our continuity equations, middle panel: the empirical distributions in agent-based simulations), and the variance of beliefs and the distance (L1 norm) to the equilibrium are both decreasing (right panel). Even though the consensus formation process starts with the same initial mean belief, a small variance of initial beliefs results in the risk dominant equilibrium  $(H, H)$  (subplot (b)), whereas a large variance of initial beliefs results in the payoff dominant equilibrium  $(S, S)$  (subplot (a)). A video for the evolution of these two belief distributions is available at https://sites.google.com/view/shuyue-hu.
plays of those populations, and responds with an aggregated smoothed best response. To allow for a meaningful study of consensus formation, we assume that agents within each population have different beliefs and different behaviors in the initial state. In other words, our model is a heterogeneous belief model [35], unlike the homogeneous belief models extensively studied in the SFP literature [8, 82, 11, 32, 44, 47, 46].
Our Approach & Key Results. To see whether a consensus is reached, one needs a measure of it. A natural approach is to assume the system state to be a probability measure (or distribution) over the set of beliefs (referred to as the belief distribution). This notion of a system state allows us to quantify how much agents reach a consensus by examining the variance of the belief distribution. Intuitively, a value of zero for the variance means that agents have the same belief and have achieved consensus. However, analyzing the evolution of the belief distribution requires new techniques beyond the classic approach of stochastic approximation [8, 11, 35, 44, 47]. We address this challenge by establishing a system of nonlinear partial differential equations—continuity equations<sup>2</sup>—to describe the continuous-time change in the belief distribution (Proposition 1). This approach allows us to analyze the evolution of the variance without deriving an analytical form of the belief distribution. We show that for any network population games, the variance of beliefs decreases to zero at a quadratic rate with time in every population (Theorem 1). This means that even if the 2-player sub-games that agents (of different populations) play are not coordinative but zero-sum, consensus will still be reached. Put differently, a coordinative reward structure is not a prerequisite for achieving consensus.
${}^{2}$  A continuity equation is a PDE commonly encountered in physics,which describes the transport phenomena of some quantity (e.g., mass, energy, momentum and other conserved quantities) in a physical system.
The next step is to examine the convergence to equilibrium. Our finding that the variance of beliefs decreases to zero suggests that the asymptotic behavior of a population that starts with different beliefs can be understood by studying that of a single representative agent. Inspired by this, we temporarily override the assumption of the heterogeneous belief model and assume a single representative agent for each population, resulting in the classic form of network games with each vertex representing a single agent. Under this assumption, we use Lyapunov arguments to prove the convergence of SFP to quantal response equilibria (QRE)<sup>3</sup> in two classes of network games that capture network competition and network coordination. Specifically, we show that for a weighted zero-sum network game, SFP converges to a unique QRE even if the underlying game has many distinct Nash equilibria (Theorem 2). We also show that for an exact potential network game that exhibits a star structure, SFP converges to the set of QRE (Theorem 3). We then use the techniques of asymptotically autonomous dynamical systems to show that the convergence to QRE holds for network population games as well (Theorem 4 and Theorem 5). Therefore, SFP converges to QRE in all weighted zero-sum network (population) games as well as all exact potential star-graph (population) games.
At this point, we have shown that SFP can achieve consensus as well as convergence to equilibrium in a wide range of network population games. Specifically, consensus is always reached, regardless of the specific setting. In the language of stability concepts, by SFP, the intra-population stability is always guaranteed, but the inter-population stability may require additional assumptions about the incentive nature as well as the network topology. This may lead to an error-prone intuition that the consensus formation process is orthogonal to the equilibrium selection process and thus has little effect on it. This is indeed not true. We illustrate this through an example of a 2-population stag hunt game, where different variances of initial beliefs result in qualitatively different long-term system behaviors. As we show in Figure 1, even though the initial mean belief is the same, the consensus formation process starting with a low level of initial disagreement (a small value for initial variance) results in the risk dominant equilibrium  $(H,H)$ , while starting with a high level of initial disagreement (a large value for initial variance) leads to the payoff dominant equilibrium  $(S,S)$ . We provide additional evidence covering all possible initial mean beliefs under the same 2-population stag-hunt game setting in Figure 2 (Section 6). Figure 2 shows that increasing the variance of initial beliefs from 0 to 0.02, 0.05, and 0.1 expands the region of attraction of the payoff dominant equilibrium  $(S,S)$ , allowing a wider range of initial mean beliefs to approach it. Thus, in the case of network coordination, consensus formation plays a crucial role in the seminal thorny problem of equilibrium selection, and a larger initial variance can promote convergence to the equilibrium  $(S,S)$ . In the case of network competition, convergence to a unique QRE is formally guaranteed (Theorem 4); we empirically verify this in 5-population asymmetric matching pennies games with many distinct Nash equilibria (Section 1 of the Appendix).
Other Related Work. SFP is a well-studied model that converges in various classes of games, including most 2-player-2-action games [32, 44, 11, 47], n-player potential game s[82, 45], and zero-sum and identical-interest stochastic games [8, 78, 59]. However, SFP in network (population) games has not been previously explored. Our research extends the findings of Fudenberg and Takahashi [35] (who showed that agents with different beliefs in a 2-population setting will converge to the same belief) to the network settings, but we demonstrate that the variance of initial beliefs plays a crucial role in equilibrium selection, which has not been found in [35]. On the other hand, networked multi-agent learning is a current frontier in AI research [90, 61, 43, 21, 17, 4, 25]. Previous studies have shown the non-convergent behavior of replicator dynamics in (weighted) zero-sum network games [72, 18, 66, 79], the convergence of smooth Q-learning in weighted potential [56] and weighted zero-sum network games [57], and the convergence of fictitious play in zero-sum network games [29]. This paper provides new convergence results for SFP in both network games and network population games, covering a wide range of generic 2-population-2-action settings, star-graph settings that admit an exact potential, and all weighted zero-sum settings. Finally, the evolution of probability distributions over initial conditions or strategies has also received increasing interest [22, 53, 49, 50, 24, 60]. Continuity equations have been used as a tool to show that under Cross learning, agents with different mixed strategies in a population game will not converge to the same mixed strategy [53], and to also describe the more complicated Q-learning dynamics in various settings [49, 50, 24, 60]. This paper formally shows that the probability distribution over
3Quantual response equilibrium is a game theoretic solution concept under bounded rationality [65].
initial conditions can eventually degenerate to a point mass, and leveraging on this, presents a novel technique for proving the convergence of learning dynamics.4
A network population game  $\Gamma = (N, (V, E), (S_i, \omega_i)_{\forall i \in V}, (\mathbf{A}_{ij})_{(i,j) \in E})$  consists of a multi-agent system  $N$  distributed over a connected graph  $(V, E)$ , where  $V = \{1, \dots, n\}$  is the set of vertices each represents a population (continuum) of agents, and  $E$  is the set of pairs,  $(i,j)$ , of population  $i \neq j \in V$ . For each population  $i \in V$ , agents of this population have a finite set  $S_i$  of pure strategies (or actions) with generic elements  $s_i \in S_i$ . Agents may also use mixed strategies (or choice distributions). For an arbitrary agent  $k$  in population  $i$ , its mixed strategy is a vector  $\mathbf{x}_i(k) \in \Delta_i$ , where  $\Delta_i$  is the simplex in  $\mathbb{R}^{|S_i|}$  such that  $\sum_{s_i \in S_i} x_{is_i}(k) = 1$  and  $x_{is_i}(k) \geq 0, \forall s_i \in S_i$ . Each edge  $(i,j) \in E$  defines a series of two-player subgames between populations  $i$  and  $j$ . We denote the payoff matrices for agents of population  $i$  and  $j$  in these two-player subgames by  $\mathbf{A}_{ij} \in \mathbb{R}^{|S_i| \times |S_j|}$  and  $\mathbf{A}_{ji} \in \mathbb{R}^{|S_j| \times |S_i|}$ , respectively. For every time step, each agent chooses a (mixed or pure) strategy and plays that strategy in all two-player subgames with agents from the connected (or neighbor) populations. Let  $\mathbf{x} = (\mathbf{x}_i, \{\mathbf{x}_j\}_{(i,j) \in E})$  be a mixed strategy profile, where  $\mathbf{x}_i$  (or  $\mathbf{x}_j$ ) denotes a generic mixed strategy in population  $i$  (or  $j$ ). Given the mixed strategy profile  $\mathbf{x}$ , the expected payoff of using  $\mathbf{x}_i$  in the game  $\Gamma$  is
$$
r _ {i} (\mathbf {x}) = r _ {i} \left(\mathbf {x} _ {i}, \left\{\mathbf {x} _ {j} \right\} _ {(i, j) \in E}\right) := \sum_ {(i, j) \in E} \mathbf {x} _ {i} ^ {\top} \mathbf {A} _ {i j} \mathbf {x} _ {j}. \tag {1}
$$
Smooth (or stochastic) fictitious play (SFP) is a belief-based model for learning in games. By SFP, agents act as if they are Bayesian such that they assume that the strategies of other agents are drawn from some fixed but unknown distribution. Given a game  $\Gamma$ , consider an arbitrary agent  $k$  in a population  $i \in V$ . Let  $V_{i} = \{j \in V : (i,j) \in E\}$  be the set of neighbor populations. Agent  $k$  maintains a weight  $\kappa_{js_j}^i(k)$  for each opponent strategy  $s_j \in S_j$  of each neighbor population  $j \in V_i$ . For every time step  $t$ , agents refine these weights based on the average play of each neighbor population, i.e., for each opponent strategy  $s_j \in S_j, j \in V_i$ ,
$$
\kappa_ {j s _ {j}} ^ {i} (k, t + 1) = \kappa_ {j s _ {j}} ^ {i} (k, t) + \bar {x} _ {j s _ {j}} (t), \tag {2}
$$
where  $\bar{x}_{js_j}$  is the mean probability of playing strategy  $s_j$  in population  $j$ . For simplicity, we assume the initial sum of weights  $\sum_{s_j \in S_j} \kappa_{js_j}^i(k,0)$  to be the same for every agent in the system and denote this initial sum by  $\lambda$ . Based on the weights, agent  $k$  forms a belief about the neighbor population  $j$  such that each opponent strategy  $s_j$  is played with probability
$$
\mu_ {j s _ {j}} ^ {i} (k) = \frac {\kappa_ {j s _ {j}} ^ {i} (k)}{\sum_ {s _ {j} ^ {\prime} \in S _ {j}} \kappa_ {j s _ {j} ^ {\prime}} ^ {i} (k)}. \tag {3}
$$
Let  $\mu_j^i (k)$  be the vector of beliefs with the  $s_j$ -th element equals  $\mu_{js_j}^i (k)$ . Given a game  $\Gamma$ , agent  $k$ 's expected payoff of using a pure strategy  $s_i\in S_i$  in response to the aggregate beliefs  $\pmb{\mu}_j^i (k),\forall j\in V_i$  about neighbor populations is
$$
u _ {i s _ {i}} (k) = \sum_ {j \in V _ {i}} \mathbf {e} _ {s _ {i}} ^ {\top} \mathbf {A} _ {i j} \boldsymbol {\mu} _ {j} ^ {i} (k), \tag {4}
$$
where  $\mathbf{e}_{s_i}$  is a unit vector with the  $s_i$ -th element equal 1. By convention [33, 34, 44], we suppose that agents' payoffs are perturbed and that agents play smooth best responses maximizing their perturbed
Our approach highlights the consensus property as a valuable tool for establishing the convergence of learning under population settings. For future research that studies learning dynamics under population settings, one can find inspiration in our approach by initially verifying the consensus property and subsequently leveraging this property to establish convergence results.
<sup>5</sup>Observing the average play of other populations is natural in settings with a large number of agents and can be viewed as mean-field observations [88, 37]; for example, the average household income of each state is made publicly available, and people form beliefs about the behaviors of taxi drivers vs non-professional drivers after observing the numerous driving behaviors on the road.
payoffs. Agent  $k$ 's perturbed payoff of using a mixed strategy  $\mathbf{x}_i(k) \in \Delta_i$  is defined as
$$
\pi_ {i} \left(\mathbf {x} _ {i} (k), \left\{\boldsymbol {\mu} _ {j} ^ {i} (k) \right\} _ {j \in V _ {i}}\right) = \sum_ {j \in V _ {i}} \mathbf {x} _ {i} (k) ^ {\top} \mathbf {A} _ {i j} \boldsymbol {\mu} _ {j} ^ {i} (k) + \epsilon v \left(\mathbf {x} _ {i} (k)\right), \tag {5}
$$
where  $\epsilon > 0$ , and the function  $v$  is strictly concave and its gradient becomes arbitrarily large near the boundary of the simplex  $\Delta_i$  [44]. For the function  $v$ , a typical choice is  $v(\mathbf{x}_i(k)) = -\frac{1}{\beta}\sum_{s_i \in S_i} x_{is_i}(k)\ln (x_{is_i}(k))$ . Under this form, the maximization of the perturbed payoff  $\pi_i$  yields a unique choice distribution  $\mathbf{x}_i(k)$  from the logit choice rule [45], that is,
$$
x _ {i s _ {i}} (k) = \frac {\exp \left(\beta u _ {i s _ {i}} (k)\right)}{\sum_ {s _ {i} ^ {\prime} \in S _ {i}} \exp \left(\beta u _ {i s _ {i} ^ {\prime}} (k)\right)}, \tag {6}
$$
where  $\beta$  is the temperature (or the degree of rationality). For clarity of our presentation, we choose to focus on the logit choice rule in this paper; however, all our results readily generalize to any function  $v$  satisfying the above two standard assumptions. In the following, we usually drop the time index  $t$  and agent index  $k$  in the bracket (depending on the context) for notational convenience.
3 Belief Dynamics in Network Population Games
Observe that for an arbitrary agent  $k$  of population  $i$ , its belief  $\mu_j^i (k)$  about population  $j$  is in the simplex  $\Delta_{j} = \{\pmb{\mu}_{j}^{i}(k)\in \mathbb{R}^{|S_{j}|}|\sum_{s_{j}\in S_{j}}\mu_{js_{j}}^{i}(k) = 1,\mu_{js_{j}}^{i}(k)\geq 0,\forall s_{j}\in S_{j}\}$ . We assume that the system state is characterized by a Borel probability measure  $P$  defined on the state space  $\Delta = \prod_{i\in V}\Delta_{i}$ . Given  $\pmb{\mu}_i\in \Delta_i$ , we write the marginal probability density function as  $p(\pmb{\mu}_i,t)$ . Note that  $p(\pmb{\mu}_i,t)$  represents the density of agents having the belief  $\pmb{\mu}_i$  about population  $i$  throughout the system; it does not represent the density of agents of population  $i$  that have the belief  $\pmb{\mu}_i$ , nor the density of agents having the belief  $\pmb{\mu}_i$  about population  $i$  in a certain population. Define  $\pmb{\mu} = \{\pmb{\mu}_i\}_{i\in V}\in \Delta$ . Since agents maintain separate beliefs about different neighbor populations, the joint probability density function  $p(\pmb{\mu},t)$  can be factorized, i.e.,  $p(\pmb{\mu},t) = \prod_{i\in V}p(\pmb{\mu}_i,t)$ . We make the following assumption for the initial marginal density functions.