SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 1. At time  $t = 0$ , for each population  $i \in V$ , the marginal density function  $p(\mu_i, t)$  is continuously differentiable and has zero mass at the boundary of the simplex  $\Delta_i$ .
This assumption excludes the setting that a population of agents starts off with the same belief as well as the setting that agents start off with extreme beliefs. Under this mild condition, we derive a partial differential equation to describe the evolution of the belief distribution.
Proposition 1. The continuous-time dynamic of the marginal density function  $p(\mu_i, t)$  for each population  $i \in V$  is described by a partial differential equation
$$
- \frac {\partial p \left(\boldsymbol {\mu} _ {i} , t\right)}{\partial t} = \nabla \cdot \left(p \left(\boldsymbol {\mu} _ {i}, t\right) \frac {\bar {\mathbf {x}} _ {i} - \boldsymbol {\mu} _ {i}}{\lambda + t + 1}\right) \tag {7}
$$
where  $\nabla \cdot$  is the divergence operator and  $\bar{\mathbf{x}}_i$  is the mean mixed strategy with each  $s_i$ -th element
$$
\bar {x} _ {i s _ {i}} = \int_ {\prod_ {j \in V _ {i}} \Delta_ {j}} \frac {\exp \left(\beta u _ {i s _ {i}}\right)}{\sum_ {s _ {i} ^ {\prime} \in S _ {i}} \exp \left(\beta u _ {i s _ {i} ^ {\prime}}\right)} \prod_ {j \in V _ {i}} p \left(\boldsymbol {\mu} _ {j}, t\right) \left(\prod_ {j \in V _ {i}} d \boldsymbol {\mu} _ {j}\right) \tag {8}
$$
where  $u_{is_i} = \sum_{j\in V_i}\mathbf{e}_{s_i}^\top \mathbf{A}_{ij}\pmb {\mu}_j$
The partial differential equation in Equation 7 is also known as a continuity equation in the study of transport phenomena (e.g., of mass or energy) in a physical system. We remark that continuity equations, tracking the evolution of a probability distribution, do not allow for general solutions. However, continuity equations provide a tractable way to examine the evolution of the moments of the probability distribution, without obtaining an analytical form of that distribution. The analytical form for the dynamic of the variance of beliefs (the second moment of the belief distribution) is given as follows.
6 Alternatively, SFP can be defined through a truly stochastic perturbation of payoffs [32].
Theorem 1. The dynamic of the variance of beliefs  $\mu_{i}$  about each population  $i\in V$  is governed by an ordinary differential equation such that for each strategy  $s_i$
$$
\frac {d \operatorname {V a r} \left(\mu_ {i s _ {i}}\right)}{d t} = - \frac {2 \operatorname {V a r} \left(\mu_ {i s _ {i}}\right)}{\lambda + t + 1}. \tag {9}
$$
At given time  $t$ ,  $\operatorname{Var}(\mu_{is_i}) = \left(\frac{\lambda + 1}{\lambda + t + 1}\right)^2 \sigma^2(\mu_{is_i})$ , where  $\sigma^2(\mu_{is_i})$  is the initial variance. Thus, the variance  $\operatorname{Var}(\mu_{is_i})$  decays to zero at a quadratic rate with time.
Note that the above theorem makes no assumption about the 2-player subgames agents play, the network structure of populations, or even the initial belief distribution. Therefore, it shows that for any network of populations, the variance of beliefs about each population always asymptotically tends to zero. As an immediate consequence of this theorem, the asymptotic distribution of the beliefs is as follows:
Corollary 1. The probability density function  $p(\mu_i, t)$  for every population  $i \in V$  will eventually evolve into a Dirac delta function, implying that in the long run agents reach a consensus and play the same mixed strategy within each population.
The intuition of Theorem 1 and Crollary 1 is that under SFP, agents revise their beliefs about other populations based on the average play of those populations; consequently, the observations are the same within an agent population, and thus it is only a matter of time that the agent population will develop the same belief and play the same strategy. Once consensus is reached, agents within a population will not develop different beliefs afterwards, as ensured by Theorem 1. In this sense, the agent population is internally cohesive and achieves intra-population stability.
However, it does not mean that different agent populations will eventually play the same strategy. This is because agents' strategies are the smooth best responses to their aggregated beliefs about their own set of neighbor populations (which generally vary depending on the network topology). This naturally raises the question of how different agent populations align with their population-wise incentives (which can be of a competitive nature). This also sets our paper apart from previous work that studies game-theoretic equilibration in the context of consensus formation [15, 62, 38, 23, 2, 14, 36, 10, 9], in which consensus is achieved as the result of agents' incentives to align their opinions or behaaviors to those of others (i.e., the incentives are of a coordinative nature).
Before diving into the convergence results, we take a moment to analyze the mean belief (first moment) dynamic. Intuitively, the mean belief represents the average system state. The following proposition provides an analytical form of the mean belief dynamic.
Proposition 2. The mean belief dynamic for each population  $i \in V$  is governed by an ordinary differential equation such that for each strategy  $s_i$ ,
$$
\frac {d \bar {\mu} _ {i s _ {i}}}{d t} \approx \frac {f _ {s _ {i}} \left(\left\{\boldsymbol {\mu} _ {j} \right\} _ {j \in V _ {i}}\right) - \bar {\mu} _ {i s _ {i}}}{\lambda + t + 1} + \frac {\sum_ {j \in V _ {i}} \sum_ {s _ {j} \in S _ {j}} \frac {\partial^ {2} f _ {s _ {i}} \left(\left\{\boldsymbol {\mu} _ {j} \right\} _ {j \in V _ {i}}\right)}{\left(\partial \mu_ {j s _ {j}}\right) ^ {2}} \operatorname {V a r} \left(\mu_ {j s _ {j}}\right)}{2 (\lambda + t + 1)}, \tag {10}
$$
where  $f_{s_i}(\{\mu\}_{j \in V_i})$  is the logit choice rule (Equation 6) applied to strategy  $s_i \in S_i$ , and  $\text{Var}(\mu_{js_j})$  is the variance of belief  $\mu_{js_j}$  in the entire system.
We remark that the mean belief dynamic, to be exact, should be under the joint influence of infinitely many moments of the belief distribution. However, the higher the order of the moments, the less of the influence of the moments, and by convention, the effects of the third and higher moments can be assumed negligible [39, 41, 64].<sup>7</sup> Thus, in this proposition, the mean belief dynamic depends on the current mean belief and also on the current variance of beliefs.
Recall that Theorem 1 shows that the variance of beliefs will eventually decrease to zero. As a result, the second term of the mean belief dynamic (Equation 10) will asymptotically tend to zero. This suggests that the asymptotic mean belief dynamic of a population that starts with different beliefs in the initial state might be understood by studying the belief dynamic of a single representative agent.
This is formally known as the moment closure approximation [39, 41, 64], which is a common method used to estimate moments of population models. As we detail in the appendix, the effects of the higher moments are counter-balanced by their coefficients as a result of Taylor expansion; moreover, the beliefs  $\mu_{is_i},\forall i\in V,\forall s_i\in S_i$  are between 0 and 1, which also lead to small values of higher moments.
Intuitively, if each population can be represented by a single agent, then the concerned network population games will effectively degenerate into the classic form of network games with each vertex representing a single agent. Inspired by this, in the next section, we temporarily focus on SFP in the classic network games.
4 Convergence of Smooth Fictitious Play in Classic Network Games
In a classic network game (or network game for short), the only difference with respect to the network population games is that now each vertex represents a single agent. We derive the belief dynamic of each vertex (or agent) in the following proposition.
Proposition 3. In a network game, the belief dynamic of each agent is governed by an ordinary differential equation such that for each agent  $i$  and each strategy  $s_i$ ,
$$
\frac {d \mu_ {i s _ {i}}}{d t} = \frac {f _ {s _ {i}} \left(\left\{\mu_ {j} \right\} _ {j \in V _ {i}}\right) - \mu_ {i s _ {i}}}{\lambda + t + 1} = \frac {x _ {i s _ {i}} - \mu_ {i s _ {i}}}{\lambda + t + 1}. \tag {11}
$$
It follows from this proposition that the equilibrium state should satisfy the following property.
Proposition 4. In a network game, the equilibrium state should satisfy that  $\mathbf{x}_i^* = \boldsymbol{\mu}_i^*$  for every strategy  $s_i \in S_i$  and every population  $i \in V$ . Thus, the equilibrium state corresponds to the solution to the system of equations
$$
x _ {i s _ {i}} ^ {*} = \frac {\exp \left(\beta \sum_ {j \in V _ {i}} \mathbf {e} _ {s _ {i}} ^ {\top} \mathbf {A} _ {i j} \mathbf {x} _ {j} ^ {*}\right)}{\sum_ {s _ {i} ^ {\prime} \in S _ {i}} \exp \left(\beta \sum_ {j \in V _ {i}} \mathbf {e} _ {s _ {i} ^ {\prime}} ^ {\top} \mathbf {A} _ {i j} \mathbf {x} _ {j} ^ {*}\right)}, \quad \forall s _ {i} \in S _ {i}, \forall i \in V. \tag {12}
$$
Such an equilibrium state coincides with the Quantal Response Equilibria [65] of the network game.
By QRE, in this paper, we refer to their canonical form also referred to as logit QRE in the literature [40]. We remark that QRE [65] is a game theoretic solution concept under bounded rationality, and is an extension of the standard Nash equilibrium which allows for errors in choice.
Weighted zero-sum network games capture the case of network competition, and are the generalization of 2-player zero-sum games to the network settings [19]. A network game is weighted zero-sum if there exist positive constants (or weights)  $\omega_{1},\ldots ,\omega_{n}$  such that
$$
\sum_ {i \in V} \omega_ {i} r _ {i} (\mathbf {x}) = \sum_ {(i, j) \in E} \left(\omega_ {i} \mathbf {x} _ {i} ^ {\top} \mathbf {A} _ {i j} \mathbf {x} _ {j} + \omega_ {j} \mathbf {x} _ {j} ^ {\top} \mathbf {A} _ {j i} \mathbf {x} _ {i}\right) = 0, \quad \forall \mathbf {x} \in \prod_ {i \in V} \Delta_ {i}, \tag {13}
$$
where  $r_i(\mathbf{x})$  is the expected payoff of using a mixed strategy  $\mathbf{x}_i$  given the mixed strategy profile  $\mathbf{x} = (\mathbf{x}_i, \{\mathbf{x}_j\}_{(i,j) \in E})$ . If subgames along every edge  $(i,j) \in E$  are 2-player zero-sum, i.e.,  $\mathbf{x}_i^\top \mathbf{A}_{ij} \mathbf{x}_j + \mathbf{x}_j^\top \mathbf{A}_{ji} \mathbf{x}_i = 0$ , then the network game is zero-sum; however, the converse is not generally true. As opposed to 2-player zero-sum games that admit a unique value at equilibrium [85], the Nash equilibrium payoffs in weighted zero-sum network games need not be unique, and it generally allows for infinitely many Nash equilibria. In the following theorem, we show that regardless of the number of Nash equilibria in a network game, the belief dynamic is guaranteed to converge to a unique QRE.
Theorem 2. For any weighted zero-sum network game, the belief dynamic converge to a unique QRE which is globally asymptotically stable.
Exact potential network games capture the case of network coordination. A network game admits an exact potential if for each edge  $(i,j) \in E$ , the payoff matrices of the two-player subgame satisfy  $\mathbf{A}_{ij} = \mathbf{A}_{ji}^{\top}$  [21, 20]. Pure or mixed Nash equilibria in network coordination are complex. As reported in recent works [20, 17, 4], finding a pure Nash equilibrium is PLS-complete. Hence, learning in the general case of network coordination is difficult and generally requires some conditions [67, 69]. In the following theorem, we show the convergence result in some cases of network coordination.
Theorem 3. For any exact potential network game that exhibits a star structure, each orbit of the belief dynamic converges to the set of QRE.
We remark that we employ the Lyapunov approach (i.e., establishing the existence of the Lyapunov function) to prove the above convergence results. As a result, the convergence to equilibrium is equivalent to the network of agents being stable in the sense of Lyapunov.
Basins of Attraction of  $(S, S)$  and  $(H, H)$  under Different Variances of Initial Beliefs
Figure 2: Initial disagreement helps select the payoff dominant equilibrium  $(S, S)$  (yellow: the equilibrium  $(S, S)$ , blue: the equilibrium  $(H, H)$ ). As the variance of initial beliefs (denoted by  $\sigma^2(\mu_{1H})$ ) increases from the left to right panel, a wider range of initial mean beliefs will approximately reach the equilibrium  $(S, S)$  in the limit. For simplicity, we set  $\sigma^2(\mu_{1H}) = \sigma^2(\mu_{2H})$ . The black line on the diagonal is there for ease of comparison across different panels.
5 Convergence of Smooth Fictitious Play in Network Population Games
We shall now formalize that the convergence results in network games can be seamlessly carried over to network population games with different beliefs in the initial state, and establish the interpopulation stability in network population games. Recall that the mean belief dynamic (Proposition 2) depends on the current mean belief as well as the current variance of beliefs, with the variance decreasing over time (Theorem 1). Leveraging on the definition of asymptotically autonomous dynamical systems [63], we establish the following lemma.
Lemma 1. In a network population game, for every population  $i \in V$ , the mean belief dynamic (Equation 10) is asymptotically autonomous with the limit equation being  $\frac{d\mu_i}{dt} = \mathbf{x}_i - \boldsymbol{\mu}_i$ , which after time-reparameterization is equivalent to the belief dynamic in a network game (Equation 11).
Thieme [84] provided the following seminal result that formally connects the asymptotic behavior of an asymptotically autonomous dynamical system to that of its associated limit equation.
Lemma 2 (Thieme [84] Theorem 4.2). Given a metric space  $(X,d)$ , let the solution flows of an asymptotically autonomous system and its limit equation be  $\phi$  and  $\Theta$ , respectively. Assume that the equilibria of  $\Theta$  are isolated compact  $\Theta$ -invariant subsets of  $X$ . The  $\omega$ - $\Theta$ -limit set of any precompact  $\Theta$ -orbit contains a  $\Theta$ -equilibrium. The point  $(s,x), s \geq t_0, x \in X$ , have a pre-compact  $\phi$ -orbit. Then the following alternative holds: 1)  $\phi(t,s,x) \to e, t \to \infty$ , for some  $\Theta$ -equilibrium  $e$ , and 2) the  $\omega$ - $\phi$ -limit set of  $(s,x)$  contains finitely many  $\Theta$ -equilibria which are chained to each other in a cyclic way.
Combining the above lemmas and the convergence results under the setting of classic network games, the convergence results of SFP in network population games are readily deducible:
Theorem 4. For any weighted zero-sum network population game, the belief (as well as the choice distribution) of every individual converges to a unique QRE.
Theorem 5. For any exact potential network population game that exhibits a star structure, the belief (as well as the choice distribution) of every individual converges to the set of QRE.
Note that under the 2-population settings, there is only one possible graph structure—one edge connecting the two populations. Hofbauer and Hopkins [44] have shown that SFP converges in all 2-player weighted potential games, and observed that almost all 2-player-2-action games are either 2-player weighted potential games or 2-player weighted zero-sum games. Therefore, combining their results and our aforementioned results, SFP converges to the set of QRE under almost all the 2-population-2-action settings (including all the 2-population-2-action weighted potential settings and all the 2-population-2-action weighted zero-sum settings).
6 Experiments: Equilibrium Selection in Two-Population Stag Hunt Games
In a two-population stag hunt game, agents from two populations are paired up to choose between two actions  $H$  (hare) and  $S$  (stag). If both players hunt a hare, they both receive a reward of 1. If
both players hunt a stag, they both receive a reward of 4. If one hunts a hare and the other hunts a stag, the former receives a reward of 2 whereas the latter receives nothing. There are two pure strategy Nash equilibria in each subgame: the risk dominant equilibrium  $(H, H)$  and the payoff dominant equilibrium  $(S, S)$ , which naturally raises the problem of equilibrium selection.
Figure 1 showed that given the same initial mean belief, changing the variances of initial beliefs can result in different limit behaviors. In Figure 2, we systematically studied the effect of the variance of initial beliefs by visualizing how it affects the regions of attraction to different equilibria. To be more specific, we numerically solved the mean belief dynamics for a wide range of initial mean beliefs, given different variances of initial beliefs, and colored the regions of attraction. We observed in Figure 2 that as the variance of initial beliefs increases (from the left to right panel), a wider range of initial mean beliefs results in the convergence to the QRE that approximates the payoff dominant equilibrium  $(S,S)$ . Hence, consensus is inevitable, but the initial disagreement still provides an approach to equilibrium selection and helps select the highly desirable equilibrium.
Note that in the case of network competition, the convergence to a unique QRE is formally guaranteed; we empirically verified this through a 5-population asymmetric matching pennies game presented in Appendix A.
This paper presents a formal treatment for SFP in network population games where each population of agents start off with different beliefs and strategies. Representing the system state with a distribution over beliefs, we show that consensus is always reached, and that agents eventually pointwise converge to the same belief, regardless of specific settings. In addition, we establish the convergence of SFP to Quantal Response Equilibria in general competitive network population games as well as exact potential network population games with star structure. Empirically, we show that although the initial belief heterogeneity vanishes in the limit, it plays a crucial role in equilibrium selection and helps select highly desirable equilibria.
Shuyue Hu would like to thank Ms. Die Hu for her assistance in generating Figure 1 and the corresponding video presented in this paper. This research is supported in part by the National Research Foundation, Singapore and DSO National Laboratories under its AI Singapore Program (AISG Award No: AISG2-RP-2020-016), NRF2019-NRFANR095 ALIAS grant, NRF 2018 Fellowship NRF-NRFF2018-07, grant PIESGP-AI-2020-01, AME Programmatic Fund (Grant No.A20H6b0151) from A\*STAR and Provost's Chair Professorship grant RGEPPV2101.
[1] Andrea Angiuli, Nils Detering, Jean-Pierre Fouque, Mathieu Lauriere, and Jimin Lin. Reinforcement learning algorithm for mixed mean field control games. arXiv preprint arXiv:2205.02330, 2022.
[2] Vincenzo Auletta, Ioannis Caragiannis, Diodato Ferraioli, Clemente Galdi, and Giuseppe Persiano. Generalized discrete preference games. In *IJCAI*, volume 16, pages 53–59, 2016.
[3] Vincenzo Auletta, Angelo Fanelli, and Diodato Ferraioli. Consensus in opinion formation processes in fully evolving environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6022-6029, 2019.
[4] Yakov Babichenko and Aviad Rubinstein. Settling the complexity of nash equilibrium in congestion games. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 1426-1437, 2021.
[5] Rafael Bailo, Mattia Bongini, José A Carrillo, and Dante Kalise. Optimal consensus control of the cucker-smale model. IFAC-PapersOnLine, 51(13):1-6, 2018.
[6] D Balduzzi, S Racaniere, J Martens, J Foerster, K Tuyls, and T Graepel. The mechanics of n-player differentiable games. In ICML, volume 80, pages 363-372. JMLR.org, 2018.
[25] Aleksander Czechowski and Georgios Piliouras. Poincar\`{e}-bendixson limit sets in multiagent learning. 2022.
[26] Abir De, Isabel Valera, Niloy Ganguly, Sourangshu Bhattacharya, and Manuel Gomez Rodriguez. Learning and forecasting opinion dynamics in social networks. Advances in neural information processing systems, 29, 2016.
[27] Morris H DeGroot. Reaching a consensus. Journal of the American Statistical association, 69(345):118-121, 1974.
[28] Yucheng Dong, Quanbo Zha, Hengjie Zhang, Gang Kou, Hamido Fujita, Francisco Chiclana, and Enrique Herrera-Viedma. Consensus reaching in social network group decision making: Research paradigms and challenges. Knowledge-Based Systems, 162:3-13, 2018.
[29] Christian Ewerhart and Kremena Valkanova. Fictitious play in networks. Games and Economic Behavior, 123:182-206, 2020.
[30] Dimitris Fotakis, Dimitris Palyvos-Giannas, and Stratis Skoulakis. Opinion dynamics with local interactions. In *IJCAI*, pages 279–285, 2016.
[31] Noah E Friedkin and Eugene C Johnsen. Social influence and opinions. Journal of Mathematical Sociology, 15(3-4):193–206, 1990.
[32] Drew Fudenberg and David M Kreps. Learning mixed equilibria. Games and economic behavior, 5(3):320-367, 1993.
[33] Drew Fudenberg and David K Levine. The theory of learning in games, volume 2. MIT press, 1998.
[34] Drew Fudenberg and David K Levine. Conditional universal consistency. Games and Economic Behavior, 29(1-2):104-130, 1999.
[35] Drew Fudenberg and Satoru Takahashi. Heterogeneous beliefs and local information in stochastic fictitious play. Games and Economic Behavior, 71(1):100-120, 2011.
[36] Jason Gaitonde, Jon Kleinberg, and Éva Tardos. Polarization in geometric opinion dynamics. In Proceedings of the 22nd ACM Conference on Economics and Computation, pages 499-519, 2021.
[37] Sriram Ganapathi Subramanian, Pascal Poupart, Matthew E Taylor, and Nidhi Hegde. Multi type mean field reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 411-419, 2020.
[38] Javad Ghaderi and Rayadurgam Srikant. Opinion dynamics in social networks with stubborn agents: Equilibrium and convergence rate. Automatica, 50(12):3209-3215, 2014.
[39] Colin S Gillespie. Moment-closure approximations for mass-action models. IET systems biology, 3(1):52-58, 2009.
[40] Jacob K Goeree, Charles A Holt, and Thomas R Palfrey. Quantal response equilibrium. In *Quantal Response Equilibrium*. Princeton University Press, 2016.
[41] Leo A Goodman. Population growth of the sexes. Biometrics, 9(2):212-225, 1953.
[42] Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. Learning mean-field games. In Advances in Neural Information Processing Systems, pages 4967-4977, 2019.
[43] Shubham Gupta, Rishi Hazra, and Ambedkar Dukkipati. Networked multi-agent reinforcement learning with emergent communication. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 1858-1860, 2020.
[44] Josef Hofbauer and Ed Hopkins. Learning in perturbed asymmetric games. Games and Economic Behavior, 52(1):133-152, 2005.