SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[80] Gordon Dennis Smith. Numerical solution of partial differential equations: finite difference methods. Clarendon Press, 1985.
[81] Sriram Srinivasan, Marc Lanctot, Vinicius Zambaldi, Julien Pérolat, Karl Tuyls, Rémi Munos, and Michael Bowling. Actor-critic policy optimization in partially observable multiagent environments. Advances in neural information processing systems, 31, 2018.
[82] Brian Swenson and H Vincent Poor. Smooth fictitious play in  $n \times 2$  potential games. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pages 1739-1743. IEEE, 2019.
[83] György Szabó and Gabor Fath. Evolutionary games on graphs. Physics reports, 446(4-6):97-216, 2007.
[84] Horst R Thieme. Convergence results and a poincaré-bendixson trichotomy for asymptotically autonomous differential equations. Journal of mathematical biology, 30(7):755-763, 1992.
[85] J v. Neumann. Zur theorie der gesellschaftsspiele. Mathematische annalen, 100(1):295-320, 1928.
[86] Tushar Vaidya, Carlos Murguia, and Georgios Piliouras. Learning agents in financial markets: Consensus dynamics on volatility. In AAMAS, pages 2106-2108, 2018.
[87] Qiaomin Xie, Zhuoran Yang, Zhaoran Wang, and Andreea Minca. Provable fictitious play for general mean-field games. arXiv preprint arXiv:2010.04211, 2020.
[88] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5567-5576, 2018.
[89] H Peyton Young. The evolution of social norms. economics, 7(1):359-387, 2015.
[90] Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning, pages 5872-5881. PMLR, 2018.
[91] Andrey Zhitnikov and Vadim Indelman. Simplified risk-aware decision making with belief-dependent rewards in partially observable domains. Artificial Intelligence, 312:103775, 2022.
A Experiments on Network Competition
We have shown in Theorem 5 (of the main paper) that SFP converges to a unique QRE in any weighted zero-sum network population game even if there are multiple Nash equilibria underlying that game. In the following, we corroborate this by providing empirical evidence in agent-based simulations with different belief initialization.
Game Description. Consider a five-population asymmetric matching pennies game [56], where the network structure is a line (depicted in Figure 1). Each agent has two actions  $\{H, T\}$ . Agents in populations 1 and 5 do not learn; they always play strategies  $H$  and  $T$ , respectively. For agents in populations 2 to 4, they receive  $+1$  if they match the strategy of the opponent in the next population, and receive  $-1$  if they mismatch. On the contrary, they receive  $+1$  if they mismatch the strategy of the opponent in the previous population, and receive  $-1$  if they match. Hence, this game has infinitely many Nash equilibria of the form: agents in populations 2 and 4 play strategy  $T$ , whereas agents in population 3 are indifferent between strategies  $H$  and  $T$ .
Experimental Setups. In this game, agents in each population form two beliefs (one for the previous population and one for the next population). We are mainly interested in the strategies of population 3, as the Nash equilibria differ in the strategies in population 3. Thus, we let the initial beliefs about populations 1, 3 and 5 remain unchanged across different cases, and vary population 3's initial beliefs about populations 2 and 4. The initial beliefs about populations 1, 3 and 5, denoted by  $\mu_{1H}$ ,  $\mu_{3H}$  and  $\mu_{5H}$ , are distributed according to the distributions Beta(20, 10), Beta(6, 4), and Beta(10, 5), respectively. The initial beliefs about populations are given in the legends of Figure 2. In all cases, the initial sum of weights  $\lambda = 10$  and the temperature  $\beta = 10$ . Note that  $\mu_{iT} = 1 - \mu_{iH}$  for all populations  $i = 1, 2, 3, 4, 5$ . We run 100 simulation runs for each initialization, and each simulation run consists of 1,000 agents in each population.
Results. As shown in Figure 4, given differential initialization of beliefs, agents in population 3 converge to the same equilibrium where they all take strategy  $H$  with probability 0.5. Therefore, even when the underlying zero-sum game has many Nash equilibria, SFP with different initial belief heterogeneity selects a unique equilibrium, addressing the problem of equilibrium selection.
Figure 3: Asymmetric Matching Pennies.
Figure 4: With different belief initialization, SFP selects a unique equilibrium where all agents in population 3 play strategy  $H$  with probability 0.5. The thin lines represent the mean mixed strategy (the choice probability of  $H$ ) and the shaded areas represent the variance of the mixed strategies in the population. In the legends,  $B$  denotes Beta distribution; the two Beta distributions correspond to the initial beliefs about the neighbor populations 2 and 4, respectively.
It follows from Equation 2 and Equation 3 of the main paper that the change in  $\mu_j^i (k,t)$  between two discrete time steps is
$$
\boldsymbol {\mu} _ {j} ^ {i} (k, t + 1) = \boldsymbol {\mu} _ {j} ^ {i} (k, t) + \frac {\bar {\mathbf {x}} _ {j} (t) - \boldsymbol {\mu} _ {j} ^ {i} (k , t)}{\lambda + t + 1}. \tag {14}
$$
Lemma 3. Under Assumption 1 of the main paper, for an arbitrary agent  $k$  in population  $i$ , its belief  $\mu_j^i (k,t)$  about a neighbor population  $j$  will never reach the extreme belief, i.e., the probability density for the boundary of the simplex  $\Delta_{i}$  will remain zero.
Proof. Assumption 1 ensures that  $\bar{\mathbf{x}}_j(0)$  is in the interior of the simplex  $\Delta_j$ . Moreover, the logit choice function (Equation 5 in the main paper) also ensures that  $\bar{\mathbf{x}}_j(t)$  stays in the interior of  $\Delta_j$  afterwards for a finite temperature  $\beta$ . Hence, from Equation 14, one can see that  $\mu_j^i(k, t)$  for every time step  $t$  will stay in the interior of  $\Delta_j$ .
In the following, for notation convenience, we sometimes drop the agent index  $k$  and the time index  $t$  depending on the context. Consider a population  $i$ . We rewrite the change in the beliefs about this population as follows.
$$
\boldsymbol {\mu} _ {i} (t + 1) = \boldsymbol {\mu} _ {i} (t) + \frac {\bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t)}{\lambda + t + 1}. \tag {15}
$$
Suppose that the amount of time that passes between two successive time steps is  $\delta \in (0,1]$ . We rewrite the above equation as
$$
\boldsymbol {\mu} _ {i} (t + \delta) = \boldsymbol {\mu} _ {i} (t) + \delta \frac {\bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t)}{\lambda + t + 1}. \tag {16}
$$
Next, we consider a test function  $\theta (\mu_i)$ . Define
$$
Y = \frac {\mathbb {E} [ \theta (\boldsymbol {\mu} _ {i} (t + \delta)) ] - \mathbb {E} [ \theta (\boldsymbol {\mu} _ {i} (t)) ]}{\delta}. \tag {17}
$$
Applying Taylor series for  $\theta (\pmb {\mu}_i(t + \delta))$  at  $\pmb{\mu}_i(t)$  , we obtain
$$
\begin{array}{l} \theta (\boldsymbol {\mu} _ {i} (t + \delta)) = \theta (\boldsymbol {\mu} _ {i} (t)) + \frac {\delta}{\lambda + t + 1} \partial_ {\boldsymbol {\mu} _ {i}} \theta (\boldsymbol {\mu} _ {i}) [ \bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t) ] \\ + \frac {\delta^ {2}}{2 (\lambda + t + 1) ^ {2}} \left[ \bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t) \right] ^ {\top} \mathbf {H} \theta (\boldsymbol {\mu} _ {i}) \left[ \bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t) \right] \\ + o \left(\left[ \delta \frac {\bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t)}{\lambda + t + 1} \right] ^ {2}\right) \tag {18} \\ \end{array}
$$
where  $\mathbf{H}$  denotes the Hessian matrix. Hence, the expectation  $\mathbb{E}[\theta (\pmb {\mu}_i(t + \delta))]$  is
$$
\begin{array}{l} \mathbb {E} [ \theta (\boldsymbol {\mu} _ {i} (t + \delta)) ] = \mathbb {E} [ \theta (\boldsymbol {\mu} _ {i} (t)) ] + \frac {\delta}{\lambda + t + 1} \mathbb {E} [ \partial_ {\boldsymbol {\mu} _ {i}} \theta (\boldsymbol {\mu} _ {i} (t)) (\bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t)) ] \\ + \frac {\delta^ {2}}{2 (\lambda + t + 1) ^ {2}} \mathbb {E} \left[ \left[ \bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t) \right] ^ {\top} \mathbf {H} \theta \left(\boldsymbol {\mu} _ {i}\right) \left[ \bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t) \right] \right] \\ + \frac {\delta^ {2}}{2 (\lambda + t + 1) ^ {2}} \mathbb {E} \left[ o \left(\left[ \bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t) \right] ^ {2}\right) \right] \tag {19} \\ \end{array}
$$
Moving the term  $\mathbb{E}[\theta (\mu_i(t))]$  to the left hand side and dividing both sides by  $\delta$ , we recover the quantity  $Y$ , i.e.,
$$
\begin{array}{l} Y = \frac {1}{\lambda + t + 1} \mathbb {E} \left[ \partial_ {\boldsymbol {\mu} _ {i}} \theta \left(\boldsymbol {\mu} _ {i} (t)\right) \left(\bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t)\right) \right] \\ + \frac {\delta}{2 (\lambda + t + 1) ^ {2}} \mathbb {E} \left[ \left[ \bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t) \right] ^ {\top} \mathbf {H} \theta \left(\boldsymbol {\mu} _ {i} (t)\right) \left[ \bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t) \right] + o \left(\left(\bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t)\right) ^ {2}\right) \right] \tag {20} \\ \end{array}
$$
Taking the limit of  $Y$  with  $\delta \to 0$ , the contribution of the second term on the right hand side vanishes, yielding
$$
\begin{array}{l} \lim  _ {\delta \rightarrow 0} Y = \frac {1}{\lambda + t + 1} \mathbb {E} \left[ \partial_ {\boldsymbol {\mu} _ {i}} \theta \left(\boldsymbol {\mu} _ {i} (t)\right)\left(\bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t)\right)\right] (21) \\ = \frac {1}{\lambda + t + 1} \int p \left(\boldsymbol {\mu} _ {i} (t), t\right) \left[ \partial_ {\boldsymbol {\mu} _ {i}} \theta \left(\boldsymbol {\mu} _ {i} (t)\right) \left(\bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t)\right) \right] d \boldsymbol {\mu} _ {i} (t). (22) \\ \end{array}
$$
Apply integration by parts. We obtain
$$
\lim  _ {\delta \rightarrow 0} Y = 0 - \frac {1}{\lambda + t + 1} \int \theta (\boldsymbol {\mu} _ {i} (t)) \nabla \cdot [ p (\boldsymbol {\mu} _ {i} (t), t) (\bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t)) ] d \boldsymbol {\mu} _ {i} (t) \tag {23}
$$
where we have leveraged that the probability mass  $p(\mu_i, t)$  at the boundary  $\partial \Delta_i$  remains zero as a result of Lemma 1. On the other hand, according to the definition of  $Y$ ,
$$
\lim  _ {\delta \rightarrow 0} Y = \lim  _ {\delta \rightarrow 0} \int \theta (\boldsymbol {\mu} _ {i} (t)) \frac {p (\boldsymbol {\mu} _ {i} , t + \delta) - p (\boldsymbol {\mu} _ {i} , t)}{\delta} d \boldsymbol {\mu} _ {i} = \int \theta (\boldsymbol {\mu} _ {i} (t)) \partial_ {t} p (\boldsymbol {\mu} _ {i}, t) d \boldsymbol {\mu} _ {i}. \tag {24}
$$
Therefore, we have the equality
$$
\int \theta (\boldsymbol {\mu} _ {i} (t)) \partial_ {t} p (\boldsymbol {\mu} _ {i}, t) d \boldsymbol {\mu} _ {i} = - \frac {1}{\lambda + t + 1} \int \theta (\boldsymbol {\mu} _ {i} (t)) \nabla \cdot [ p (\boldsymbol {\mu} _ {i} (t), t) (\bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t)) ] d \boldsymbol {\mu} _ {i} (t). \tag {25}
$$
As  $\theta$  is a test function, this leads to
$$
\partial_ {t} p \left(\boldsymbol {\mu} _ {i}, t\right) = - \frac {1}{\lambda + t + 1} \nabla \cdot \left[ p \left(\boldsymbol {\mu} _ {i} (t), t\right) \left(\bar {\mathbf {x}} _ {i} (t) - \boldsymbol {\mu} _ {i} (t)\right) \right]. \tag {26}
$$
Rearranging the terms, we obtain Equation 7 of the main paper. By the definition of expectation given a probability distribution, it is straightforward to obtain Equation 8 of the main paper. Q.E.D.
Without loss of generality, we consider the variance of the belief  $\mu_{is_i}$  about strategy  $s_i$  of population  $i$ . Note that
$$
\operatorname {V a r} \left(\mu_ {i s _ {i}}\right) = \mathbb {E} \left[ \left(\mu_ {i s _ {i}}\right) ^ {2} \right] - \left(\bar {\mu} _ {i s _ {i}}\right) ^ {2}. \tag {27}
$$
$$
\frac {d \operatorname {V a r} \left(\mu_ {i s _ {i}}\right)}{d t} = \frac {d \mathbb {E} \left[ \left(\mu_ {i s _ {i}}\right) ^ {2} \right]}{d t} - 2 \bar {\mu} _ {i s _ {i}} \frac {d \bar {\mu} _ {i s _ {i}}}{d t}. \tag {28}
$$
Consider the first term on the right hand side. We apply the Leibniz rule to interchange differentiation and integration, and then substitute  $\frac{\partial p(\boldsymbol{\mu}_i,t)}{\partial t}$  with Equation 8 in the main paper.
$$
\begin{array}{l} \frac {d \mathbb {E} [ (\mu_ {i s _ {i}}) ^ {2} ]}{d t} \\ = \int \left(\mu_ {i s _ {i}}\right) ^ {2} \frac {\partial p \left(\boldsymbol {\mu} _ {i} , t\right)}{\partial t} d \boldsymbol {\mu} _ {i} (29) \\ = - \int \left(\mu_ {i s _ {i}}\right) ^ {2} \nabla \cdot \left(p \left(\boldsymbol {\mu} _ {i}, t\right) \frac {\bar {\mathbf {x}} _ {i} - \boldsymbol {\mu} _ {i}}{\lambda + t + 1}\right) d \boldsymbol {\mu} _ {i} (30) \\ = - \int \left(\mu_ {i s _ {i}}\right) ^ {2} \sum_ {s _ {i} \in S _ {i}} \partial_ {\mu_ {i s _ {i}}} \left(p \left(\boldsymbol {\mu} _ {i}, t\right) \frac {\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}}{\lambda + t + 1}\right) d \boldsymbol {\mu} _ {i} (31) \\ = \gamma \int \left(\mu_ {i s _ {i}}\right) ^ {2} \sum_ {s _ {i} \in S _ {i}} \partial_ {\mu_ {i s _ {i}}} p \left(\boldsymbol {\mu} _ {i}, t\right) \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) d \boldsymbol {\mu} _ {i} + \gamma \int \left(\mu_ {i s _ {i}}\right) ^ {2} p \left(\boldsymbol {\mu} _ {i}, t\right) \sum_ {s _ {i} \in S _ {i}} \partial_ {\mu_ {i s _ {i}}} \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) d \boldsymbol {\mu} _ {i} (32) \\ \end{array}
$$
where  $\gamma \coloneqq -\frac{1}{\lambda + t + 1}$ . Applying integration by parts to the first term in Equation 32 yields
$$
\begin{array}{l} \int \left(\mu_ {i s _ {i}}\right) ^ {2} \sum_ {s _ {i} \in S _ {i}} \partial_ {\mu_ {i s _ {i}}} p (\boldsymbol {\mu} _ {i}, t) \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) d \boldsymbol {\mu} _ {i} \\ = - \int \left(\mu_ {i s _ {i}}\right) ^ {2} p \left(\boldsymbol {\mu} _ {i}, t\right) \left[ \sum_ {s _ {i} ^ {\prime} \in S _ {i}} \partial_ {\mu_ {i s _ {i} ^ {\prime}}} \left(\bar {x} _ {i s _ {i} ^ {\prime}} - \mu_ {i s _ {i} ^ {\prime}}\right) \right] + p \left(\boldsymbol {\mu} _ {i}, t\right) \partial_ {\mu_ {i s _ {i}}} \left[ \left(\mu_ {i s _ {i}}\right) ^ {2} \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) \right] d \boldsymbol {\mu} _ {i} \tag {33} \\ \end{array}
$$
where we have leveraged that the probability mass at the boundary remains zero (Lemma 1). Combining the above two equations, we obtain
$$
\begin{array}{l} \frac {d \mathbb {E} [ (\mu_ {i s _ {i}}) ^ {2} ]}{d t} \\ = - \gamma \int (\mu_ {i s _ {i}}) ^ {2} p (\boldsymbol {\mu} _ {i}, t) \left[ \sum_ {s _ {i} ^ {\prime} \in S _ {i}} \partial_ {\mu_ {i s _ {i} ^ {\prime}}} (\bar {x} _ {i s _ {i} ^ {\prime}} - \mu_ {i s _ {i} ^ {\prime}}) \right] + p (\boldsymbol {\mu} _ {i}, t) \partial_ {\mu_ {i s _ {i}}} \left[ (\mu_ {i s _ {i}}) ^ {2} (\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}) \right] d \boldsymbol {\mu} _ {i} \\ + \gamma \int \left(\mu_ {i s _ {i}}\right) ^ {2} p \left(\boldsymbol {\mu} _ {i}, t\right) \sum_ {s _ {i} \in S _ {i}} \partial_ {\mu_ {i s _ {i}}} \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) d \boldsymbol {\mu} _ {i} (34) \\ = \gamma \int \left[ - p \left(\mu_ {i}, t\right) \partial_ {\mu_ {i s _ {i}}} \left[ \left(\mu_ {i s _ {i}}\right) ^ {2} \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) \right] \right] + \left(\mu_ {i s _ {i}}\right) ^ {2} p \left(\mu_ {i}, t\right) \partial_ {\mu_ {i s _ {i}}} \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) d \mu_ {i} (35) \\ = \gamma \int 2 \left(\mu_ {i s i}\right) ^ {2} p \left(\boldsymbol {\mu} _ {i}, t\right) d \boldsymbol {\mu} _ {i} - \gamma \int 2 \bar {x} _ {i s i} \mu_ {i s i} p \left(\boldsymbol {\mu} _ {i}, t\right) d \boldsymbol {\mu} _ {i} (36) \\ = - \frac {2 \mathbb {E} \left[ \left(\mu_ {i s _ {i}}\right) ^ {2} \right] - 2 \bar {x} _ {i s _ {i}} \bar {\mu} _ {i s _ {i}}}{\lambda + t + 1}. (37) \\ \end{array}
$$
Next, we consider the second term in Equation 28. By Lemma 4, we have
$$
2 \bar {\mu} _ {i s _ {i}} \frac {d \bar {\mu} _ {i s _ {i}}}{d t} = \frac {2 \bar {\mu} _ {i s _ {i}} \left(\bar {x} _ {i s _ {i}} - \bar {\mu} _ {i s _ {i}}\right)}{\lambda + t + 1}. \tag {38}
$$
Combining Equations 37 and 38, the dynamics of the variance is
$$
\begin{array}{l} \frac {d \operatorname {V a r} \left(\mu_ {i s _ {i}}\right)}{d t} = - \frac {2 \mathbb {E} \left[ \left(\mu_ {i s _ {i}}\right) ^ {2} \right] - 2 \bar {x} _ {i s _ {i}} \bar {\mu} _ {i s _ {i}}}{\lambda + t + 1} - \frac {2 \bar {\mu} _ {i s _ {i}} \left(\bar {x} _ {i s _ {i}} - \bar {\mu} _ {i s _ {i}}\right)}{\lambda + t + 1} (39) \\ = \frac {2 \left(\bar {\mu} _ {i s _ {i}}\right) ^ {2} - 2 \mathbb {E} \left[ \left(\mu_ {i s _ {i}}\right) ^ {2} \right]}{\lambda + t + 1} (40) \\ = - \frac {2 \operatorname {V a r} \left(\mu_ {i s _ {i}}\right)}{\lambda + t + 1}. (41) \\ \end{array}
$$
Lemma 4. The dynamics of the mean belief  $\bar{\mu}_i$  about each population  $i\in V$  is governed by a differential equation
$$
\frac {d \bar {\mu} _ {i s i}}{d t} = \frac {\bar {x} _ {i s i} - \bar {\mu} _ {i s i}}{\lambda + t + 1}, \quad \forall s _ {i} \in S _ {i}. \tag {42}
$$
Proof. The time derivative of the mean belief about strategy  $s_i$  is
$$
\frac {d \bar {\mu} _ {i s i}}{d t} = \frac {d}{d t} \int \mu_ {i s i} p (\boldsymbol {\mu} _ {i}, t) d \boldsymbol {\mu} _ {i}. \tag {43}
$$
We apply the Leibniz rule to interchange differentiation and integration, and then substitute  $\frac{\partial p(\mu_i,t)}{\partial t}$  with Equation 8 in the main paper.
$$
\begin{array}{l} \frac {d}{d t} \int \mu_ {i s _ {i}} p (\boldsymbol {\mu} _ {i}, t) d \boldsymbol {\mu} _ {i} (44) \\ = \int \mu_ {i s _ {i}} \frac {\partial p (\boldsymbol {\mu} _ {i} , t)}{\partial t} d \boldsymbol {\mu} _ {i} (45) \\ = - \int \mu_ {i s _ {i}} \nabla \cdot \left(p \left(\boldsymbol {\mu} _ {i}, t\right) \frac {\bar {\mathbf {x}} _ {i} - \boldsymbol {\mu} _ {i}}{\lambda + t + 1}\right) d \boldsymbol {\mu} _ {i} (46) \\ = - \int \mu_ {i s _ {i}} \sum_ {s _ {i} \in S _ {i}} \partial_ {\mu_ {i s _ {i}}} \left(p \left(\boldsymbol {\mu} _ {i}, t\right) \frac {\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}}{\lambda + t + 1}\right) d \boldsymbol {\mu} _ {i} (47) \\ = \gamma \left[ \int \mu_ {i s _ {i}} \sum_ {s _ {i} \in S _ {i}} \left(\partial_ {\mu_ {i s _ {i}}} p (\boldsymbol {\mu} _ {i}, t)\right) \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) d \boldsymbol {\mu} _ {i} + \int \mu_ {i s _ {i}} p (\boldsymbol {\mu} _ {i}, t) \sum_ {s _ {i} \in S _ {i}} \partial_ {\mu_ {i s _ {i}}} \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) d \boldsymbol {\mu} _ {i} \right] (48) \\ \end{array}
$$
where  $\gamma := -\frac{1}{\lambda + t + 1}$ . Apply integration by parts to the first term in Equation 48.
$$
\begin{array}{l} \int \mu_ {i s _ {i}} \sum_ {s _ {i} \in S _ {i}} \left(\partial_ {\mu_ {i s _ {i}}} p (\boldsymbol {\mu} _ {i}, t)\right) \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) d \boldsymbol {\mu} _ {i} \\ = - \int \mu_ {i s _ {i}} p \left(\boldsymbol {\mu} _ {i}, t\right) \left[ \sum_ {s _ {i} ^ {\prime} \in S _ {i}} \partial_ {\mu_ {i s _ {i} ^ {\prime}}} \left(\bar {x} _ {i s _ {i} ^ {\prime}} - \mu_ {i s _ {i} ^ {\prime}}\right) \right] + p \left(\boldsymbol {\mu} _ {i}, t\right) \partial_ {\mu_ {i s _ {i}}} \left[ \mu_ {i s _ {i}} \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) \right] d \boldsymbol {\mu} _ {i} \tag {49} \\ \end{array}
$$
where we have leveraged that the probability mass at the boundary remains zero. Hence, it follows from Equation 48 that
$$
\frac {d}{d t} \int \mu_ {i s _ {i}} p (\boldsymbol {\mu} _ {i}, t) d \boldsymbol {\mu} _ {i} \tag {50}
$$
$$
\begin{array}{l} = - \gamma \int \mu_ {i s _ {i}} p (\boldsymbol {\mu} _ {i}, t) \sum_ {s _ {i} ^ {\prime} \in S _ {i}} \partial_ {\mu_ {i s _ {i} ^ {\prime}}} (\bar {x} _ {i s _ {i} ^ {\prime}} - \mu_ {i s _ {i} ^ {\prime}}) d \boldsymbol {\mu} _ {i} - \gamma \int p (\boldsymbol {\mu} _ {i}, t) \partial_ {\mu_ {i s _ {i}}} [ \mu_ {i s _ {i}} (\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}) ] d \boldsymbol {\mu} _ {i} \\ + \gamma \int \mu_ {i s _ {i}} p (\boldsymbol {\mu} _ {i}, t) \sum_ {s _ {i} \in S _ {i}} \partial_ {\mu_ {i s _ {i}}} \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) d \boldsymbol {\mu} _ {i} \tag {51} \\ \end{array}
$$
$$
\begin{array}{l} = \gamma \int p \left(\boldsymbol {\mu} _ {i}, t\right) \left[ \mu_ {i s _ {i}} \partial_ {\mu_ {i s _ {i}}} \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) - \partial_ {\mu_ {i s _ {i}}} \left[ \mu_ {i s _ {i}} \left(\bar {x} _ {i s _ {i}} - \mu_ {i s _ {i}}\right) \right] \right] d \boldsymbol {\mu} _ {i} (52) \\ = \gamma \int p (\boldsymbol {\mu} _ {i}, t) \mu_ {i s _ {i}} d \boldsymbol {\mu} _ {i} - \int p (\boldsymbol {\mu} _ {i}, t) \bar {x} _ {i s _ {i}} d \boldsymbol {\mu} _ {i} (53) \\ = \frac {\bar {x} _ {i s _ {i}} - \bar {\mu} _ {i s _ {i}}}{\lambda + t + 1} (54) \\ \end{array}
$$
We repeat the mean probability  $\bar{x}_{is_i}$ , which has been given in Equation 8 in the main paper, as follows:
$$
\bar {x} _ {i s _ {i}} = \int \frac {\exp (\beta u _ {i s _ {i}})}{\sum_ {s _ {i} ^ {\prime} \in S _ {i}} \exp (\beta u _ {i s _ {i} ^ {\prime}})} \prod_ {j \in V _ {i}} p \left(\boldsymbol {\mu} _ {j}, t\right) \left(\prod_ {j \in V _ {i}} d \boldsymbol {\mu} _ {j}\right) \tag {55}
$$
where  $u_{is_i} = \sum_{j\in V_i}\mathbf{e}_{s_i}^\top \mathbf{A}_{ij}\pmb {\mu}_j$  . Define  $\bar{\mathbf{\mu}}\coloneqq \{\bar{\mathbf{\mu}}_j\}_{j\in V_i}$  and
$$
f _ {s _ {i}} \left(\left\{\boldsymbol {\mu} _ {j} \right\} _ {j \in V _ {i}}\right) := \frac {\exp \left(\beta \sum_ {j \in V _ {i}} \mathbf {e} _ {s _ {i}} ^ {\top} \mathbf {A} _ {i j} \boldsymbol {\mu} _ {j}\right)}{\sum_ {s _ {i} ^ {\prime} \in S _ {i}} \exp \left(\beta \sum_ {j \in V _ {i}} \mathbf {e} _ {s _ {i} ^ {\prime}} ^ {\top} \mathbf {A} _ {i j} \boldsymbol {\mu} _ {j}\right)}. \tag {56}
$$
Applying the Taylor expansion to approximate this function at the mean belief  $\bar{\mu}$ , we have
$$
f _ {s _ {i}} \left(\left\{\boldsymbol {\mu} _ {j} \right\} _ {j \in V _ {i}}\right) = f _ {s _ {i}} (\bar {\boldsymbol {\mu}}) + \nabla f _ {s _ {i}} (\bar {\boldsymbol {\mu}}) \cdot (\boldsymbol {\mu} - \bar {\boldsymbol {\mu}}) + \frac {1}{2 !} (\boldsymbol {\mu} - \bar {\boldsymbol {\mu}}) ^ {\top} \mathbf {H} f _ {s _ {i}} (\bar {\boldsymbol {\mu}}) (\boldsymbol {\mu} - \bar {\boldsymbol {\mu}}) + \frac {1}{3 !} O \left(\left\| \boldsymbol {\mu} - \bar {\boldsymbol {\mu}} \right\| ^ {3}\right) \tag {57}
$$
where  $\mathbf{H}$  denotes the Hessian matrix. Hence, we can rewrite Equation 55 as
$$
\begin{array}{l} \bar {x} _ {i s _ {i}} = \int f _ {s _ {i}} \left(\left\{\boldsymbol {\mu} _ {j} \right\} _ {j \in V _ {i}}\right) \prod_ {j \in V _ {i}} p \left(\boldsymbol {\mu} _ {j}, t\right) \left(\prod_ {j \in V _ {i}} d \boldsymbol {\mu} _ {j}\right) (58) \\ \approx f _ {s _ {i}} (\bar {\mathbf {u}}) + \int \nabla f _ {s _ {i}} (\bar {\mathbf {u}}) \cdot \boldsymbol {\mu} \prod_ {j \in V _ {i}} p (\boldsymbol {\mu} _ {j}, t) \left(\prod_ {j \in V _ {i}} d \boldsymbol {\mu} _ {j}\right) - \nabla f _ {s _ {i}} (\bar {\mathbf {u}}) \cdot \bar {\mathbf {u}} \\ + \int \frac {1}{2} (\boldsymbol {\mu} - \bar {\boldsymbol {\mu}}) ^ {\top} \mathbf {H} f _ {s _ {i}} (\bar {\boldsymbol {\mu}}) (\boldsymbol {\mu} - \bar {\boldsymbol {\mu}}) \prod_ {j \in V _ {i}} p (\boldsymbol {\mu} _ {j}, t) \left(\prod_ {j \in V _ {i}} d \boldsymbol {\mu} _ {j}\right) \\ + \int \frac {1}{3 !} O \left(\left\| \boldsymbol {\mu} - \bar {\boldsymbol {\mu}} \right\|\right) ^ {3} \prod_ {j \in V _ {i}} p \left(\boldsymbol {\mu} _ {j}, t\right) \left(\prod_ {j \in V _ {i}} d \boldsymbol {\mu} _ {j}\right) (59) \\ \end{array}
$$