SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 1 (Stronger sufficient condition). We note that for an MG  $(\mathcal{N},\mathcal{S},\mathcal{A},\mathbb{P},(r_i)_{i\in \mathcal{N}},\gamma)$ , if for every agent  $i$ ,  $r_i(s,a_i,\cdot)$  is decomposable with respect to some  $\mathcal{E}_r\subseteq \mathcal{E}_Q$ , and  $\mathbb{P}(s^{\prime}|s,\cdot)$  is decomposable with respect to some  $\mathcal{N}_P\subseteq \mathcal{N}_C$ , then one can still prove the if part, i.e., there exists some  $\mathcal{G} = (\mathcal{N},\mathcal{E}_Q)$  such that the game is an NMG with respect to this  $\mathcal{G}$ . See Figure 1 for the illustration. This is because by our definition, being decomposable with respect to a subset implies being decomposable with respect to a larger set, as one can choose the functions  $f_{i}$  for the  $i$  in the complement of the subset to be simply zero. We chose to state as in Proposition 1 just for the purpose of presenting both the if and only if conditions in a concise and unified way.
Remark 2 (An alternative NMG definition). Another reasonable definition of NMG may be as follows: if for any policy  $\pi$ , there exist a set of functions  $(Q_{i,j}^{\pi})_{(i,j) \in \mathcal{E}_Q}$  and an undirected connected graph  $\mathcal{G} = (\mathcal{N}, \mathcal{E}_Q)$  such that  $Q_i^\pi(s, \boldsymbol{a}) = \sum_{j \in \mathcal{E}_{Q,i}} Q_{i,j}^\pi(s, a_i, a_j)$  holds for every  $i \in \mathcal{N}$ ,  $s \in S$ ,  $\boldsymbol{a} \in \mathcal{A}$ . Note that such a definition can be useful in developing policy-based algorithms (while Definition 1 is more amenable to developing value-based algorithms), e.g., policy iteration, policy gradient, actor-critic methods, where the  $Q$ -value under certain policy  $\pi$  will appear in the updates and may need to preserve certain decomposability structure, for any policy  $\pi$  encountered in the algorithm updates. However, in this case, we cannot always guarantee the decomposability of  $\mathbb{P}(s' | s, \cdot)$  or  $r_i(s, a_i, \cdot)$ . For example, if we assume that  $r_i(s, \boldsymbol{a}) = 0$  for every  $i \in \mathcal{N}$ ,  $s \in S$ ,  $\boldsymbol{a} \in \mathcal{A}$ , then  $Q_i^\pi(s, \boldsymbol{a}) = \sum_{j \in \mathcal{E}_{Q,i}} 0$  and thus  $Q_i^\pi(s, a_i, \cdot)$  is always decomposable regardless of  $\mathbb{P}(s' | s, \boldsymbol{a})$ . However, interestingly, we can show that the decomposability of the transition dynamics and the reward function as in Proposition 1 can still be guaranteed, as long as some degenerate cases as above do not occur. In particular, if there exist no  $i \in \mathcal{N}$  and  $s \in S$  such that  $Q_i^\pi(s, \boldsymbol{a})$  is a constant function of  $\boldsymbol{a}$  for any  $\pi$ , then the results in Proposition 1 and hence after still hold. We defer a detailed discussion on this alternative definition to Appendix B.
Remark 3 (Implication of decomposable transition dynamics). For an MG to be an NMG, by Proposition 1 the transition dynamics should be decomposable, i.e.,  $\mathbb{P}(\cdot |s,\pmb {a}) = \sum_{j\in \mathcal{N}_C}\mathbb{F}_j(\cdot |s,a_j)$  or  $\mathbb{P}(\cdot |s,\pmb {a}) = \mathbb{F}_o(s'|s)$ . We first focus on the discussion of the former case. Define  $w_{j}(s,a_{j}):= \sum_{s^{\prime}\in S}\mathbb{F}_{j}(s^{\prime}|s,a_{j})$ . If we fix the value of  $s$  and  $a_{-j}$ , then  $w_{j}(s,a_{j})$  has to be the same for different values of  $a_{j}$  due to the fact  $\sum_{j\in \mathcal{N}_C}w_j(s,a_j) = 1$ . Also note that by definition,  $w_{j}(s,a_{j})$  does not depend on the choice of this fixed  $a_{-j}$ . Therefore, such a  $w_{j}(s,a_{j})$  can be written as  $w_{j}(s)$ , where  $w_{j}(s) = \sum_{s^{\prime}\in S}\mathbb{F}_{j}(s^{\prime}|s,a_{j})$  for all  $a_{j}\in \mathcal{A}_{j}$ . We can thus rewrite  $(\mathbb{F}_j)_{j\in \mathcal{N}_C}$  using some actual probability distributions  $(\mathbb{P}_j)_{j\in \mathcal{N}_C}$ , such that if  $w_{j}(s)\neq 0$ , then we rewrite  $\mathbb{F}_j$  as  $\mathbb{F}_j(s'|s,a_j) = w_j(s)\frac{\mathbb{F}_j(s'|s,a_j)}{w_j(s)} = w_j(s)\mathbb{P}_j(s'|s,a_j)$ , and if  $w_{j}(s) = 0$ , we rewrite  $\mathbb{F}_j$  as  $\mathbb{F}_j(s'|s,a_j) = w_j(s)\mathbb{P}_j(s'|s,a_j)$  for an arbitrary probability distribution  $\mathbb{P}_j(\cdot |s,a_j)$ . Notice that  $\sum_{s^{\prime}\in S}\mathbb{P}_{j}(s^{\prime}|s,a_{j}) = 1$  for any  $j\in \mathcal{N}_C$ . Then, the decomposable transition dynamics can be represented as  $\mathbb{P}(\cdot |s,\pmb {a}) = \sum_{j\in \mathcal{N}_C}w_j(s)\mathbb{P}_j(\cdot |s,a_j)$ , i.e., an ensemble of the transition dynamics that is only controlled by single controllers. The model's transition dynamics thus act according to the following two steps: (1) sampling the controller according to the distribution Multinomial  $((w_i(s))_{i\in \mathcal{N}_C})$ , and (2) transitioning the state following the sampled controller's dynamics. Such a model has also been investigated under the name of transition dynamics with additive structures in [33]. Note that our model is more general and thus covers the single-controller MG setting [34], where there is only one agent controlling the transition dynamics at all states. It also covers the setting of turn-based MGs [34], where in each round, depending on the current state  $s$ , the transition dynamics is by turns affected by only one of the agents. This can be captured by the proper choice of  $(w_{i}(s))_{i\in \mathcal{N}_{C}}$  that takes value 1 only for one agent at each state  $s$  (while takes value 0 for all other non-controller agents at each state  $s$ ). Additionally, the second case where  $\mathbb{P}(\cdot |s,\pmb {a}) = \mathbb{F}_o(s'|s)$  corresponds to the one with no ensemble of controller agents.
Proposition 2 (Decomposition of  $(Q_i^V)_{i\in \mathcal{N}}$ ). For an infinite-horizon  $\gamma$ -discounted NMG with  $\mathcal{G} = (\mathcal{N},\mathcal{E}_Q)$  such that  $\mathcal{N}_C\neq \emptyset$ , if we know that  $\mathbb{P}(s^{\prime}\mid s,\pmb {a}) = \sum_{i\in \mathcal{N}_C}\mathbb{F}_i(s^{\prime}\mid s,a_i)$ , and
Figure 1: Relationship between  $\mathcal{E}_r$ ,  $\mathcal{E}_{\mathcal{N}_P,\mathrm{connected}}$ , and  $\mathcal{E}_Q$ . Here, we define  $\mathcal{E}_{\mathcal{N}_P,\mathrm{connected}} := \{(i,j) \mid i \in \mathcal{N}_P \text{ or } j \in \mathcal{N}_P, i \neq j\}$ .  $r_i(s,a_i,\cdot)$  is decomposable with respect to  $\mathcal{E}_{r,i}$  for all  $i \in \mathcal{N}$ , and  $\mathbb{P}(s' \mid s,\cdot)$  is decomposable with respect to  $\mathcal{N}_P$  (See Remark 1). The transition dynamics  $\mathbb{P}$  is expressed as the ensemble of controllers in the set  $\mathcal{N}_P = \{5\}$  while the  $\mathcal{N}_C$  in this case is  $\{1,5\}$ .
$r_i(s, \mathbf{a}) = \sum_{j \in \mathcal{E}_{Q,i}} r_{i,j}(s, a_i, a_j)$  for some  $\{\mathbb{F}_i\}_{i \in \mathcal{N}_C}$  and  $\{r_{i,j}\}_{(i,j) \in \mathcal{E}_Q}$ , then the  $Q_{i,j}^V$  given in Definition 1 can be represented as
$Q_{i,j}^{V}(s,a_{i},a_{j}) = r_{i,j}(s,a_{i},a_{j}) + \sum \gamma \left(1(j\in \mathcal{N}_{C})\mathbb{F}_{j}(s^{\prime}\mid s,a_{j}) + 1(i\in \mathcal{N}_{C})\lambda_{i,j}(s)\mathbb{F}_{i}(s^{\prime}\mid s,a_{i})\right)V(s^{\prime})$  for any non-negative  $(\lambda_{i,j}(s))_{(i,j)\in \mathcal{E}_Q}^{\bar{s}'\in S}$  such that  $\sum_{j\in \mathcal{E}_{Q,i}}\lambda_{i,j}(s) = 1$ , for all  $i\in \mathcal{N}$  and  $s\in S$ . We call it the canonical decomposition of  $\{Q_i^V\}_{i\in \mathcal{N}}$  when  $Q_{i,j}^{V}$  can be represented as above with  $\lambda_{i,j}(s) = 1 / |\mathcal{E}_{Q,i}|$  for  $j\in \mathcal{E}_{Q,i}$ . The case with  $\mathcal{N}_C = \emptyset$  is deferred to Appendix B.
We introduce this canonical decomposition since the representation of  $Q_{i,j}^{V}$  is in general not unique, and we may use this canonical form to simplify the algorithm design later.
3.2 Examples of multi-player (zero-sum) NMGs
We now provide several examples of (multi-player) MGs with networked separable interactions here and in Section B.5.
Example 1 (Markov fashion games). Fashion games are an intriguing class of games [25, 26, 27] that plays a vital role not only in Economics theory but also in practice. A fashion game is a networked extension of the Matching Pennies game, in which each player has the action space  $\mathcal{A}_i = \{-1, +1\}$ , which means light and dark color fashions, respectively, for example. There are two types of players: conformists (c), who prefer to conform to their neighbors' fashion (action), and rebels (r), who prefer to oppose their neighbors' fashion (action). Such interactions with the neighbors are exactly captured by polymatrix games. We denote the interaction network between players as  $\mathcal{G} = (\mathcal{N}, \mathcal{E})$ .
Such a game naturally involves the following state transition dynamics: we introduce the state  $s \in S = \mathbb{Z}$  by setting  $s_0 = 0$  and  $s_{t + 1} \sim s_t + \mathrm{Unif}((a_{t,c})_{c\in \mathcal{C}})$ , which indicates the fashion trend where  $\mathcal{C} \subseteq \mathcal{N}$  is the set of influencers. The fashion trend favors either light or dark colors if  $s \geq 0$  or  $s < 0$ , respectively. We can think of dynamics as the impact of the influencers on the fashion trend at time  $t$ . For each  $(s, \pmb{a})$ , the reward function for player  $i$ , depending on whether she is a conformist or a rebel, are defined as  $r_{\mathfrak{r},i}(s,\pmb{a}) = \sum_{j \in \mathcal{E}_i} r_{\mathfrak{r},i,j}(s,a_i,a_j) = \sum_{j \in \mathcal{E}_i} \left( \frac{1}{|\mathcal{E}_i|} \mathbf{1}(\operatorname{sgn}(s) = a_i) + \mathbf{1}(a_i = a_j) \right)$  and  $r_{\mathfrak{r},i}(s,\pmb{a}) = \sum_{j \in \mathcal{E}_i} r_{\mathfrak{r},i,j}(s,a_i,a_j) = \sum_{j \in \mathcal{E}_i} \left( \frac{1}{|\mathcal{E}_i|} \mathbf{1}(\operatorname{sgn}(s) \neq a_i) + \mathbf{1}(a_i \neq a_j) \right)$ , respectively. This is an NMG as defined in Definition 1. Moreover, if the conformists and rebels constitute a bipartite graph, i.e., the neighbors of a conformist are all rebels and vice versa, it becomes a multiplayer constant-sum MG with networked separable interactions, and we can subtract the constant offset to make it a zero-sum NMG.
3.3 Relationship between CCE and NE in zero-sum NMGs
A well-known property for zero-sum NGs is that marginalizing a CCE leads to a NE, which makes it computationally tractable to find the NE [23, 24]. We now provide below a counterpart in the Markov game setting, and provide a more detailed statement of the result in Appendix B.
Proposition 3. Given an  $\epsilon$ -approximate Markov CCE of an infinite-horizon  $\gamma$ -discounted zero-sum NMG, marginalizing it at each state results in an  $\frac{(n+1)}{(1-\gamma)}$ $\epsilon$ -approximate Markov NE of the zero-sum NMG. The same argument also holds for the finite-horizon episodic setting with  $(1-\gamma)^{-1}$  being replaced by  $H$ .
This result holds for both stationary and non-stationary  $\epsilon$ -approximate Markov CCEs. We defer the proof of Proposition 3 to Appendix B. This proposition suggests that if we can have some algorithms to find an approximate Markov CCE for a zero-sum NMG, we can obtain an approximate Markov NE by marginalizing the approximate CCE at each state. We also emphasize that the Markovian property of the equilibrium policies is important for the result to hold. As a result, the learning algorithm in [30], which learns an approximate Markov non-stationary CCE with polynomial time and samples, may thus be used to find an approximate Markov non-stationary NE in zero-sum NMGs. However, as the focus of [30] was the more challenging setting of model-free learning, the complexity therein has a high dependence on the problem parameters, and the algorithm can only find non-perfect equilibria. When it comes to (perfect) equilibrium computation, one may exploit the multi-player zero-sum structure of zero-sum NMGs, and develop more natural and faster algorithms to find a Markov non-stationary NE. Moreover, when it comes to stationary equilibrium computation, even Markov CCE is not tractable in general-sum cases [30, 29]. Hereafter, we will focus on approaching zero-sum NMGs from these perspectives.
4 Hardness for Stationary CCE Computation
Given the results in Section 3.3, it seems tempting and sufficient to compute the Markov CCE of the zero-sum NMG. Indeed, computing CCE (and thus NE) in zero-sum polymatrix games is known to be tractable [23, 24]. It is thus natural to ask: Is finding Markov CCE computationally tractable? Next, we answer the question with different answers for finding stationary CCE (in infinite-horizon  $\gamma$ -discounted setting) and non-stationary CCE (in finite-horizon episodic setting), respectively.
For two-player infinite-horizon  $\gamma$ -discounted zero-sum MGs, significant progress in computing/learning the (Markov) stationary NE has been made recently [35, 36, 37, 38, 39, 40, 41, 42]. On the other hand, for multi-player general-sum MGs, recent results in [30, 29] showed that computing (Markov) stationary CCE can be PPAD-hard and thus believed to be computationally intractable. We next show that this hardness persists in most non-degenerate cases even if one enforces the zero-sum and networked interaction structures in the multi-player case. We state the formal result as follows, whose detailed proof is available in Section C.
Theorem 1. There is a constant  $\epsilon >0$  for which computing an  $\epsilon$ -approximate Markov perfect stationary CCE in infinite-horizon  $\frac{1}{2}$ -discounted zero-sum NMGs, whose underlying network structure contains either a triangle or a 3-path subgraph, is PPAD-hard. Moreover, given the PCP for PPAD conjecture [43], there is a constant  $\epsilon >0$  such that computing even an  $\epsilon$ -approximate Markov non-perfect stationary CCE in such zero-sum NMGs is PPAD-hard.
Proof Sketch of Theorem 1. Due to space constraints, we focus on the case with three players, and the underlying network structure has a triangle subgraph. Proof for the 3-path case is similar and can be found in Section C. We will show that for any general-sum two-player turn-based MG (A), the problem of computing its Markov stationary CCE, which is inherently a PPAD-hard problem [30], can be reduced to computing the Markov stationary CCE of a three-player zero-sum MG with a triangle structure networked separable interactions (B). Consider an MG (A) with two players, players 1 and 2, and reward functions  $r_1(s,a_1,a_2)$  and  $r_2(s,a_2,a_1)$ , where  $a_i$  is the action of the  $i$ -th player and  $r_i$  is the reward function of the  $i$ -th player. The transition dynamics is given by  $\mathbb{P}(s' | s,a_1,a_2)$ . In even rounds, player 2's action space is limited to Noop2, and in odd rounds, player 1's action space is limited to Noop1, where Noop is an abbreviation of "no-operation", i.e., the player does not affect the transition dynamics or the reward in that round. We denote player 1's action space in even rounds as  $\mathcal{A}_{1,\text{even}}$  and player 2's action space in odd rounds as  $\mathcal{A}_{2,\text{odd}}$ , respectively.
Now, we construct a three-player zero-sum NMG with a triangle network structure. We set the reward function as  $\widetilde{r}_i(s,\pmb {a}) = \sum_{j\neq i}\widetilde{r}_{i,j}(s,a_i,a_j)$  and  $\widetilde{r}_{i,j}(s,a_i,a_j) = -\widetilde{r}_{j,i}(s,a_j,a_i)$ . The reward functions are designed so that  $\widetilde{r}_{i,j} = -\widetilde{r}_{j,i}$  for all  $i,j$ ,  $\widetilde{r}_{1,2} + \widetilde{r}_{1,3} = r_1$ , and  $\widetilde{r}_{2,1} + \widetilde{r}_{2,3} = r_2$ , where  $r_1,r_2$  are the reward functions in game (A), by introducing a dummy player, player 3. In even rounds, player 2's action space is limited to Noop2, and in odd rounds, player 1's action space is limited to Noop1. Player 3's action space is always limited to Noop3 in all rounds. The transition dynamics is defined as  $\widetilde{\mathbb{P}} (s^{\prime}|s,a_{1},a_{2},a_{3}) = \mathbb{P}(s^{\prime}|s,a_{1},a_{2})$ , since  $a_3$  is always chosen from Noop3. In other words, player 3's action does not affect the rewards of the other two players, nor the transition dynamics, and players 1 and 2 will receive the reward as in the two-player turn-based MG. Also, note that due to the turn-based structure of the game (A), the transition dynamics satisfy the decomposable condition in our Proposition 1, and it is thus a zero-sum NMG. In fact, turn-based dynamics can be represented as an ensemble of single controller dynamics, as we have discussed in Section 3.
Figure 2: (Left, Middle): PPAD-hardness reduction visualization of  $\mathcal{E}_Q$ . (Right): A star-shaped zero-sum NMG.
Note that the new game (B) is still a turn-based game, and thus the Markov stationary CCE is the same as the Markov stationary NE. Also, note that by construction, the equilibrium policies of players 1 and 2 at the Markov stationary CCE of the game (B) constitute a Markov stationary CCE of the game (A). If the underlying network is more general than a triangle, but contains a triangle subgraph, we can specify the reward and transition dynamics of these three players as above, and specify all other players to be dummy players, whose reward functions are all zero, and do not affect the reward functions of these three players, nor the transition dynamics. This completes the proof.
Figure 2 briefly explains how we may reduce the equilibrium computation problem of (A) to that of (B). In fact, a connected graph that does not contain a subgraph of a triangle or a 3-path has to be a star-shaped network (Proposition 7), which is proved in Section C. Hence, by Theorem 1, we know that in the infinite-horizon discounted setting, finding Markov stationary NE/CE/CCE is a computationally hard problem unless the underlying network is star-shaped. This may also imply that learning Markov stationary NE in zero-sum NMGs, e.g., using natural dynamics like fictitious play to reach the NE, can be challenging, unless in the star-shaped case. In turn, one may hope fictitious-play dynamics to converge for star-shaped zero-sum NMGs. We instantiate this idea next in Section 5. Furthermore, in light of Theorem 1, we will shift gear to computing Markov non-stationary NE by utilizing the structure of networked separable interactions, as to be detailed in Section 6.
5 Fictitious-Play Property
In this section, we study the fictitious-play property of multi-player zero-sum games with networked separable interactions, for both the matrix and Markov game settings. Following the convention in [8], we refer to the games in which fictitious-play dynamics converge to the NE as the games that have the fictitious-play property. We defer the matrix game case results to Section D, where we have also established convergence of the well-known variant of FP, smooth FP [7], in zero-sum NGs.
Echoing the computational intractability of computing CCE of zero-sum NMG unless the underlying network structure is star-shaped in the infinite-horizon discounted setting (c.f. Theorem 1), we now consider the FP property in such games. Note that by Proposition 1,  $\mathcal{E}_Q$  is a star-shape if and only if the reward structure is a star shape and  $\mathcal{N}_C = \{1\}$ , where player 1 is the center of the star (Figure 2), or there are only two players in zero-sum NMG. There is already existing literature for the latter case [15, 16], so we focus on the former case, which is a single-controller case where player 1 controls the transition dynamics, i.e.,  $\mathbb{P}(s' | s, \boldsymbol{a}) = \mathbb{P}_1(s' | s, a_1)$  for some  $\mathbb{P}_1$ . We now introduce the fictitious-play dynamics for such zero-sum NMGs.
Each player  $i$  first initializes her beliefs of other players' policies as uniform distributions, and also initializes her belief of the  $Q$ -value estimates with arbitrary values. Then, at iteration  $k$ , player  $i$  takes the best-response action based on her belief of other players' policies  $(\widehat{\pi}_{-i}^{(k)}(s^{(k)}))$ , and their  $Q$  beliefs  $\widehat{Q}_i^{(k)}(s^{(k)},\pmb {a})$ :
$$
a_{i}^{(k)}\in \operatorname *{argmax}_{a_{i}\in \mathcal{A}_{i}}\widehat{Q}_{i}^{(k)}(s^{(k)},e_{a_{i}},\widehat{\pi}_{-i}^{(k)}(s^{(k)})).
$$
Then, player  $i$  implements the action  $a_{j}^{(k)}$ , observes other players' actions  $a_{-i}^{(k)}$ , and updates her beliefs as follows: for each player  $i \in \mathcal{N}$ , she updates her belief of the opponents' policies as
$$
\widehat {\pi} _ {- i} ^ {(k + 1)} (s) = \widehat {\pi} _ {- i} ^ {(k)} (s) + \mathbf {1} (s = s ^ {(k)}) \alpha^ {N (s)} \left(e _ {a _ {- i} ^ {(k)}} - \widehat {\pi} _ {- i} ^ {(k)} (s)\right)
$$
for all  $s \in S$ , with stepsize  $\alpha^{N(s)} \geq 0$  where  $N(s)$  is the visitation count for the state  $s$ ; then if  $i = 1$ , this player 1 updates the belief of  $Q_{1,j}$  for all  $j \in \mathcal{N} / \{1\}$  and her own  $\widehat{Q}_1(s, a)$  for all  $s \in S$  as
$$
\begin{array}{l} \widehat {Q} _ {1, j} ^ {(k + 1)} (s, a _ {1}, a _ {j}) = \widehat {Q} _ {1, j} ^ {(k)} (s, a _ {1}, a _ {j}) \\ + \mathbf {1} (s = s ^ {(k)}) \beta^ {N (s)} \Big (r _ {1, j} (s, a _ {1}, a _ {j}) + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} \frac {\mathbb {P} _ {1} \left(s ^ {\prime} \mid s , a _ {1}\right)}{n - 1} \cdot \widehat {V} _ {1} ^ {(k)} (s ^ {\prime}) - \widehat {Q} _ {1, j} ^ {(k)} (s, a _ {1}, a _ {i}) \Big), \\ \end{array}
$$
which is based on the canonical decomposition given in Proposition 2, where  $\widehat{V}_1^{(k)}(s) = \max_{a_1\in \mathcal{A}_1}\widehat{Q}_1^{(k)}(s,e_{a_1},\widehat{\pi}_{-1}^{(k)}(s))$ , and  $\beta^{N(s)}\geq 0$  is the stepsize. The agent then updates  $\widehat{Q}_1^{(k + 1)}(s,\pmb {a}) = \sum_{j\in \mathcal{N} / \{1\}}\widehat{Q}_{1,j}^{(k + 1)}(s,a_1,a_j)$ , for all  $s\in S,\pmb {a}\in \mathcal{A}$ . Otherwise, if  $i\neq 1$ , then player  $i$  updates the belief of her  $\widehat{Q}_{i,1}(s,\pmb {a})$  for all  $s\in S,\pmb {a}\in \mathcal{A}$  as
$$
\begin{array}{l} \widehat {Q} _ {i, 1} ^ {(k + 1)} (s, a _ {i}, a _ {1}) = \widehat {Q} _ {i, 1} ^ {(k)} (s, a _ {i}, a _ {1}) \\ + \mathbf {1} (s = s ^ {(k)}) \beta^ {N (s)} \Big (r _ {i, 1} (s, a _ {i}, a _ {1}) + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} \mathbb {P} _ {1} (s ^ {\prime} \mid s, a _ {1}) \cdot \widehat {V} _ {i} ^ {(k)} (s ^ {\prime}) - \widehat {Q} _ {i, 1} ^ {(k)} (s, a _ {i}, a _ {1}) \Big), \\ \end{array}
$$
where  $\widehat{V}_i^{(k)}(s) = \max_{a_i\in \mathcal{A}_i}\widehat{Q}_i^{(k)}(s,e_{a_i},\widehat{\pi}_{-i}^{(k)}(s))$ , and we let  $\widehat{Q}_i^{(k + 1)}(s,\pmb {a}) = \widehat{Q}_{i,1}^{(k + 1)}(s,a_i,a_1)$  for these  $i\neq 1$ . The overall dynamics are summarized in Algorithm 4, which resembles the FP dynamics for two-player zero-sum [16] and identical-interest [19] MGs. Now we are ready to present the convergence guarantees.
Assumption 1. The sequences of step sizes  $\{\alpha^k\in (0,1]\}_{k\geq 0}$  and  $\{\beta^{k}\in (0,1]\}_{k\geq 0}$  satisfy the following conditions: (1)  $\sum_{k = 0}^{\infty}\alpha^{k} = \infty$ $\sum_{k = 0}^{\infty}\beta^{k} = \infty$  , and  $\lim_{k\to \infty}\alpha^k = \lim_{k\to \infty}\beta^k = 0$  ; (2)  $\lim_{k\to \infty}\frac{\beta^k}{\alpha^k} = 0$  , indicating that the rate at which the beliefs about  $Q$  -functions are updated is slower than the rate at which the beliefs about policies are updated.
Theorem 2. Suppose Assumption 1 holds and Algorithm 4 visits every state infinitely often with probability 1. Then, for a star-shaped multi-player zero-sum NMG, the belief  $(\widehat{\pi}^{(k)})_{k\geq 0}$  converges to a Markov stationary NE and the belief  $(\widehat{Q}^{(k)})_{k\geq 0}$  converges to the corresponding NE value of the zero-sum NMG with probability 1, as  $k\to \infty$
We defer the proof to Section D.2 due to space constraints. Note that to illustrate the idea, we only present the result for the model-based case, i.e., when the transition dynamics  $\mathbb{P}$  is known. With this result, it is direct to extend to the model-free and learning case, where  $\mathbb{P}$  is not known [16, 19, 17], still using the tool of stochastic approximation [44]. See Section D for more details.