SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {k \in [ n ]} w _ {k, 1} \geq \sum_ {k \in [ n ]} \sum_ {h = 1} ^ {H} \left(\prod_ {\tau = 1} ^ {h} \mathbb {P} _ {\tau} (\boldsymbol {\pi} _ {\tau})\right) \boldsymbol {r} _ {k, h} (\boldsymbol {\pi} _ {h}) = 0.
$$
Where the equality holds due to the zero-sum property, (1).
An approximate NE is an approximate global minimum. We show that an  $\epsilon$ -approximate NE,  $\pi^{\star}$ , achieves an  $n\epsilon$ -approximate global minimum of the program. Utilizing Lemma A.1, setting  $w_{k}^{\star}(s_{1}) = V_{k,1}^{\dagger,\pi_{-k}^{\star}}(s_{1})$ , and the definition of an  $\epsilon$ -approximate NE we see that,
$$
\begin{array}{l} \sum_ {k \in [ n ]} \left(w _ {k, 1} ^ {\star} (s _ {1}) - e _ {s _ {1}} ^ {\top} \sum_ {h = 1} ^ {H} \left(\prod_ {\tau = 1} ^ {h} \mathbb {P} _ {\tau} \left(\boldsymbol {\pi} _ {\tau} ^ {\star}\right)\right) \boldsymbol {r} _ {k, h} \left(\boldsymbol {\pi} _ {h} ^ {\star}\right)\right) = \sum_ {k \in [ n ]} \left(w _ {k, 1} ^ {\star} (s _ {1}) - V _ {k, 1} ^ {\boldsymbol {\pi} ^ {\star}} (s _ {1})\right) \\ \leq \sum_ {k \in [ n ]} \epsilon = n \epsilon . \\ \end{array}
$$
Indeed, this means that  $\pi^{\star},\pmb{w}^{\star}$  is an  $n\epsilon$  -approximate global minimizer of  $(\mathrm{P}_{\mathrm{NE}})$ .
An approximate global minimum is an approximate NE. For the opposite direction, we let a feasible  $\epsilon$ -approximate global minimizer of the program  $(\mathrm{P}_{\mathrm{NE}}), (\pi^{\star}, w^{\star})$ . Because a global minimum of the program is equal to 0, an  $\epsilon$ -approximate global optimum must be at most  $\epsilon > 0$ . We observe that for every  $k \in [n]$ ,
$$
w _ {k, 1} ^ {\star} \left(s _ {1}\right) \geq e _ {s _ {1}} ^ {\top} \sum_ {h = 1} ^ {H} \left(\prod_ {\tau = 1} ^ {h} \mathbb {P} _ {\tau} \left(\boldsymbol {\pi} _ {\tau} ^ {\star}\right)\right) \boldsymbol {r} _ {k, h} \left(\boldsymbol {\pi} _ {h} ^ {\star}\right), \tag {6}
$$
which follows from induction on the inequality constraint over all  $h$  similar to (5).
Consequently, the assumption that
$$
\epsilon \geq \sum_ {k \in [ n ]} \left(w _ {k, 1} ^ {\star} (s _ {1}) - e _ {s _ {1}} ^ {\top} \sum_ {h = 1} ^ {H} \left(\prod_ {\tau = 1} ^ {h} \mathbb {P} _ {\tau} \left(\boldsymbol {\pi} _ {\tau} ^ {\star}\right)\right) \boldsymbol {r} _ {k, h} \left(\boldsymbol {\pi} _ {h} ^ {\star}\right)\right),
$$
and Equation (6), yields the fact that
$$
\begin{array}{l} \epsilon \geq w _ {k, 1} ^ {\star} (s _ {1}) - e _ {s _ {1}} ^ {\top} \sum_ {h = 1} ^ {H} \left(\prod_ {\tau = 1} ^ {h} \mathbb {P} _ {\tau} \left(\boldsymbol {\pi} _ {\tau} ^ {\star}\right)\right) \boldsymbol {r} _ {k, h} \left(\boldsymbol {\pi} _ {h} ^ {\star}\right) \\ \geq V _ {k, 1} ^ {\dagger , \boldsymbol {\pi} _ {- k} ^ {\star}} (s _ {1}) - V _ {k, 1} ^ {\boldsymbol {\pi} ^ {\star}} (s _ {1}), \\ \end{array}
$$
where the second inequality holds from the fact that  $w^{\star}$  is feasible for  $(\mathrm{P}_{\mathrm{BR}})$ . The latter concludes the proof, as the display coincides with the definition of an  $\epsilon$ -approximate NE.
A.3 Proof of Claim 3.1
Proof. The value function of  $s_1$  for  $h = 1$  of players 1 and 2 read:
$$
\begin{array}{l} V _ {1, 1} ^ {\boldsymbol {\sigma}} (s _ {1}) = \boldsymbol {e} _ {s _ {1}} ^ {\top} (\boldsymbol {r} _ {1} (\boldsymbol {\sigma}) + \mathbb {P} (\boldsymbol {\sigma}) \boldsymbol {r} _ {1} (\boldsymbol {\sigma})) \\ = - \frac {9 \sigma \left(a _ {1} , b _ {1} \mid s _ {1}\right)}{2 0} + \frac {\sigma \left(a _ {1} , b _ {2} \mid s _ {1}\right)}{2 0} + \frac {\left(1 - \sigma \left(a _ {1} , b _ {1} \mid s _ {1}\right)\right) \left(\sigma \left(a _ {1} , b _ {1} \mid s _ {2}\right) + \sigma \left(a _ {1} , b _ {2} \mid s _ {2}\right)\right)}{2 0}, \\ \end{array}
$$
$$
\begin{array}{l} V _ {2, 1} ^ {\boldsymbol {\sigma}} (s _ {1}) = \boldsymbol {e} _ {s _ {1}} ^ {\top} (\boldsymbol {r} _ {2} (\boldsymbol {\sigma}) + \mathbb {P} (\boldsymbol {\sigma}) \boldsymbol {r} _ {2} (\boldsymbol {\sigma})) \\ = - \frac {9 \sigma \left(a _ {1} , b _ {1} \mid s _ {1}\right)}{2 0} + \frac {\sigma \left(a _ {2} , b _ {2} \mid s _ {1}\right)}{2 0} + \frac {\left(1 - \sigma \left(a _ {1} , b _ {1} \mid s _ {1}\right)\right) \left(\sigma \left(a _ {1} , b _ {1} \mid s _ {2}\right) + \sigma \left(a _ {2} , b _ {1} \mid s _ {2}\right)\right)}{2 0}. \\ \end{array}
$$
We are indifferent to the corresponding value function of player 3 as they only have one available action per state and hence, cannot affect their rewards. For the joint policy  $\sigma$ , the corresponding value functions of both players 1 and 2 are  $V_{1,1}^{\sigma}(s_1) = V_{2,1}^{\sigma}(s_1) = \frac{1}{20}$ .
Deviations. We will now prove that no deviation of player 1 manages to accumulate a reward greater than  $\frac{1}{20}$ . The same follows for player 2 due to symmetry.
When a player deviates unilaterally from a joint policy, they experience a single agent Markov decision process (MDP). It is well-known that MDPs always have a deterministic optimal policy. As such, it suffices to check whether  $V_{1,1}^{\pi_1,\sigma^{-1}}(s_1)$  is greater than  $\frac{1}{20}$  for any of the four possible deterministic policies:
$$
\bullet \pi_ {1} (s _ {1}) = \pi_ {1} (s _ {2}) = (1 \quad 0),
$$
$$
\bullet \pi_ {1} (s _ {1}) = (1 0), \pi_ {1} (s _ {2}) = (0 1),
$$
$$
\bullet \pi_ {1} (s _ {1}) = \pi_ {1} (s _ {2}) = (0 \quad 1),
$$
$$
\bullet \pi_ {1} (s _ {1}) = (0 \quad 1), \pi_ {1} (s _ {2}) = (1 \quad 0).
$$
Finally, the value function of any deviation  $\pi_1^{\prime}$  writes,
$$
V _ {1, 1} ^ {\boldsymbol {\pi} _ {1} ^ {\prime} \times \boldsymbol {\sigma} _ {- 1}} (s _ {1}) = - \frac {\pi_ {1} ^ {\prime} (a _ {1} | s _ {1})}{5} - \frac {\pi_ {1} ^ {\prime} (a _ {1} | s _ {2}) (\pi_ {1} ^ {\prime} (a _ {1} | s _ {1}) - 2)}{4 0}.
$$
We can now check that for all deterministic policies  $V_{1,1}^{\pi_1' \times \sigma_{-1}}(s_1) \leq \frac{1}{20}$ . By symmetry, it follows that  $V_{2,1}^{\pi_2' \times \sigma_{-2}}(s_1) \leq \frac{1}{20}$  and as such  $\sigma$  is indeed a CCE.
A.4 Proof of Claim 3.2
Proof. In general, the value functions of each player 1 and 2 are:
$$
V _ {1, 1} ^ {\pi_ {1} \times \pi_ {2}} (s _ {1}) = - \frac {\pi_ {1} (a _ {1} | s _ {1}) \pi_ {2} (b _ {1} | s _ {1})}{2} + \frac {\pi_ {1} (a _ {1} | s _ {1})}{2 0} - \frac {\pi_ {1} (a _ {1} | s _ {2}) (\pi_ {1} (a _ {1} | s _ {1}) \pi_ {2} (b _ {1} | s _ {1}) - 1)}{2 0},
$$
$$
V _ {2, 1} ^ {\boldsymbol {\pi} _ {1} \times \boldsymbol {\pi} _ {2}} (s _ {1}) = - \frac {\pi_ {1} (a _ {1} | s _ {1}) \pi_ {2} (b _ {1} | s _ {1})}{2} + \frac {\pi_ {1} (b _ {1} | s _ {1})}{2 0} - \frac {\pi_ {1} (b _ {1} | s _ {2}) (\pi_ {1} (a _ {1} | s _ {1}) \pi_ {2} (b _ {1} | s _ {1}) - 1)}{2 0}.
$$
Plugging in  $\pi_1^\sigma, \pi_2^\sigma$  yields  $V_{1,1}^{\pi_1^\sigma \times \pi_2^\sigma}(s_1) = V_{2,1}^{\pi_1^\sigma \times \pi_2^\sigma}(s_1) = -\frac{13}{160}$ . But, if player 1 deviates to say  $\pi_1'(s_1) = \pi_1'(s_2) = (0 - 1)$ , they get a value equal to 0 which is clearly greater than  $-\frac{13}{160}$ . Hence,  $\pi_1^\sigma \times \pi_2^\sigma$  is not a NE.
Proof. The proof follows from the game of Example 1, and Claims 3.1 and 3.2.  $\square$
B Proofs for infinite-horizon Zero-Sum Polymatrix Markov Games
In this section we will explicitly state definitions, theorems and proofs relating to the infinite-horizon discounted zero-sum polymatrix Markov games.
B.1 Definitions of equilibria for the infinite-horizon
Let us restate the definition specifically for infinite-horizon Markov games. They are defined as a tuple  $\Gamma(H, S, \{\mathcal{A}_k\}_{k \in [n]}, \mathbb{P}, \{r_k\}_{k \in [n]}, \gamma, \rho)$ .
-  $H = \infty$  denotes the time horizon
-  $S$ , with cardinality  $S \coloneqq |\mathcal{S}|$ , stands for the state space,
-  $\{\mathcal{A}_k\}_{k\in [n]}$  is the collection of every player's action space, while  $\mathcal{A} := \mathcal{A}_1\times \dots \times \mathcal{A}_n$  denotes the joint action space; further, an element of that set—a joint action—is generally noted as  $\pmb{a} = (a_{1},\ldots ,a_{n})\in \mathcal{A}$ ,
-  $\mathbb{P}: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$  is the transition probability function,
-  $r_k: \mathcal{S}, \mathcal{A} \to [-1, 1]$  yields the reward of player  $k$  at a given state and joint action,
- a discount factor  $0 < \gamma < 1$ ,
- an initial state distribution  $\rho \in \Delta(S)$ .
Policies and value functions. In infinite-horizon Markov games policies can still be distinguished in two main ways, Markovian/non-Markovian and stationary/nonstationary. Moreover, a joint policy can be a correlated policy or a product policy.
Markovian policies attribute a probability over the simplex of actions solely depending on the running state  $s$  of the game. On the other hand, non-Markovian policies attribute a probability over the simplex of actions that depends on any subset of the history of the game. I.e., they can depend on any sub-sequence of actions and states up until the running timestep of the horizon.
Stationary policies are those that will attribute the same probability distribution over the simplex of actions for every timestep of the horizon. Nonstationary policies, on the contrary can change depending on the timestep of the horizon.
A joint Markovian stationary policy  $\sigma$  is said to be correlated when for every state  $s\in S$ , attributes a probability distribution over the simplex of joint actions  $\mathcal{A}$  for all players, i.e.,  $\sigma (s)\in \Delta (\mathcal{A})$ . A Markovian stationary policy  $\pi$  is said to be a product policy when for every  $s\in S$ ,  $\pi (s)\in \prod_{k = 1}^{n}\Delta (\mathcal{A}_k)$ . It is rather easy to define correlated/product policies for the case of non-Markovian and nonstationary policies.
Given a Markovian stationary policy  $\pi$ , the value function for an infinite-horizon discounted game is defined as,
$$
V _ {k} ^ {\boldsymbol {\pi}} \left(s _ {1}\right) = \mathbb {E} _ {\boldsymbol {\pi}} \left[ \sum_ {h = 1} ^ {H} \gamma^ {h - 1} r _ {k, h} \left(s _ {h}, \boldsymbol {a} _ {h}\right) \mid s _ {1} \right] = \boldsymbol {e} _ {s _ {1}} ^ {\top} \sum_ {h = 1} ^ {H} \left(\gamma^ {h - 1} \prod_ {\tau = 1} ^ {h} \mathbb {P} _ {\tau} \left(\boldsymbol {\pi} _ {\tau}\right)\right) \boldsymbol {r} _ {k, h} \left(\boldsymbol {\pi} _ {h}\right).
$$
It is possible to express the value function of each player  $k$  in the following way,
Where  $\mathbf{I}$  is the identity matrix of appropriate dimensions. Also, when the initial state is drawn from the initial state distribution, we denote, the value function reads  $V_{k}^{\pi}(\pmb {\rho}) = \pmb{\rho}^{\top}(\mathbf{I} - \gamma \mathbb{P}(\pmb {\pi}))^{-1}\pmb {r}(\pmb {\pi})$
Best-response policies. Given an arbitrary joint policy  $\sigma$  (which can be either a correlated or product policy), a best-response policy of a player  $k$  is defined to be  $\pi_k^\dagger \in \Delta (\mathcal{A}_k)^S$  such that  $\pi_k^\dagger \in \arg \max_{\pi_k'}V_k^{\pi_k'\times \sigma_{-k}}(s)$ . Also, we will denote  $V_{k}^{\dagger,\sigma_{-k}}(s) = \max_{\pi_k'}V_k^{\pi_k',\sigma_{-k}}(s)$ . It is rather straightforward to see that the problem of computing a best-response to a given policy is equivalent to solving a single-agent MDP problem.
Notions of equilibria. Now that best-response policies have been defined, it is straightforward to define the different notions of equilibria. First, we define the notion of a coarse-correlated equilibrium.
Definition B.1 (CCE—infinite-horizon). A joint (potentially correlated) policy  $\sigma \in \Delta (\mathcal{A})^S$  is an  $\epsilon$ -approximate coarse-correlated equilibrium if it holds that for an  $\epsilon$ ,
$$
V _ {k} ^ {\dagger , \sigma_ {- k}} (\boldsymbol {\rho}) - V _ {k} ^ {\sigma} (\boldsymbol {\rho}) \leq \epsilon , \forall k \in [ n ].
$$
Second, we define the notion of a Nash equilibrium. The main difference of the definition of the coarse-correlated equilibrium, is the fact that a NE Markovian stationary policy is a product policy.
Definition B.2 (NE—infinite-horizon). A joint (potentially correlated) policy  $\pi \in \prod_{k\in [n]}\Delta (\mathcal{A}_k)^S$  is an  $\epsilon$ -approximate coarse-correlated equilibrium if it holds that for an  $\epsilon$ ,
$$
V _ {k} ^ {\dagger , \pi - k} (\boldsymbol {\rho}) - V _ {k} ^ {\pi} (\boldsymbol {\rho}) \leq \epsilon , \forall k \in [ n ].
$$
As it is folklore by now, infinite-horizon discounted Markov games have a stationary Markovian Nash equilibrium.
C Main results for infinite-horizon games
The workhorse of our arguments in the following results is still the following nonlinear program with variables  $\pi, w$ ,
$$
\begin{array}{l} \min  \sum_ {k \in [ n ]} \boldsymbol {\rho} ^ {\top} \left(\boldsymbol {w} _ {k} - (\mathbf {I} - \gamma \mathbb {P} (\boldsymbol {\pi})) ^ {- 1} \boldsymbol {r} _ {k} (\boldsymbol {\pi})\right) \\ \begin{array}{c} \text {s . t .} w _ {k} (s) \geq r _ {k} (s, a, \boldsymbol {\pi} _ {- k}) + \gamma   \mathbb {P} (s, a, \boldsymbol {\pi} _ {- k}) \boldsymbol {w} _ {k}, \\ \forall s \in \mathcal {S}, \forall k \in [ n ], \forall a \in \mathcal {A} _ {k}; \end{array} \\ \pi_ {k} (s) \in \Delta (\mathcal {A} _ {k}), \\ \forall s \in \mathcal {S}, \forall k \in [ n ], \forall a \in \mathcal {A} _ {k}. \\ \end{array}
$$
As we will prove, approximate NE's correspond to approximate global minima of  $(\mathrm{P}_{\mathrm{NE}}^{\prime})$  and vice-versa. Before that, we need some intermediate lemmas. The first lemma we prove is about the best-response program.
The best-response program. Even for the infinite-horizon, we can define a linear program for the best-responses of all players. That program is the following, with variables  $\boldsymbol{w}$ ,
$$
\min  \sum_ {k \in [ n ]} \boldsymbol {\rho} ^ {\top} \left(\boldsymbol {w} _ {k} - (\mathbf {I} - \gamma \mathbb {P} (\hat {\boldsymbol {\sigma}})) ^ {- 1} \boldsymbol {r} _ {k} (\hat {\boldsymbol {\sigma}})\right)
$$
$$
\begin{array}{c} (\mathrm {P} _ {\mathrm {B R}} ^ {\prime}) \hskip 1 4. 2 2 6 3 7 8 p t \text {s . t .} w _ {k} (s) \geq r _ {k} (s, a, \hat {\boldsymbol {\sigma}} _ {- k}) + \mathbb {P} (s, a, \hat {\boldsymbol {\sigma}} _ {- k}) \boldsymbol {w} _ {k}, \\ \forall s \in \mathcal {S}, \forall k \in [ n ], \forall a \in \mathcal {A} _ {k}. \end{array}
$$
Lemma C.1 (Best-response LP—infinite-horizon). Let a (possibly correlated) joint policy  $\hat{\sigma}$ . Consider the linear program  $(\mathrm{P}_{\mathrm{BR}}^{\prime})$ . The optimal solution  $\boldsymbol{w}^{\dagger}$  of the program is unique and corresponds to the value function of each player  $k \in [n]$  when player  $k$  best-responds to  $\hat{\sigma}$ .
Proof. We observe that the program is separable to  $n$  independent linear programs, each with variables  $\boldsymbol{w}_k \in \mathbb{R}^n$ ,
$$
\min  \rho^ {\top} \boldsymbol {w} _ {k}
$$
$$
\mathrm {s . t .} w _ {k} (s) \geq r _ {k} (s, a, \hat {\sigma} _ {- k}) + \gamma \mathbb {P} (s, a, \hat {\sigma} _ {- k}) \boldsymbol {w} _ {k},
$$
$$
\forall s \in \mathcal {S}, \forall a \in \mathcal {A} _ {k}.
$$
Each of these linear programs describes the problem of a single agent MDP—that agent being  $k$ . It follows that the optimal  $\boldsymbol{w}_k^\dagger$  for every program is unique (each program corresponds to a set of Bellman optimality equations).
Lemma C.2 (Feasibility of  $\mathrm{(P_{NE}^{\prime})}$  and global optimum). The nonlinear program  $(\mathrm{P}_{\mathrm{NE}}^{\prime})$  is feasible, has a nonnegative objective value, and its global minimum is equal to 0.
Proof. For the feasibility of the nonlinear program, we invoke the theorem of the existence of a Nash equilibrium. i.e., let a NE product policy,  $\pi^{\star}$ , and a vector  $\pmb{w}^{\star} \in \mathbb{R}^{n \times H \times S}$  such that  $w_{k,s}^{\star}(s) = V_k^{\dagger, \pmb{\pi}_{-k}^{\star}}(s)$ ,  $\forall k \in [n] \times S$ .
By Lemma C.1, we know that  $(\pmb{\pi}^{\star}, \pmb{w}^{\star})$  satisfies all the constraints of  $(\mathrm{P}_{\mathrm{NE}}')$ . Additionally, because  $\pmb{\pi}^{\star}$  is a NE,  $V_{k}^{\pmb{\pi}^{\star}}(\pmb{\rho}) = V_{k}^{\dagger, \pmb{\pi}_{-k}^{\star}}(\pmb{\rho})$  for all  $k \in [n]$ . Observing that,
$$
\boldsymbol {\rho} ^ {\top} \left(\boldsymbol {w} _ {k} ^ {\star} - (\mathbf {I} - \gamma \mathbb {P} (\boldsymbol {\pi} ^ {\star})) ^ {- 1} \boldsymbol {r} _ {k} (\boldsymbol {\pi} ^ {\star})\right) = V _ {k} ^ {\dagger , \boldsymbol {\pi} _ {- k} ^ {\star}} (\boldsymbol {\rho}) - V _ {k} ^ {\boldsymbol {\pi} ^ {\star}} (\boldsymbol {\rho}) = 0,
$$
$$
\begin{array}{l} \sum_ {k \in [ n ]} \left(\boldsymbol {\rho} ^ {\top} \boldsymbol {w} _ {k} - \boldsymbol {\rho} ^ {\top} (\mathbf {I} - \gamma \mathbb {P} (\boldsymbol {\pi})) ^ {- 1} \boldsymbol {r} _ {k} (\boldsymbol {\pi})\right) \\ = \sum_ {k \in [ n ]} \boldsymbol {\rho} ^ {\top} \boldsymbol {w} _ {k} - \boldsymbol {\rho} ^ {\top} (\mathbf {I} - \gamma \mathbb {P} (\boldsymbol {\pi})) ^ {- 1} \sum_ {k \in [ n ]} \boldsymbol {r} _ {k} (\boldsymbol {\pi} _ {h}) \\ = \sum_ {k \in [ n ]} \boldsymbol {\rho} ^ {\top} \boldsymbol {w} _ {k}. \\ \end{array}
$$
$$
w _ {k} (s) \geq r _ {k} (s, a, \boldsymbol {\pi} _ {- k}) + \gamma \mathbb {P} (s, a, \boldsymbol {\pi} _ {- k}) \boldsymbol {w} _ {k}
$$
which holds for all  $s \in S$ , all players  $k \in [n]$ , and all  $a \in \mathcal{A}_k$ .
By summing over  $a \in \mathcal{A}_k$  while multiplying each term with a corresponding coefficient  $\pi_k(a|s)$ , the display written in an equivalent element-wise vector inequality reads:
$$
\boldsymbol {w} _ {k} \geq \boldsymbol {r} _ {k, h} (\boldsymbol {\pi}) + \gamma \mathbb {P} (\boldsymbol {\pi}) \boldsymbol {w} _ {k}.
$$
Finally, after consecutively substituting  $\pmb{w}_k$  with the element-wise lesser term  $\pmb{r}_k(\pmb{\pi}) + \gamma \mathbb{P}(\pmb{\pi})\pmb{w}_k$ , we end up with the inequality:
$$
\boldsymbol {w} _ {k} \geq (\mathbf {I} - \gamma \mathbb {P} (\boldsymbol {\pi})) ^ {- 1} \boldsymbol {r} _ {k} (\boldsymbol {\pi}). \tag {9}
$$
We note that  $\mathbf{I} + \gamma \mathbb{P}(\pmb {\pi}) + \gamma^2\mathbb{P}^2 (\pmb {\pi}) + \dots = (\mathbf{I} - \gamma \mathbb{P}(\pmb {\pi}))^{-1}$
$$
\sum_ {k \in [ n ]} \boldsymbol {w} _ {k} \geq \sum_ {k \in [ n ]} (\mathbf {I} - \gamma \mathbb {P} (\boldsymbol {\pi})) ^ {- 1} \boldsymbol {r} _ {k} (\boldsymbol {\pi}) = (\mathbf {I} - \gamma \mathbb {P} (\boldsymbol {\pi})) ^ {- 1} \sum_ {k \in [ n ]} \boldsymbol {r} _ {k} (\boldsymbol {\pi}) = 0.
$$
Theorem C.1 (NE and global optima of  $(\mathrm{P}_{\mathrm{NE}}^{\prime})$  infinite-horizon). If  $(\pi^{\star},\pmb{w}^{\star})$  yields an  $\epsilon$ -approximate global minimum of  $(\mathrm{P}_{\mathrm{NE}}^{\prime})$ , then  $\pmb{\pi}^{\star}$  is an  $n\epsilon$ -approximate NE of the infinite-horizon zero-sum polymatrix switching controller  $MG, \Gamma$ . Conversely, if  $\pmb{\pi}^{\star}$  is an  $\epsilon$ -approximate NE of the  $MG\Gamma$  with corresponding value function vector  $\pmb{w}^{\star}$  such that  $w_{k}^{\star}(s) = V_{k}^{\pi^{\star}}(s)\forall (k,s)\in [n]\times S$ , then  $(\pi^{\star},\pmb{w}^{\star})$  attains an  $\epsilon$ -approximate global minimum of  $(\mathrm{P}_{\mathrm{NE}}^{\prime})$ .
An approximate NE is an approximate global minimum. We show that an  $\epsilon$ -approximate NE,  $\pi^{\star}$ , achieves an  $n\epsilon$ -approximate global minimum of the program. Utilizing Lemma C.1 by setting  $\boldsymbol{w}_k^{\star} = \mathbf{V}^{\dagger,\pi_{-k}^{\star}}(\rho)$ , feasibility, and the definition of an  $\epsilon$ -approximate NE we see that,
$$
\begin{array}{l} \sum_ {k \in [ n ]} \left(\boldsymbol {\rho} ^ {\top} \boldsymbol {w} _ {k} ^ {\star} - \boldsymbol {\rho} ^ {\top} \left(\mathbf {I} - \gamma \mathbb {P} \left(\boldsymbol {\pi} ^ {\star}\right)\right) ^ {- 1} \boldsymbol {r} _ {k} \left(\boldsymbol {\pi} ^ {\star}\right)\right) = \sum_ {k \in [ n ]} \left(\boldsymbol {\rho} ^ {\top} \boldsymbol {w} _ {k} ^ {\star} - V _ {k} ^ {\boldsymbol {\pi} ^ {\star}} (\boldsymbol {\rho})\right) \\ \leq \sum_ {k \in [ n ]} \epsilon = n \epsilon . \\ \end{array}
$$
Indeed, this means that  $\pi^{\star},\pmb{w}^{\star}$  is an  $n\epsilon$  -approximate global minimizer of  $(\mathrm{P}_{\mathrm{NE}}^{\prime})$
An approximate global minimum is an approximate NE. For this direction, we let a feasible  $\epsilon$ -approximate global minimizer of the program  $(\mathrm{P}_{\mathrm{NE}}^{\prime})$ ,  $(\pi^{\star}, w^{\star})$ . Because a global minimum of the program is equal to 0, an  $\epsilon$ -approximate global optimum must be at most  $\epsilon > 0$ . We observe that for every  $k \in [n]$ ,
$$
\boldsymbol {\rho} ^ {\top} \boldsymbol {w} _ {k} ^ {\star} \geq \boldsymbol {\rho} ^ {\top} \left(\mathbf {I} - \gamma \mathbb {P} \left(\boldsymbol {\pi} ^ {\star}\right)\right) ^ {- 1} \boldsymbol {r} _ {k} \left(\boldsymbol {\pi} ^ {\star}\right), \tag {10}
$$
which follows from induction on the inequality constraint (9).
$$
\epsilon \geq \rho^ {\top} \boldsymbol {w} _ {k} ^ {\star} - \rho^ {\top} \left(\mathbf {I} - \gamma \mathbb {P} \left(\boldsymbol {\pi} ^ {\star}\right)\right) ^ {- 1} \boldsymbol {r} _ {k} \left(\boldsymbol {\pi} ^ {\star}\right)
$$
and Equation (10), yields the fact that
$$
\begin{array}{l} \epsilon \geq \rho^ {\top} \boldsymbol {w} _ {k} ^ {\star} - \rho^ {\top} \left(\mathbf {I} - \gamma \mathbb {P} \left(\boldsymbol {\pi} ^ {\star}\right)\right) ^ {- 1} \boldsymbol {r} _ {k} \left(\boldsymbol {\pi} ^ {\star}\right) \\ \geq V _ {k} ^ {\dagger , \boldsymbol {\pi} _ {- k} ^ {\star}} (\boldsymbol {\rho}) - V _ {k} ^ {\boldsymbol {\pi} ^ {\star}} (\boldsymbol {\rho}), \\ \end{array}
$$
where the second inequality holds from the fact that  $\pmb{w}^{\star}$  is also feasible for  $(\mathrm{P}_{\mathrm{BR}}^{\prime})$ . The latter concludes the proof, as the display coincides with the definition of an  $\epsilon$ -approximate NE.
Theorem C.2 (CCE collapse to NE in polymatrix MG—infinite-horizon). Let a zero-sum polymatrix switching-control Markov game, i.e., a Markov game for which Assumptions 1 and 2 hold. Further, let an  $\epsilon$ -approximate CCE of that game  $\sigma$ . Then, the marginal product policy  $\pi^{\sigma}$ , with  $\pi_k^\sigma(a|s) = \sum_{a_{-k} \in \mathcal{A}_{-k}} \sigma(a, a_{-k})$ ,  $\forall k \in [n]$  is an  $n\epsilon$ -approximate NE.
$$
\begin{array}{l} w _ {k} ^ {\dagger} (s) \geq r _ {k} (s, a, \boldsymbol {\sigma} _ {- k}) + \gamma \mathbb {P} _ {h} (s, a, \boldsymbol {\sigma} _ {- k}) \boldsymbol {w} _ {k} ^ {\dagger} \\ = \sum_ {j \in \operatorname {a d j} (k)} r _ {(k, j), h} (s, a, \boldsymbol {\pi} _ {j} ^ {\sigma}) + \gamma \mathbb {P} (s, a, \boldsymbol {\sigma} _ {- k}) \boldsymbol {w} _ {k} ^ {\dagger}. \\ \end{array}
$$
$$
\mathbb {P} (s, a, \boldsymbol {\sigma} _ {- k}) \boldsymbol {w} _ {k} ^ {\dagger} = \mathbb {P} (s, a, \boldsymbol {\pi} _ {\operatorname {a r g c t r l} (s)} ^ {\boldsymbol {\sigma}}) \boldsymbol {w} _ {k} ^ {\dagger},
$$
$$
\mathbb {P} (s, a, \boldsymbol {\sigma} _ {- k}) \boldsymbol {w} _ {k} ^ {\dagger} = \mathbb {P} (s, a, \boldsymbol {\pi} ^ {\boldsymbol {\sigma}}) \boldsymbol {w} _ {k} ^ {\dagger}.
$$
Putting these pieces together, we reach the conclusion that  $(\pi^{\sigma},\pmb{w}^{\dagger})$  is feasible for the nonlinear program  $\mathrm{(P_{NE}^{\prime})}$
What is left is to prove that it is also an  $\epsilon$ -approximate global minimum. Indeed, if  $\sum_{k} \rho^{\top} \boldsymbol{w}_{k}^{\dagger} \leq \epsilon$  (by assumption of an  $\epsilon$ -approximate CCE), then the objective function of  $(\mathrm{P}_{\mathrm{NE}}')$  will attain an  $\epsilon$ -approximate global minimum. In turn, due to Theorem C.1 the latter implies that  $\pi^{\sigma}$  is an  $n\epsilon$ -approximate NE.
C.1 No equilibrium collapse with more than one controllers per-state