SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. The reward vector  $r_i \in \mathbb{R}^{\prod_{i=1}^{n} |\mathcal{A}_i|}$  can be viewed as an  $\prod_{i=1}^{n} |\mathcal{A}_i|$ -dimensional linear vector. There are in total  $(|\mathcal{A}_i| - 1) \prod_{j \neq i} |\mathcal{A}_j| = \prod_{i=1}^{n} |\mathcal{A}_i| - \prod_{j \neq i} |\mathcal{A}_j|$  independent constraints w.r.t  $r_i$  and potential function  $\phi$ . Since  $\mathcal{R}_i$  is a linear space with dimension  $\prod_{j \neq i} |\mathcal{A}_j|$ ,  $\mathcal{R}_i$  is the space of valid reward functions for potential games.
B Proof for Markov Potential Games in Section 3.2
We first introduce the following lemma to fill the gap between general MPGs and fully-cooperative MPGs. Define  $h_i(s, \pmb{a}) \coloneqq r_i(s, \pmb{a}) - \phi(s, \pmb{a})$ , which implies  $V_i^\pi(s) = \Phi^\pi(s) + V_{h_i}^\pi(s)$ .
Lemma B.1 (Restatement of Lemma 3.4). Define  $\bar{A}_{h_i}^{\pi}(s,a_i)$  with respect to  $h_i$  similar to (2). We then have
$$
\sum_ {a _ {i}} (\pi_ {i} ^ {\prime} (a _ {i} | s) - \pi_ {i} (a _ {i} | s)) \bar {A} _ {h _ {i}} ^ {\pi} (s, a _ {i}) = 0, \quad \forall s \in \mathcal {S}, i \in [ n ], \pi_ {i} ^ {\prime}, \pi_ {i} \in \Delta (\mathcal {A} _ {i}).
$$
Proof of Lemma 3.4. Due to the definition of Markov potential game, we have
$$
V _ {i} ^ {\pi_ {i} ^ {\prime}, \pi_ {- i}} (\mu) - V _ {i} ^ {\pi_ {i}, \pi_ {- i}} (\mu) = \Phi^ {\pi_ {i} ^ {\prime}, \pi_ {- i}} (\mu) - \Phi^ {\pi_ {i}, \pi_ {- i}} (\mu),
$$
for any  $\mu, \pi_i'$  and  $\pi_i$ . It then follows that
$$
0 = V _ {h _ {i}} ^ {\pi_ {i} ^ {\prime}, \pi_ {- i}} (\mu) - V _ {h _ {i}} ^ {\pi_ {i}, \pi_ {- i}} (\mu) = \frac {1}{1 - \gamma} \sum_ {s} d _ {\mu} ^ {\pi_ {i} ^ {\prime}, \pi_ {- i}} (s) \sum_ {a _ {i}} (\pi_ {i} ^ {\prime} (a _ {i} | s) - \pi_ {i} (a _ {i} | s)) \bar {A} _ {h _ {i}} ^ {\pi} (s, a _ {i}).
$$
Denote  $[\tilde{P}]_{s,s'} = \mathbb{E}_{a\sim \pi_i'\times \pi_{-i}}[P(s'|s,a)]$  as the transition matrix generated by policy  $\pi_i^\prime \times \pi_{-i}$ . Then
$$
\frac {1}{1 - \gamma} d _ {\mu} ^ {\pi_ {i} ^ {\prime}, \pi_ {- i}} = \mu (I + \gamma \tilde {P} + \gamma^ {2} \tilde {P} ^ {2} + \dots) = \mu (I - \gamma \tilde {P}) ^ {- 1},
$$
where  $d_{\mu}^{\pi_i', \pi_{-i}}$  and  $\mu$  are row vectors and  $(I - \gamma \tilde{P})^{-1}$  is clearly full rank. Since the above relationship holds for any  $\mu \in \Delta_{\mathcal{S}}$ , we get  $0 = \sum_{a_i} (\pi_i'(a_i|s) - \pi_i(a_i|s)) \bar{A}_{h_i}^{\pi}(s, a_i)$ ,  $\forall s \in \mathcal{S}$ .
Then, we analyze the lower bound of the potential value function difference.
Lemma B.2 (Restatement of Lemma 3.5). Given policy  $\pi^k$  and marginalized advantage function  $\bar{A}_i^{\pi^k}(s,a_i)$ , for  $\pi^{k + 1}$  generated using NPG update in (7), we have the following inequality,
Proof of Lemma 3.5. We introduce
$$
\tilde {\pi} _ {- i} ^ {k} \left(a _ {- i} | s\right) := \prod_ {j <   i} \pi_ {j} ^ {k + 1} \left(a _ {j} | s\right) \prod_ {\ell > i} \pi_ {\ell} ^ {k} \left(a _ {\ell} | s\right)
$$
to denote the mixed strategy where each agent with index  $j < i$  follows  $\pi_j^{k+1}$  and each agent with index  $\ell > i$  follows  $\pi_\ell^k$  instead. Let  $\bar{A}_{i,\phi}^{\tilde{\pi}^k}$  be the associated marginalized advantage value based on potential function, i.e.,
$$
\bar {A} _ {i, \phi} ^ {\tilde {\pi} ^ {k}} (s, a _ {i}) = \sum_ {a _ {- i} \in \mathcal {A} _ {- i}} A _ {\phi} ^ {\pi^ {k}} (s, a _ {i}, a _ {- i}) \prod_ {j <   i} \pi_ {j} ^ {k + 1} (a _ {j} | s) \prod_ {\ell > i} \pi_ {\ell} ^ {k} (a _ {\ell} | s).
$$
We now decompose  $\Phi^{\pi^{k + 1}} - \Phi^{\pi^k}$  as follows:
$$
\begin{array}{l} \Phi^ {\pi^ {k + 1}} (\rho) - \Phi^ {\pi^ {k}} (\rho) \\ = \frac {1}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \sum_ {a} \left(\pi^ {k + 1} (a | s) - \pi^ {k} (a | s)\right) A _ {\phi} ^ {\pi^ {k}} (s, a) \\ = \frac {1}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \sum_ {a} \sum_ {i = 1} ^ {n} \left(\prod_ {j = 1} ^ {i} \pi_ {j} ^ {k + 1} \left(a _ {j} | s\right) \prod_ {\ell = i + 1} ^ {n} \pi_ {\ell} ^ {k} \left(a _ {\ell} | s\right) - \prod_ {j = 1} ^ {i - 1} \pi_ {j} ^ {k + 1} \left(a _ {j} | s\right) \prod_ {\ell = i} ^ {n} \pi_ {\ell} ^ {k} \left(a _ {\ell} | s\right)\right) A _ {\phi} ^ {\pi^ {k}} (s, a) \\ = \frac {1}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \sum_ {i = 1} ^ {n} \sum_ {a _ {i}} \left(\pi_ {i} ^ {k + 1} \left(a _ {i} | s\right) - \pi_ {i} ^ {k} \left(a _ {i} | s\right)\right) \bar {A} _ {i, \phi} ^ {\tilde {\pi} ^ {k}} (s, a _ {i}) \\ = \frac {1}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \sum_ {i = 1} ^ {n} \sum_ {a _ {i}} \left(\pi_ {i} ^ {k + 1} \left(a _ {i} | s\right) - \pi_ {i} ^ {k} \left(a _ {i} | s\right)\right) \bar {A} _ {i} ^ {\pi^ {k}} (s, a _ {i}) \\ - \frac {1}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \sum_ {i = 1} ^ {n} \sum_ {a _ {i}} \left(\pi_ {i} ^ {k + 1} \left(a _ {i} | s\right) - \pi_ {i} ^ {k} \left(a _ {i} | s\right)\right) \bar {A} _ {h _ {i}} ^ {\pi^ {k}} (s, a _ {i}) \\ + \frac {1}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \sum_ {i = 1} ^ {n} \sum_ {a _ {i}} (\pi_ {i} ^ {k + 1} (a _ {i} | s) - \pi_ {i} ^ {k} (a _ {i} | s)) (\bar {A} _ {i, \phi} ^ {\tilde {\pi} ^ {k}} (s, a _ {i}) - \bar {A} _ {i, \phi} ^ {\pi^ {k}} (s, a _ {i})). \\ \end{array}
$$
$$
\begin{array}{l} | \bar {A} _ {i, \phi} ^ {\tilde {\pi} ^ {k}} (s, a _ {i}) - \bar {A} _ {i, \phi} ^ {\pi^ {k}} (s, a _ {i}) | \\ = \left| \sum_ {a _ {- i}} \left(\prod_ {j = 1} ^ {i - 1} \pi_ {j} ^ {k + 1} (a _ {j} | s) - \prod_ {j = 1} ^ {i - 1} \pi_ {j} ^ {k} (a _ {j} | s)\right) \prod_ {\ell = i + 1} ^ {n} \pi_ {\ell} ^ {k} (a _ {\ell} | s) A _ {\phi} ^ {\pi^ {k}} (s, \boldsymbol {a}) \right| \\ \leq \frac {\phi_ {m a x}}{1 - \gamma} \| \tilde {\pi} _ {- i} ^ {k} (\cdot | s) - \pi_ {- i} ^ {k} (\cdot | s) \| _ {1}, \\ \end{array}
$$
using Lemma 3.4, we have
$$
\begin{array}{l} \Phi^ {\pi^ {k + 1}} (\rho) - \Phi^ {\pi^ {k}} (\rho) \geq \frac {1}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \sum_ {i = 1} ^ {n} \sum_ {a _ {i}} \pi_ {i} ^ {k + 1} (a _ {i} | s) \bar {A} _ {i} ^ {\pi^ {k}} (s, a _ {i}) \\ - \frac {\phi_ {m a x}}{(1 - \gamma) ^ {2}} \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \sum_ {i = 1} ^ {n} \| \pi_ {i} ^ {k + 1} (\cdot | s) - \pi_ {i} ^ {k} (\cdot | s) \| _ {1} \| \tilde {\pi} _ {- i} ^ {k} (\cdot | s) - \pi_ {- i} ^ {k} (\cdot | s) \| _ {1}. \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {i = 1} ^ {n} \left\| \pi_ {i} ^ {k + 1} (\cdot | s) - \pi_ {i} ^ {k} (\cdot | s) \right\| _ {1} \left\| \tilde {\pi} _ {- i} ^ {k} (\cdot | s) - \pi_ {- i} ^ {k} (\cdot | s) \right\| _ {1} \\ \leq \frac {\sqrt {n}}{2} \sum_ {i = 1} ^ {n} \| \pi_ {i} ^ {k + 1} (\cdot | s) - \pi_ {i} ^ {k} (\cdot | s) \| _ {1} ^ {2} + \frac {1}{2 \sqrt {n}} \sum_ {i = 1} ^ {n} \| \tilde {\pi} _ {- i} ^ {k} (\cdot | s) - \pi_ {- i} ^ {k} (\cdot | s) \| _ {1} ^ {2} \\ \leq \sqrt {n} \sum_ {i = 1} ^ {n} \left(K L (\pi_ {i} ^ {k + 1} (\cdot | s) | | \pi_ {i} ^ {k} (\cdot | s)) + K L (\pi_ {i} ^ {k} (\cdot | s) | | \pi_ {i} ^ {k + 1} (\cdot | s))\right) \\ = \sqrt {n} \sum_ {i = 1} ^ {n} \left\langle \pi_ {i} ^ {k + 1} (\cdot | s) - \pi_ {i} ^ {k} (\cdot | s), \log \pi_ {i} ^ {k + 1} (\cdot | s) - \log \pi_ {i} ^ {k} (\cdot | s) \right\rangle \\ = \frac {\sqrt {n} \eta}{1 - \gamma} \sum_ {i = 1} ^ {n} \left\langle \pi_ {i} ^ {k + 1} (\cdot | s) - \pi_ {i} ^ {k} (\cdot | s), \bar {A} _ {i} ^ {\pi^ {k}} (s, \cdot) \right\rangle = \frac {\sqrt {n} \eta}{1 - \gamma} \sum_ {i = 1} ^ {n} \left\langle \pi_ {i} ^ {k + 1} (\cdot | s), \bar {A} _ {i} ^ {\pi^ {k}} (s, \cdot) \right\rangle , \\ \end{array}
$$
Next, we introduce the following lemma, which is the MPG version of Lemma 3.2.
Lemma B.3. At iteration  $k$ , define
$$
f ^ {k} (\alpha) = \sum_ {i \in [ n ]} \sum_ {a _ {i}} \pi_ {i, \alpha} (a _ {i} | s) \bar {A} _ {i} ^ {\pi^ {k}} (s, a _ {i}), w h e r e \pi_ {i, \alpha} (\cdot | s) \propto \pi_ {i} ^ {k} (\cdot | s) \exp \left\{\frac {\alpha \bar {A} _ {i} ^ {\pi^ {k}} (s , \cdot)}{1 - \gamma} \right\}.
$$
Then, for any  $\alpha > 0$ ,
$$
f ^ {k} (\alpha) \geq f ^ {k} (\infty) \left[ 1 - \frac {1}{c \left(\exp \left(\frac {\delta^ {k} \alpha}{1 - \gamma}\right) - 1\right) + 1} \right],
$$
with  $c$  and  $\delta^k$  defined in Theorem 3.6.
Proof. For any  $i \in [n]$ , we have
$$
\begin{array}{l} f ^ {k} (\alpha) = \sum_ {i \in [ n ]} \frac {\sum_ {a _ {i}} \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i}) \pi_ {i} ^ {k} (a _ {i} | s) \exp \left\{\frac {\alpha \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i})}{1 - \gamma} \right\}}{\sum_ {a _ {i}} \pi_ {i} ^ {k} (a _ {i} | s) \exp \left\{\frac {\alpha \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i})}{1 - \gamma} \right\}} \\ \geq \sum_ {i \in [ n ]} \frac {\bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i _ {p}} ^ {k}) \left(\sum_ {a _ {j} \in \mathcal {A} _ {i _ {p}} ^ {k}} \pi_ {i} ^ {k} (a _ {j} | s)\right) \exp \left\{\frac {\alpha \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i _ {p}} ^ {k})}{1 - \gamma} \right\} + \sum_ {a _ {j} \in \mathcal {A} _ {i} \setminus \mathcal {A} _ {i _ {p}} ^ {k}} \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {j}) \pi_ {i} ^ {k} (a _ {j} | s) \exp \left\{\frac {\alpha \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i _ {q}} ^ {k})}{1 - \gamma} \right\}}{\left(\sum_ {a _ {j} \in \mathcal {A} _ {i _ {p}} ^ {k}} \pi_ {i} ^ {k} (a _ {j} | s)\right) \exp \left\{\frac {\alpha \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i _ {p}} ^ {k})}{1 - \gamma} \right\}} + \sum_ {a _ {j} \in \mathcal {A} _ {i} \setminus \mathcal {A} _ {i _ {p}} ^ {k}} \pi_ {i} ^ {k} (a _ {j} | s) \exp \left\{\frac {\alpha \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i _ {q}} ^ {k})}{1 - \gamma} \right\}} \\ = \sum_ {i \in [ n ]} \frac {\bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i _ {p}} ^ {k}) \left(\sum_ {a _ {j} \in \mathcal {A} _ {i _ {p}} ^ {k}} \pi_ {i} ^ {k} (a _ {j} | s)\right) \exp \left\{\frac {\alpha (\bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i _ {p}} ^ {k}) - \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i _ {q}} ^ {k}))}{1 - \gamma} \right\} + \sum_ {a _ {j} \in \mathcal {A} _ {i} \backslash \mathcal {A} _ {i _ {p}} ^ {k}} \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {j}) \pi_ {i} ^ {k} (a _ {j} | s)}{\left(\sum_ {a _ {j} \in \mathcal {A} _ {i _ {p}} ^ {k}} \pi_ {i} ^ {k} (a _ {j} | s)\right) \exp \left\{\frac {\alpha (\bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i _ {p}} ^ {k}) - \bar {A} _ {i i} ^ {\pi^ {k}} (s , a _ {i _ {q}} ^ {k}))}{1 - \gamma} \right\} + \sum_ {a _ {j} \in \mathcal {A} _ {i} \backslash \mathcal {A} _ {i _ {p}} ^ {k}} \pi_ {i} ^ {k} (a _ {j} | s)} \\ = \sum_ {i \in [ n ]} \left[ \bar {A} _ {i} ^ {\pi^ {k}} (s, a _ {i _ {p}} ^ {k}) - \frac {\sum_ {a _ {j} \in \mathcal {A} _ {i} \setminus \mathcal {A} _ {i _ {p}} ^ {k}} (\bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i _ {p}} ^ {k}) - \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {j})) \pi_ {i} ^ {k} (a _ {j} | s)}{\left(\sum_ {a _ {j} \in \mathcal {A} _ {i _ {p}} ^ {k}} \pi_ {i} ^ {k} (a _ {j} | s)\right) \exp \left\{\frac {\alpha (\bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i _ {p}} ^ {k}) - \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i _ {q}} ^ {k}))}{1 - \gamma} \right\} + \sum_ {a _ {j} \in \mathcal {A} _ {i} \setminus \mathcal {A} _ {i _ {p}} ^ {k}} \pi_ {i} ^ {k} (a _ {j} | s)} \right] \\ \end{array}
$$
$$
\begin{array}{l} \geq \sum_ {i \in [ n ]} \bar {A} _ {i} ^ {\pi^ {k}} (s, a _ {i _ {p}} ^ {k}) \left[ 1 - \frac {1}{c \left(\exp \left(\frac {\delta^ {k} \alpha}{1 - \gamma}\right) - 1\right) + 1} \right] \\ \geq f ^ {k} (\infty) \left[ 1 - \frac {1}{c \left(\exp \left(\frac {\delta^ {k} \underline {{\alpha}}}{1 - \gamma}\right) - 1\right) + 1} \right], \\ \end{array}
$$
where the first inequality holds due to Lemma D.1.
Combining the above lemmas, we show the proof of Theorem 3.6.
Proof of Theorem 3.6. Choose  $\alpha = \eta = \frac{(1 - \gamma)^2}{2\sqrt{n}\phi_{max}}$  in Lemma B.3,
$$
\begin{array}{l} \mathrm {N E - g a p} (\pi^ {k}) \leq \frac {1}{1 - \gamma} \sum_ {i, s} d _ {\rho} ^ {\pi_ {i} ^ {* k}, \pi_ {- i} ^ {k}} (s) \max  _ {a _ {i}} \bar {A} _ {i} ^ {\pi^ {k}} (s, a _ {i}) \\ \leq \frac {1}{1 - \gamma} \sum_ {i, s} \| \frac {d _ {\rho} ^ {\pi_ {i} ^ {* k}} , \pi_ {- i} ^ {k}}{d _ {\rho} ^ {\pi^ {k + 1}}} \| _ {\infty} d _ {\rho} ^ {\pi^ {k + 1}} (s) \max  _ {a _ {i}} \bar {A} _ {i} ^ {\pi^ {k}} (s, a _ {i}) \\ \leq \frac {1}{1 - \gamma} M \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \sum_ {i} \max  _ {a _ {i}} \bar {A} _ {i} ^ {\pi^ {k}} (s, a _ {i}) \\ \leq \frac {1}{1 - \gamma} M \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \lim  _ {\alpha \rightarrow \infty} f ^ {k} (\alpha) \\ \leq \frac {1}{1 - \gamma} M \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \frac {1}{1 - \frac {1}{c (\exp \left(\frac {\delta^ {k} \eta}{1 - \gamma}\right) - 1) + 1}} f ^ {k} (\eta) \\ \leq \frac {1}{2 (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \sum_ {i = 1} ^ {n} \sum_ {a _ {i}} \pi_ {i} ^ {k + 1} (a _ {i} | s) \bar {A} _ {i} ^ {\pi^ {k}} (s, a _ {i}) 2 M (1 + \frac {1 - \gamma}{c \delta^ {k} \eta}) \\ \leq \left(\Phi^ {\pi^ {k + 1}} (\rho) - \Phi^ {\pi^ {k}} (\rho)\right) 2 M \left(1 + \frac {2 \sqrt {n} \phi_ {m a x}}{c \delta^ {k} (1 - \gamma)}\right) \\ \leq \left(\Phi^ {\pi^ {k + 1}} (\rho) - \Phi^ {\pi^ {k}} (\rho)\right) 2 M \left(1 + \frac {2 \sqrt {n} \phi_ {m a x}}{c \delta_ {K} (1 - \gamma)}\right). \\ \end{array}
$$
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathrm {N E - g a p} (\pi^ {k}) \leq \frac {2 M (\Phi^ {\pi^ {K}} (\rho) - \Phi^ {\pi^ {0}} (\rho))}{K} (1 + \frac {2 \sqrt {n} \phi_ {m a x}}{c \delta_ {K} (1 - \gamma)}) \leq \frac {2 M \phi_ {m a x}}{K (1 - \gamma)} (1 + \frac {2 \sqrt {n} \phi_ {m a x}}{c \delta_ {K} (1 - \gamma)}).
$$
Finally, we show a corollary of Theorem 3.6.
Corollary B.3.1 (Restatement of Corollary 3.6.1). Consider a MPG that satisfies Assumption 3.2 with NPG update using algorithm 7. There exists  $K'$ , such that for any  $K \geq 1$ , choosing  $\eta = \frac{(1 - \gamma)^2}{2\sqrt{n}\phi_{max}}$ , we have
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} N E - g a p \left(\pi^ {k}\right) \leq \frac {2 M \phi_ {m a x}}{K (1 - \gamma)} \left(1 + \frac {4 \sqrt {n} \phi_ {m a x}}{c \delta^ {*} (1 - \gamma)} + \frac {K ^ {\prime}}{2 M}\right).
$$
Proof of Corollary 3.6.1. Following from the main paper, we know that there exists  $K'$  such that for all  $k \geq K'$ ,  $\delta^k \geq \frac{\delta^*}{2}$ .
From the proof of Theorem 3.6, we know that for MPGs, we get the following relationship:
$$
\mathrm {N E - g a p} (\pi^ {k}) \leq \left(\Phi^ {\pi^ {k + 1}} (\rho) - \Phi^ {\pi^ {k}} (\rho)\right) 2 M \left(1 + \frac {2 \sqrt {n} \phi_ {m a x}}{c \delta^ {k} (1 - \gamma)}\right).
$$
Therefore, for all  $k \geq K'$ ,
$$
\mathrm {N E - g a p} (\pi^ {k}) \leq \left(\Phi^ {\pi^ {k + 1}} (\rho) - \Phi^ {\pi^ {k}} (\rho)\right) 2 M \left(1 + \frac {4 \sqrt {n} \phi_ {m a x}}{c \delta^ {*} (1 - \gamma)}\right).
$$
Then, taking summation from  $K'$  to  $K - 1$ , we get
$$
\sum_ {k = K ^ {\prime}} ^ {K - 1} \mathrm {N E - g a p} \left(\pi^ {k}\right) \leq \frac {2 M \phi_ {m a x}}{1 - \gamma} \left(1 + \frac {4 \sqrt {n} \phi_ {m a x}}{c \delta^ {*} (1 - \gamma)}\right),
$$
Additionally, we know that  $\mathrm{NE - gap}(\pi^k)\leq \frac{\phi_{max}}{1 - \gamma}$  for all  $k$  by the definition of MPG. Combining the equations above, we have
$$
\begin{array}{l} \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathrm {N E - g a p} \left(\pi^ {k}\right) = \frac {1}{K} \left(\sum_ {k = 0} ^ {K ^ {\prime} - 1} \mathrm {N E - g a p} \left(\pi^ {k}\right) + \sum_ {k = K ^ {\prime}} ^ {K - 1} \mathrm {N E - g a p} \left(\pi^ {k}\right)\right) \\ \leq \frac {1}{K} \left(\frac {2 M \phi_ {\max }}{(1 - \gamma)} \left(1 + \frac {4 \sqrt {n} \phi_ {\max }}{c \delta^ {*} (1 - \gamma)}\right) + \frac {\phi_ {\max } K ^ {\prime}}{1 - \gamma}\right) \\ = \frac {2 M \phi_ {m a x}}{K (1 - \gamma)} \left(1 + \frac {4 \sqrt {n} \phi_ {m a x}}{c \delta^ {*} (1 - \gamma)} + \frac {K ^ {\prime}}{2 M}\right). \\ \end{array}
$$
C Extension to General MPG Formulation
In this section, we extend our results to a more general form of potential function.
Note that the detailed analysis is omitted for brevity. Proofs of lemmas and theorem in this section either exist in previous works or closely follow our analysis in Section B.
Definition C.1 (Alternate definition of MPG [8]). For a MPG, at any state  $s$ , there exists a global function  $\Phi^{\pi}(s):\Pi \times S\to \mathbb{R}$  such that
$$
V _ {i} ^ {\pi_ {i}, \pi_ {- i}} (s) - V _ {i} ^ {\pi_ {i} ^ {\prime}, \pi_ {- i}} (s) = \Phi^ {\pi_ {i}, \pi_ {- i}} (s) - \Phi^ {\pi_ {i} ^ {\prime}, \pi_ {- i}} (s) \tag {9}
$$
is true for any policies  $\pi_i, \pi_i' \in \Pi_i$ .
This definition of MPG differs from Definition 2.1 and does not enforce the existence of additional structure in function  $\phi$ . However, whether the two definitions are equivalent, similar to Lemma A.4, is still an open question to the best of the authors' knowledge.
We present the following Lemmas from [8], using which we provide a similar result to Lemma 3.5.
Lemma C.1 ([8]). For any function  $\Psi^{\pi}:\Pi \to \mathbb{R}$ , and any two policies  $\pi ,\pi^{\prime}\in \Pi$
$$
\begin{array}{l} \Psi^ {\pi^ {\prime}} - \Psi^ {\pi} = \sum_ {i = 1} ^ {N} \left(\Psi^ {\pi_ {i} ^ {\prime}, \pi_ {- i}} - \Psi^ {\pi}\right) \\ + \sum_ {i = 1} ^ {N} \sum_ {j = i + 1} ^ {N} \left(\Psi^ {\pi_ {<   i, i j}, \pi_ {> j} ^ {\prime}, \pi_ {i} ^ {\prime}, \pi_ {j} ^ {\prime}} - \Psi^ {\pi_ {<   i, i j}, \pi_ {> j} ^ {\prime}, \pi_ {i}, \pi_ {j} ^ {\prime}} - \Psi^ {\pi_ {<   i, i j}, \pi_ {> j} ^ {\prime}, \pi_ {i} ^ {\prime}, \pi_ {j}} + \Psi^ {\pi_ {<   i, i j}, \pi_ {> j} ^ {\prime}, \pi_ {i}, \pi_ {j}}\right) \\ \end{array}
$$
Lemma C.2 ([8]). Consider a two-player game (group of players) with a common-payoff Markov game with state space  $S$  and action sets  $\mathcal{A}_1, \mathcal{A}_2$ , with reward function  $r$ , transition function  $P$  and policy sets  $\Pi_1, \Pi_2$  be defined with respect to general MDPs. Then for any  $\pi_1, \pi_1' \in \Pi_1, \pi_2, \pi_2' \in \Pi_2$ , the for value function  $V^{\pi}(\mu)$ ,
$$
\begin{array}{l} V ^ {\pi_ {1}, \pi_ {2}} (\mu) - V ^ {\pi_ {1} ^ {\prime}, \pi_ {2}} (\mu) - V ^ {\pi_ {1}, \pi_ {2} ^ {\prime}} (\mu) + V ^ {\pi_ {1} ^ {\prime}, \pi_ {2} ^ {\prime}} (\mu) \\ \leq \frac {2 M \operatorname* {m a x} _ {i} | \mathcal {A} _ {i} |}{(1 - \gamma) ^ {4}} \sum_ {s} d _ {\mu} ^ {\pi_ {1} ^ {\prime}, \pi_ {2} ^ {\prime}} (s) \left(\| \pi_ {1} (\cdot | s) - \pi_ {1} ^ {\prime} (\cdot | s) \| ^ {2} + \| \pi_ {2} (\cdot | s) - \pi_ {2} ^ {\prime} (\cdot | s) \| ^ {2}\right) \\ \end{array}
$$
We refer to [8] for complete proof for Lemmas C.1 and C.2. By using Lemmas C.1 and C.2, we can replace Lemma 3.5 in our paper with the following lemma.
Lemma C.3. Given policy  $\pi^k$  and marginalized advantage function  $\bar{A}_i^{\pi^k}(s,a_i)$ , for  $\pi^{k + 1}$  generated using independent NPG update, we have the following inequality,
$$
\Phi^ {\pi^ {k + 1}} (\rho) - \Phi^ {\pi^ {k}} (\rho) \geq \left(\frac {1}{1 - \gamma} - \frac {2 M ^ {3} \max _ {i} | \mathcal {A} _ {i} | n \eta}{(1 - \gamma) ^ {3}}\right) \sum_ {s} d _ {\rho} ^ {\pi_ {i} ^ {k + 1}, \pi_ {- i} ^ {k}} (s) \sum_ {i = 1} ^ {n} \langle \pi_ {i} ^ {k + 1} (\cdot | s), \bar {A} _ {i} ^ {\pi^ {k}} (s, \cdot) \rangle .
$$
The proof of Lemma C.3 closely follows the proof of Lemma 3.5. First the potential function difference  $\Phi^{\pi^{k + 1}}(\rho) - \Phi^{\pi^k}(\rho)$  is decomposed following Lemma 3.5, with each term defined as a potential function that only differs by the local policy  $\pi_{i}$ . Different from 3.5, the decomposition is based on Definition C.1 instead. Then each term can be bounded using Lemma C.1 with the addition of Lemma C.2 to handle the residual terms.
Combining Lemma C.3 and Lemma B.3, we can get a similar convergence guarantee without the explicit definition of  $\phi$ , as stated in the following theorem:
Theorem C.4. Consider a MPG in the sense of Definition C.1 with isolated stationary points, where the policy update follows NPG update in (7). For any  $K \geq 1$ , choosing  $\eta = \frac{(1 - \gamma)^2}{4nM^3\max_i|\mathcal{A}_i|}$ , we have
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} N E \text {- g a p} (\pi^ {k}) \leq \frac {2 M \phi_ {\text {m a x}}}{K (1 - \gamma)} (1 + \frac {8 n M ^ {3} \max _ {i} | \mathcal {A} _ {i} |}{c \delta^ {*} (1 - \gamma)} + \frac {K ^ {\prime}}{2 M}).
$$
The proof of Theorem C.4 follows Theorem 3.6 and Corollary 3.6.1, and the only difference in the process is the different choice of  $\eta$  from Lemma C.3. Compared to Corollary 3.6.1, Theorem C.4 incorporates additional terms on  $M$  (distribution mismatch coefficient) and  $|\mathcal{A}_i|$  (size of action space), but the  $O(1 / \epsilon)$  order is still maintained.
Lemma D.1. Given a reward vector  $\mathbf{r} = [r_1, r_2, \dots, r_n]$  where  $r_1 > r_2 > r_3 > \dots > r_n$  with weights  $\pi_1 e^{\alpha r_1}, \pi_2 e^{\alpha r_2}, \dots, \pi_n e^{\alpha r_n}$ , we have
$$
\frac {\sum_ {i = 1} ^ {n} r _ {i} \pi_ {i} e ^ {\alpha r _ {i}}}{\sum_ {i = 1} ^ {n} \pi_ {i} e ^ {\alpha r _ {i}}} \geq \frac {r _ {1} \pi_ {1} e ^ {\alpha r _ {1}} + \sum_ {i = 2} ^ {n} r _ {i} \pi_ {i} e ^ {\alpha r _ {2}}}{\pi_ {1} e ^ {\alpha r _ {1}} + \sum_ {i = 2} ^ {n} \pi_ {i} e ^ {\alpha r _ {2}}}.
$$
Proof. The above claim can be proved by induction. We first have
$$
\begin{array}{l} \frac {\sum_ {i = 1} ^ {n} r _ {i} \pi_ {i} e ^ {\alpha r _ {i}}}{\sum_ {i = 1} ^ {n} \pi_ {i} e ^ {\alpha r _ {i}}} \geq \frac {\sum_ {i = 1} ^ {n - 1} r _ {i} \pi_ {i} e ^ {\alpha r _ {i}} + r _ {n} \pi_ {n} e ^ {\alpha r _ {n - 1}}}{\sum_ {i = 1} ^ {n - 1} \pi_ {i} e ^ {\alpha r _ {i}} + \pi_ {n} e ^ {\alpha r _ {n - 1}}} \\ = \frac {\sum_ {i = 1} ^ {n - 2} r _ {i} \pi_ {i} e ^ {\alpha r _ {i}} + \frac {r _ {n - 1} \pi_ {n - 1} + r _ {n} \pi_ {n}}{\pi_ {n - 1} + \pi_ {n}} (\pi_ {n - 1} + \pi_ {n}) e ^ {\alpha r _ {n - 1}}}{\sum_ {i = 1} ^ {n - 2} \pi_ {i} e ^ {\alpha r _ {i}} + (\pi_ {n - 1} + \pi_ {n}) e ^ {\alpha r _ {n - 1}}}. \\ \end{array}
$$
This gives a new reward vector
$$
r _ {1} ^ {\prime} = r _ {1} > r _ {2} ^ {\prime} = r _ {2} > \dots > r _ {n - 2} ^ {\prime} = r _ {n - 2} > r _ {n - 1} ^ {\prime} = \frac {r _ {n - 1} \pi_ {n - 1} + r _ {n} \pi_ {n}}{\pi_ {n - 1} + \pi_ {n}},
$$
$$
\pi_ {1} e ^ {\alpha r _ {1}}, \pi_ {2} e ^ {\alpha r _ {2}}, \dots , \pi_ {n - 2} e ^ {\alpha r _ {n - 2}}, (\pi_ {n - 1} + \pi_ {n}) e ^ {\alpha r _ {n - 1}}.
$$
(a) NE-gap of matrix game with varying  $\delta^{*}$
(b)  $L_{1}$  accuracy of matrix game with varying  $\delta^{*}$
Define  $\pi_{n - 1}^{\prime}\coloneqq \pi_{n - 1} + \pi_{n}$  and by similar argument as above,
$$
\begin{array}{l} R H S = \frac {\sum_ {i = 1} ^ {n - 2} r _ {i} \pi_ {i} e ^ {\alpha r _ {i}} + r _ {n - 1} ^ {\prime} \pi_ {n - 1} ^ {\prime} e ^ {\alpha r _ {n - 1}}}{\sum_ {i = 1} ^ {n - 2} \pi_ {i} e ^ {\alpha r _ {i}} + \pi_ {n - 1} ^ {\prime} e ^ {\alpha r _ {n - 1}}} \\ \geq \frac {\sum_ {i = 1} ^ {n - 3} r _ {i} \pi_ {i} e ^ {\alpha r _ {i}} + \frac {r _ {n - 2} \pi_ {n - 2} + r _ {n - 1} ^ {\prime} \pi_ {n - 1} ^ {\prime}}{\pi_ {n - 2} + \pi_ {n - 1} ^ {\prime}} (\pi_ {n - 2} + \pi_ {n - 1} ^ {\prime}) e ^ {\alpha r _ {n - 2}}}{\sum_ {i = 1} ^ {n - 3} \pi_ {i} e ^ {\alpha r _ {i}} + (\pi_ {n - 2} + \pi_ {n - 1} ^ {\prime}) e ^ {\alpha r _ {n - 2}}} \\ = \frac {\sum_ {i = 1} ^ {n - 3} r _ {i} \pi_ {i} e ^ {\alpha r _ {i}} + \frac {r _ {n - 2} \pi_ {n - 2} + r _ {n - 1} \pi_ {n - 1} + r _ {n} \pi_ {n}}{\pi_ {n - 2} + \pi_ {n - 1} + \pi_ {n}} (\pi_ {n - 2} + \pi_ {n - 1} + \pi_ {n}) e ^ {\alpha r _ {n - 2}}}{\sum_ {i = 1} ^ {n - 3} \pi_ {i} e ^ {\alpha r _ {i}} + (\pi_ {n - 2} + \pi_ {n - 1} + \pi_ {n}) e ^ {\alpha r _ {n - 2}}}. \\ \end{array}
$$
This gives a reward vector
$$
r _ {1} ^ {\prime \prime} = r _ {1} > r _ {2} ^ {\prime \prime} = r _ {2} > \dots > r _ {n - 2} ^ {\prime \prime} = \frac {r _ {n - 2} \pi_ {n - 2} + r _ {n - 1} \pi_ {n - 1} + r _ {n} \pi_ {n}}{\pi_ {n - 2} + \pi_ {n - 1} + \pi_ {n}},
$$
$$
\pi_ {1} e ^ {\alpha r _ {1}}, \pi_ {2} e ^ {\alpha r _ {2}}, \dots , \pi_ {n - 3} e ^ {\alpha r _ {n - 3}}, (\pi_ {n - 2} + \pi_ {n - 1} + \pi_ {n}) e ^ {\alpha r _ {n - 2}}.
$$
By induction, we have
$$
R H S \geq \frac {r _ {1} \pi_ {1} e ^ {\alpha r _ {1}} + \sum_ {i = 2} ^ {n} r _ {i} \pi_ {i} e ^ {\alpha r _ {2}}}{\pi_ {1} e ^ {\alpha r _ {1}} + \sum_ {i = 2} ^ {n} \pi_ {i} e ^ {\alpha r _ {2}}}.
$$