SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\pi_ {i} ^ {k + 1} \left(a _ {i} \mid s\right) \propto \pi_ {i} ^ {k} \left(a _ {i} \mid s\right) \exp \left(\frac {\eta \bar {A} _ {i} ^ {\pi^ {k}} (s , a _ {i})}{1 - \gamma}\right), \tag {7}
$$
where  $\bar{A}_i$  is defined in (2).
Different agents in a MPG do not share reward functions in general, which makes it difficult to compare evaluations of gradients across agents. However, with the introduction of Lemma 3.4, we find that MPGs have similar properties as fully cooperative games with a shared reward function. This enables us to establish relationships between policy updates of all agents. We first define  $h_i(s, \pmb{a}) \coloneqq r_i(s, \pmb{a}) - \phi(s, \pmb{a})$ , which implies  $V_i^\pi(s) = \Phi^\pi(s) + V_{h_i}^\pi(s)$ .
Lemma 3.4. Define  $\bar{A}_{h_i}^{\pi}(s,a_i)$  with respect to  $h_i$  similar to (2). We then have
$$
\sum_ {a _ {i}} \left(\pi_ {i} ^ {\prime} (a _ {i} | s) - \pi_ {i} (a _ {i} | s)\right) \bar {A} _ {h _ {i}} ^ {\pi} (s, a _ {i}) = 0, \quad \forall s \in \mathcal {S}, i \in [ n ], \pi_ {i} ^ {\prime}, \pi_ {i} \in \Delta (\mathcal {A} _ {i}).
$$
Lemma 3.4 shows a unique property of function  $h_i$ , where the expectation of the marginalized advantage function over every local policy  $\pi_i'$  yields the same effect. This property is directly associated with the MPG problem structure and is later used in Lemma 3.5. Next, we introduce the following assumption on the state visitation distribution, which is crucial and standard for studying the Markov dynamics of the system.
Assumption 3.1 ([1, 38, 8]). The Markov potential game  $\mathcal{M}$  satisfies:  $\inf_{\pi} \min_{s} d_{\rho}^{\pi}(s) > 0$ .
Similar to potential games, when the potential function  $\phi$  of a MPG is bounded, the marginalized advantage function  $\bar{A}_i$  for two policies can be bounded by the total variation between the policies.
Additionally, similar to Lemma 3.1, we present a lower bound in the following lemma for the potential function difference in two consecutive rounds.
Lemma 3.5. Given policy  $\pi^k$  and marginalized advantage function  $\bar{A}_i^{\pi^k}(s,a_i)$ , for  $\pi^{k + 1}$  generated using NPG update in (7), we have the following inequality,
$$
\Phi^ {\pi^ {k + 1}} (\rho) - \Phi^ {\pi^ {k}} (\rho) \geq \left(\frac {1}{1 - \gamma} - \frac {\sqrt {n} \phi_ {m a x} \eta}{(1 - \gamma) ^ {3}}\right) \sum_ {s} d _ {\rho} ^ {\pi^ {k + 1}} (s) \sum_ {i = 1} ^ {n} \langle \pi_ {i} ^ {k + 1} (\cdot | s), \bar {A} _ {i} ^ {\pi^ {k}} (s, \cdot) \rangle .
$$
Thus, using a function  $f$  adapted from (5) as a connection, we are able to establish the convergence of NE-gap for MPGs in the following theorem.
Theorem 3.6. Consider a MPG with isolated stationary points and the policy update following NPG update (7). For any  $K \geq 1$ , choosing  $\eta = \frac{(1 - \gamma)^2}{2\sqrt{n}\phi_{max}}$ , we have
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} N E \text {- g a p} (\pi^ {k}) \leq \frac {2 M \phi_ {\max}}{K (1 - \gamma)} (1 + \frac {2 \sqrt {n} \phi_ {\max}}{c \delta_ {K} (1 - \gamma)}),
$$
$$
M := \sup  _ {\pi} \max  _ {s} \frac {1}{d _ {\rho} ^ {\pi} (s)},
$$
$$
c := \inf _ {i \in [ n ], s \in \mathcal {S}, k \geq 0} \left(\sum_ {a _ {j} \in \mathcal {A} _ {i p} ^ {k}} \pi_ {i} ^ {k} (a _ {j} | s)\right) \in (0, 1),
$$
$$
\delta^ {k} := \min  _ {i \in [ n ], s \in \mathcal {S}} \left[ \bar {A} _ {i} ^ {\pi^ {k}} (s, a _ {i _ {p}} ^ {k}) - \bar {A} _ {i} ^ {\pi^ {k}} (s, a _ {i _ {q}} ^ {k}) \right], \quad a n d \quad \delta_ {K} = \min  _ {0 \leq k \leq K - 1} \delta^ {k} \in (0, 1),
$$
similar to (6).
Here, isolated implies that no other stationary points exist in any sufficiently small open neighborhood of any stationary point. This convergence result is similar to that provided in Theorem 3.3 for potential games. We note that our theorem also applies to an alternate definition for MPGs in works such as [8], which we discuss in Appendix C. Compared to potential games, the major difference is the introduction of  $M$ , which measures the distribution mismatch in the system. Generally, Assumption 3.1 implies a finite value for  $M$  in the MPG setup.
Discussion and Comparison on Convergence Rate Compared to the iteration complexity of previous works listed in Table 1, the convergence rate in Theorem 3.6 presents multiple improvements. Most importantly, the theorem guarantees that the averaged NE-gap reaches  $\epsilon$  in  $\mathcal{O}(1 / \epsilon)$  iterations, improving the best previously known result of  $\mathcal{O}\big(1 / \epsilon^2\big)$  in Table 1. Furthermore, this rate of convergence does not depend on the size of action space  $|\mathcal{A}_i|$ , and it has milder dependence on other system parameters, such as the distribution mismatch coefficient  $M$  and  $(1 - \gamma)$ . Note that many of the parameters above could be arbitrarily large in practice (e.g.,  $1 - \gamma = 0.01$  in our congestion game experiment). Theorem 3.6 indicates a tighter convergence bound with respect to the discussed factors in general.
3.3 The Consideration of Suboptimality Gap
In Theorems 3.3 and 3.6, we established the ergodic convergence rates of NPG in potential game and Markov potential game settings, which depend on  $1 / \delta_{K}$ . In these results,  $\delta^k$  generally encapsulates
the difference between the gradients evaluated at the best and second best policies, and  $\delta_K$  is a lower bound on  $\delta^k$ . We refer to  $\delta^k$  as the suboptimality gap at iteration  $k$ . In our analysis, the suboptimality gap provides a vehicle to establish the improvement of the potential function in two consecutive steps. In particular, it enables us to draw a connection between  $\Phi^{\pi^{k+1}} - \Phi^{\pi^k}$  and NE-gap  $(\pi^k)$  using Lemma 3.2 and Lemma B.3 in the appendix. On the contrary, most of the existing work does not rely on this approach and generally studies the relationship between  $\Phi^{\pi^{k+1}} - \Phi^{\pi^k}$  and the squared NE-gap  $(\pi^k)$ , which suffers a slower convergence rate of  $\mathcal{O}(1/\sqrt{K})$  [38].
The analysis leveraging the suboptimality gap, though has not been adopted in the MARL studies, was considered in the single-agent scenario. Khodadadian et al. [16] proved asymptotic geometric convergence of single-agent RL with the introduction of optimal advantage function gap  $\Delta^k$ , which shares similar definition as the gap  $\delta^k$  studied in this work. Moreover, the notion of suboptimality gap is commonly used in the multi-armed bandit literature [18], so as to give the instance-dependent analysis.
While our convergence rate is sharper, a lower bound on the suboptimality gap is generally not guaranteed, and scenarios with zero optimality gap can be constructed in MPGs. In practice, we find that even if  $\delta^k$  approaches zero in certain iterations, it may not converge to zero, and in these scenarios the system still converges without a slowdown. We provide a numerical example in Section 4.1 (Fig. 1a) to support our claim. Nevertheless, in what follows, we further identify a sufficient condition that allows us to alleviate this problem altogether by focusing on the asymptotic profile of the sub-optimality gap.
Theoretical Relaxation The following proposition guarantees the asymptotic results of independent NPG.
Proposition 3.1 ([38]). Suppose that Assumption 3.1 holds and that the stationary policies are isolated. Then independent NPG with  $\eta \leq \frac{(1 - \gamma)^2}{2\sqrt{n}\phi_{max}}$  guarantees that  $\lim_{k\to \infty}\pi^k = \pi^\infty$ , where  $\pi^\infty$  is a Nash policy.
Since asymptotic convergence of policy is guaranteed by Proposition 3.1, the suboptimality gap  $\delta^k$  is also guaranteed to converge to some  $\delta^*$ . We make the following assumption about the asymptotic suboptimality gap, which only depends on the property of the game itself.
Assumption 3.2. Assume that  $\lim_{k\to \infty}\delta^k = \delta^* >0$
Assumption 3.2 provides a relaxation in the sense that instead of requiring a lower bound  $\inf \delta^k$  for all  $k$ , we only need  $\delta^{*} > 0$ , a lower bound on the limit as the agents approach some Nash policies. This will allow us to disregard the transition behavior of the system and focus on the rate for large enough  $k$ .
By the definition of  $\delta^{*}$ , we know that there exists finite  $K'$  such that  $\forall k > K'$ ,  $|\delta^k - \delta^*| \leq \frac{\delta^*}{2}$ ,  $\delta^k \geq \frac{\delta^*}{2}$ . Using these results, we can rework the proofs of Theorems 3.3 and 3.6 to get the following corollary.
Corollary 3.6.1. Consider a MPG that satisfies Assumption 3.2 with NPG update using algorithm 7. There exists  $K'$ , such that for any  $K \geq 1$ , choosing  $\eta = \frac{(1 - \gamma)^2}{2\sqrt{n}\phi_{max}}$ , we have
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} N E - g a p (\pi^ {k}) \leq \frac {2 M \phi_ {m a x}}{K (1 - \gamma)} \left(1 + \frac {4 \sqrt {n} \phi_ {m a x}}{c \delta^ {*} (1 - \gamma)} + \frac {K ^ {\prime}}{2 M}\right),
$$
where  $M, c$  are defined as in Theorem 3.6.
In previous sections, we established the theoretical convergence of NPG in MPGs. In order to verify the results, we construct two experimental settings for the NPG update and compare our empirical results with existing algorithms. We consider a synthetic potential game scenario with randomly generated rewards and a congestion problem studied in [19] and [8]. We also provide the source code<sup>4</sup> for all experiments.
<sup>4</sup>https://github.com/sundave1998/Independent-NPG-MPG
(b)
Figure 1: (a) The suboptimality gap; (b) Learning curve in synthetic experiments; (c) Learning curve for congestion game.
4.1 Synthetic Potential Games
We first construct a potential game with a fully cooperative objective, where the reward tensor  $r(\pmb{a})$  is randomly generated from a uniform distribution. We set the agent number to  $n = 3$ , each with different action space  $|\mathcal{A}_1| = 3, |\mathcal{A}_2| = 4, |\mathcal{A}_3| = 5$ . At the start of the experiment, all agents are initialized with uniform policies. We note that the experimental setting in [38] falls under this broad setting, although the experiment therein was a two-player game with carefully designed rewards.
The results are shown in Figure 1b. We compare the performance of independent NPG with other commonly studied policy updates, such as projected Q ascent [8], entropy regularized NPG [38] as well as log-barrier regularized NPG method [38]. We set the learning rate of all algorithms as  $\eta = 0.1$ . We also fine-tune regularization parameters to find the entropy regularization factor  $\tau = 0.05$  and the log-barrier regularization factor  $\lambda = 0.005$ . As discussed in Section 3.1 (the effect of  $c$ ), we observe in Figure 1b that the regularized algorithms fail to reach an exact Nash policy despite exhibiting good convergence performance at the start. The empirical results here align with our theoretical findings in Section 3.
Impact of  $c$  and  $\delta$  We demonstrate the impact of the initialization dependent factor  $c^k$  and suboptimality gap  $\delta^k$  in MPGs with the same experimental setup. Figure 1a depicts the change in value for  $c^k$ ,  $\delta^k$ , and the NE-gap. We can see from the figure that as the algorithm updates over iterations, the value of  $c^k$  increases and approaches one, while  $\delta^k$  approaches some non-zero constant. Figure 1a shows that although  $\delta^k$  could be arbitrarily small in theory if we do not impose technical assumptions, it is not the case in general. Therefore, one should not automatically assume that the suboptimality gap diminishes with respect to iteration, as the limit entirely depends on the problem environment.
The effect of suboptimality gap  $\delta^{*}$  in PG is illustrated in Section E of the appendix under a set of carefully constructed numerical examples. We verify that a larger gap indicates a faster convergence, which corroborates our theory.
4.2 Congestion Game
We now consider a class of MDPs where each state defines a congestion game. We borrow the specific settings for this experiment from [19] [8].
For the congestion game experiments, we consider the agent number  $n = 8$  with the number of facilities  $|\mathcal{A}_i| = 4$ , where  $\mathcal{A}_i = \{A, B, C, D\}$  as the corresponding individual action spaces. There are two states defined as  $S = \{\text{safe}, \text{distancing}\}$ . In each state, all agents prefer to be taking the same action with as many agents as possible. The reward for an agent selecting action  $k$  is defined by predefined weights  $w_s^k$  multiplied by the number of other agents taking the same action. Additionally, we set  $w_s^A < w_s^B < w_s^C < w_s^D$  and the reward in distancing state is reduced by some constant compared to the safe state. The state transition depends on the joint actions of all agents. If more than half of all agents take the same action, the system enters a distancing state with lower rewards. If the agents are evenly distributed over all actions, the system enters safe state with higher rewards.
We use episodic updates with  $T = 20$  steps and collect 20 trajectories in each mini-batch and estimate the value function and Q-functions as well as the discounted visitation distribution. We use a discount factor of  $\gamma = 0.99$ . We adopt the same step size used in [19, 8] and determine optimal step-sizes of
softmax PG and NPG with grid-search. Since regularized methods in Section 4.1 generally do not converge to Nash policies, they are excluded in this experiment. To make the experiment results align with previous works, we provide the  $L_{1}$  distance between the current-iteration policies compared to Nash policies. We plot the mean and variance of  $L_{1}$  distance across multiple runs in Figure 1c. Compared to the direct parameterized algorithms, the two softmax parameterized algorithms exhibit faster convergence, and softmax parameterized NPG has the best performance across all tested algorithms.
In this paper, we studied Markov potential games in the context of multi-agent reinforcement learning. We focused on the independent natural policy gradient algorithm and studied its convergence rate to the Nash equilibrium. The main theorem of the paper shows that the convergence rate of NPG in MPGs is  $\mathcal{O}(1 / K)$ , which improves upon the previous results. Additionally, we provided detailed discussions on the impact of some problem factors (e.g.,  $c$  and  $\delta$ ) and compared our rate with the best known results with respect to these factors. Two empirical results were presented as a verification of our analysis.
Despite our newly proposed results, there are still many open problems that need to be addressed. One of the limitations of this work is the assumption of Markov potential games, the relaxation of which could extend our analysis to more general stochastic games. As a matter of fact, the gradient-based algorithm studied in this work will fail for a zero-sum game as simple as Tic-Tac-Toe. A similar analysis could also be applied to regularized games and potentially sharper bounds could be obtained. The agents are also assumed to receive gradient information from an oracle in this paper. When such oracle is unavailable, the gradient can be estimated via trajectory samples, which we leave as a future work. Other future directions are the convergence analysis of policy gradient-based algorithms in safe MARL and robust MARL, following the recent exploration of safe single-agent RL [9, 22, 41] and robust single-agent RL [21, 40].
This material is based upon work partially supported by the US Army Contracting Command under W911NF-22-1-0151 and W911NF2120064, US National Science Foundation under CMMI-2038625, and US Office of Naval Research under N00014-21-1-2385. The views expressed herein and conclusions contained in this document are those of the authors and should not be interpreted as representing the views or official policies, either expressed or implied, of the U.S. NSF, ONR, ARO, or the United States Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
[1] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431-4506, 2021.
[2] Jupiter Bakakeu, Schirin Baer, Hans-Henning Klos, Joern Peschke, Matthias Brossog, and Joerg Franke. Multi-agent reinforcement learning for the energy optimization of cyber-physical production systems. Artificial Intelligence in Industry 4.0: A Collection of Innovative Research Case-studies that are Reworking the Way We Look at Industry 4.0 Thanks to Artificial Intelligence, pages 143–163, 2021.
[3] Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156-172, 2008.
[4] Shicong Cen, Fan Chen, and Yuejie Chi. Independent natural policy gradient methods for potential games: Finite-time global convergence with entropy regularization. In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 2833-2838. IEEE, 2022.
A Proof for Potential Games in Section 3.1
We first find the upper bound on the difference in marginalized rewards under two policies. The following lemma establishes a "smooth" relationship between the marginalized rewards and their respective policies. For ease of notation, we denote  $r(\pi) \coloneqq \mathbb{E}_{\pmb{a} \sim \pi}[r(\pmb{a})]$  and  $\phi(\pi) \coloneqq \mathbb{E}_{\pmb{a} \sim \pi}[\phi(\pmb{a})]$ , where these function values are taken expectation over policy  $\pi$ .
Lemma A.1. For any two sets of policies  $\pi, \pi' \in \Delta(\mathcal{A}_1) \times \ldots \times \Delta(\mathcal{A}_n)$ , the difference in marginalized reward of any agent  $i$  is bounded by the total variation distance of policies,
$$
\left\| \bar {r} _ {i} ^ {\pi} - \bar {r} _ {i} ^ {\pi^ {\prime}} \right\| _ {\infty} \leq 2 T V \left(\pi_ {- i}, \pi_ {- i} ^ {\prime}\right).
$$
Proof. Given any  $\pi, \pi' \in \Delta(\mathcal{A})$ , we have
$$
\begin{array}{l} | \bar {r} _ {i} ^ {\pi} (a _ {i}) - \bar {r} _ {i} ^ {\pi^ {\prime}} (a _ {i}) | = | \mathbb {E} _ {a _ {- i} \sim \pi_ {- i}} [ r _ {i} (a _ {i}, a _ {- i}) ] - \mathbb {E} _ {a _ {- i} \sim \pi_ {- i} ^ {\prime}} [ r _ {i} (a _ {i}, a _ {- i}) ] | \\ \leq \| r _ {i} \| _ {\infty} \| \pi_ {- i} - \pi_ {- i} ^ {\prime} \| _ {1} \leq 2 T V (\pi_ {- i}, \pi_ {- i} ^ {\prime}). \\ \end{array}
$$
Next, we provide the proof of Lemma 3.1, which demonstrates the lower bound of the potential function difference.
Lemma A.2 (Restatement of Lemma 3.1). Given policy  $\pi^k$  and marginalized reward  $\bar{r}_i^k(a_i)$ , for  $\pi^{k+1}$  generated using the NPG update in (4), we have the following inequality for any  $\eta < \frac{1}{\sqrt{n}}$ ,
Proof of Lemma 3.1. We introduce
$$
\tilde {\pi} _ {- i} ^ {k} \left(a _ {- i}\right) := \prod_ {j <   i} \pi_ {j} ^ {k} \left(a _ {j}\right) \prod_ {\ell > i} \pi_ {\ell} ^ {k + 1} \left(a _ {\ell}\right)
$$
to denote the mixed strategy where any agent with index  $j < i$  follows  $\pi_j^k$  and any agent with index  $\ell > i$  follows  $\pi_\ell^{k+1}$  instead. Let  $\tilde{r}_i^k$  be the associated marginalized reward function, i.e.,
$$
\tilde {r} _ {i} ^ {k} (a _ {i}) = \mathbb {E} _ {a _ {- i} \sim \tilde {\pi} _ {- i} ^ {k}} \left[ r _ {i} (\boldsymbol {a}) \right] = \sum_ {a _ {- i} \in \mathcal {A} _ {- i}} r _ {i} \left(a _ {i}, a _ {- i}\right) \prod_ {j <   i} \pi_ {j} ^ {k} \left(a _ {j}\right) \prod_ {\ell > i} \pi_ {\ell} ^ {k + 1} \left(a _ {\ell}\right).
$$
$$
\phi \left(\pi_ {i} ^ {k}, \tilde {\pi} _ {- i} ^ {k}\right) = \phi \left(\pi_ {1} ^ {k}, \dots , \pi_ {i} ^ {k}, \pi_ {i + 1} ^ {k + 1}, \dots , \pi_ {n} ^ {k + 1}\right) = \phi \left(\pi_ {i + 1} ^ {k + 1}, \tilde {\pi} _ {- (i + 1)} ^ {k}\right).
$$
We now decompose  $\phi^{k + 1} - \phi^k$  as follows:
$$
\begin{array}{l} \phi (\pi^ {k + 1}) - \phi (\pi^ {k}) = \phi (\pi_ {1} ^ {k + 1}, \tilde {\pi} _ {- 1} ^ {k}) - \phi (\pi_ {n} ^ {k}, \tilde {\pi} _ {- n} ^ {k}) \\ = \sum_ {i = 1} ^ {n} \phi \left(\pi_ {i} ^ {k + 1}, \tilde {\pi} _ {- i} ^ {k}\right) - \phi \left(\pi_ {i} ^ {k}, \tilde {\pi} _ {- i} ^ {k}\right) \\ = \sum_ {i = 1} ^ {n} r _ {i} \left(\pi_ {i} ^ {k + 1}, \tilde {\pi} _ {- i} ^ {k}\right) - r _ {i} \left(\pi_ {i} ^ {k}, \tilde {\pi} _ {- i} ^ {k}\right) \\ = \sum_ {i = 1} ^ {n} \langle \tilde {r} _ {i} ^ {k} - \bar {r} _ {i} ^ {k}, \pi_ {i} ^ {k + 1} - \pi_ {i} ^ {k} \rangle + \sum_ {i = 1} ^ {n} \langle \bar {r} _ {i} ^ {k}, \pi_ {i} ^ {k + 1} - \pi_ {i} ^ {k} \rangle . \\ \end{array}
$$
The third equality follows from the definition (3).
$$
\sum_ {i = 1} ^ {n} \left| \left\langle \tilde {r} _ {i} ^ {k} - \bar {r} _ {i} ^ {k}, \pi_ {i} ^ {k + 1} - \pi_ {i} ^ {k} \right\rangle \right| \leq 2 \sum_ {i = 1} ^ {n} T V \left(\tilde {\pi} _ {- i} ^ {k}, \pi_ {- i} ^ {k}\right) \| \pi_ {i} ^ {k + 1} - \pi_ {i} ^ {k} \| _ {1}
$$
$$
\begin{array}{l} \leq \frac {2}{\sqrt {n}} \sum_ {i = 1} ^ {n} T V \left(\tilde {\pi} _ {- i} ^ {k}, \pi_ {- i} ^ {k}\right) ^ {2} + \frac {\sqrt {n}}{2} \sum_ {i = 1} ^ {n} \| \pi_ {i} ^ {k + 1} - \pi_ {i} ^ {k} \| _ {1} ^ {2} \\ \leq \frac {1}{\sqrt {n}} \sum_ {i = 1} ^ {n} K L \left(\pi_ {- i} ^ {k} | | \tilde {\pi} _ {- i} ^ {k}\right) + \sqrt {n} \sum_ {i = 1} ^ {n} K L \left(\pi_ {i} ^ {k + 1} | | \pi_ {i} ^ {k}\right) \\ \leq \sqrt {n} \left[ K L \left(\pi^ {k} | | \pi^ {k + 1}\right) + K L \left(\pi^ {k + 1} | | \pi^ {k}\right) \right] \\ = \sqrt {n} \sum_ {i = 1} ^ {n} \left\langle \pi_ {i} ^ {k + 1} - \pi_ {i} ^ {k}, \log \pi_ {i} ^ {k + 1} - \log \pi_ {i} ^ {k} \right\rangle \\ = \eta \sqrt {n} \sum_ {i = 1} ^ {n} \left\langle \pi_ {i} ^ {k + 1} - \pi_ {i} ^ {k}, \bar {r} _ {i} ^ {k} \right\rangle , \\ \end{array}
$$
Additionally, we provide the proof of Lemma 3.2, which builds the relationship between potential function difference and NE-gap.
Lemma A.3 (Restatement of Lemma 3.2). For function  $f^{k}(\alpha)$  defined in (5) and any  $\alpha > 0$ , we have the following inequality.
Proof. Recall that we define  $a_{i_p}^k \in \arg \max_{a_j \in \mathcal{A}_i} \bar{r}_i^k(a_j) =: \mathcal{A}_{i_p}^k$  and  $a_{i_q}^k \in \arg \max_{a_j \in \mathcal{A}_i \setminus \mathcal{A}_{i_p}^k} \bar{r}_i^k(a_j)$ , where  $\mathcal{A}_{i_p}^k$  denotes the set of the best possible actions for agent  $i$ . For any  $i \in [n]$ , we have
$$
\geq \left\langle \pi_ {i} ^ {* k} - \pi_ {i} ^ {k}, \bar {r} _ {i} ^ {k} \right\rangle \left[ 1 - \frac {1}{c \left(\exp \left\{\delta^ {k} \alpha \right\} - 1\right) + 1} \right],
$$
where the first inequality holds due to Lemma D.1. By taking sum over all agents  $i \in [n]$ , we get
$$
\begin{array}{l} f ^ {k} (\alpha) = \sum_ {i \in [ n ]} \left\langle \pi_ {i, \alpha} - \pi_ {i} ^ {k}, \bar {r} _ {i} ^ {k} \right\rangle \\ \geq \sum_ {i \in [ n ]} \left\langle \pi_ {i} ^ {* k} - \pi_ {i} ^ {k}, \bar {r} _ {i} ^ {k} \right\rangle \left[ 1 - \frac {1}{c \left(\exp \left\{\delta^ {k} \alpha \right\} - 1\right) + 1} \right] \\ \geq f ^ {k} (\infty) \left[ 1 - \frac {1}{c (\exp \{\delta_ {K} \alpha \} - 1) + 1} \right]. \\ \end{array}
$$
We next provide the following lemma on the structure of potential games. This lemma is not directly used in the analysis of this paper but instead provides a discussion on the function class of potential games, which could be of separate interest.
Lemma A.4 (Structure of potential game reward function class). For any agent  $i$ , the set of reward functions for potential games,  $\mathcal{R}_i$ , has the following structure within the ambient space of  $\mathbb{R}^{\Pi_j|\mathcal{A}_j|}$ .
$$
\mathcal {R} _ {i} = \left\{\phi \left(a _ {i}, a _ {- i}\right) + g _ {i} \left(a _ {- i}\right): \forall g _ {i}: \prod_ {j \neq i} \mathcal {A} _ {j} \rightarrow \mathbb {R} \right\} \subset \mathbb {R} ^ {\Pi_ {j} | \mathcal {A} _ {j} |}. \tag {8}
$$