SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Youbang Sun*
Northeastern University sun.youb@northeastern.edu
Texas A&M University ruida@amu.edu
Texas A&M University prk@tamu.edu
Shahin Shahrampour
Northeastern University
s.shahrampour@northeastern.edu
This work studies an independent natural policy gradient (NPG) algorithm for the multi-agent reinforcement learning problem in Markov potential games. It is shown that, under mild technical assumptions and the introduction of the suboptimality gap, the independent NPG method with an oracle providing exact policy evaluation asymptotically reaches an  $\epsilon$ -Nash Equilibrium (NE) within  $\mathcal{O}(1/\epsilon)$  iterations. This improves upon the previous best result of  $\mathcal{O}(1/\epsilon^2)$  iterations and is of the same order,  $\mathcal{O}(1/\epsilon)$ , that is achievable for the single-agent case. Empirical results for a synthetic potential game and a congestion game are presented to verify the theoretical bounds.
Reinforcement learning (RL) is often impacted by the presence and interactions of several agents in a multi-agent system. This challenge has motivated recent studies of multi-agent reinforcement learning (MARL) in stochastic games [37, 3]. Applications of MARL include robotics [30], modern production systems [2], economic decision making [33], and autonomous driving [31]. Among the various types of stochastic games, we focus on a commonly studied model for MARL, known as Markov Potential Games (MPGs). MPGs are seen as a generalization of canonical Markov Decision Processes (MDPs) in the multi-agent setting. In MPGs, there exists a potential function that can track the value changes of all agents. Unlike single-agent systems, where the goal is to find the optimal policy, the objective in this paper is to find a global policy, formed by the joint product of a set of local policies, that leads the system to reach a Nash equilibrium (NE) [29], which is precisely defined in Section 2.
A major challenge in the analysis of multi-agent systems is the restriction on joint policies of agents. For single-agent RL, policy updates are designed to increase the probability of selecting the action with the highest reward. However, in multi-agent systems, the global policy is constructed by taking the product of local agents' policies, which makes MARL algorithms suffer a greater risk of being trapped near undesirable stationary points. Consequently, finding a NE in MARL is more challenging than finding the global optimum in the single-agent case, and it is therefore difficult for MPGs to recover the convergence rates of single-agent Markov decision processes (MDPs).
Additionally, the global action space in MPGs scales exponentially with the number of agents within the system, making it crucial to find an algorithm that scales well for a large number of agents. Recent
Table 1: Convergence rate results for policy gradient-based methods in Markov potential games. Some results have been modified to ensure better comparison.
ALGORITHM	ITERATION COMPLEXITY2
PG + DIRECT [39, 19]	O(∑i=1n|M2(1-γ)4ε2)
PG + SOFTMAX [38]	O(n max_i |A_i|M^2/(1-γ)4c^2ε^2)
NPG + SOFTMAX [38]	O(nM/(1-γ)3cε^2)
PG + SOFTMAX + LOG-BARRIER REG. [38]	O(n max_i |A_i|^2M^2/(1-γ)4ε^2)
NPG + SOFTMAX + LOG-BARRIER REG. [38]	O(n max_i |A_i|M^2/(1-γ)4ε^2)
PROJECTED Q ASCENT [8]	O(n^2 max_i |A_i|^2M^2/(1-γ)7ε^4) OR O(n max_i |A_i|M^4/(1-γ)2ε^2)
PROJECTED Q ASCENT (FULLY COOP) [8]	O(n max_i |A_i|M/(1-γ)3ε^2)
(Ours) NPG + SOFTMAX	O(√nM/(1-γ)^2cδ*ε)3
studies [8, 14] addressed the issue by an approach called independent learning, where each agent performs policy update based on local information without regard to policies of the other agents. Independent learning algorithms scale only linearly with respect to the number of agents and are therefore preferred for large-scale multi-agent problems.
Using algorithms such as policy gradient (PG) and natural policy gradient (NPG), single-agent RL can provably converge to the global optimal policy [1]. However, extending these algorithms from single-agent to multi-agent settings presents natural challenges as discussed above. Multiple recent works have analyzed PG and NPG in multi-agent systems. However, due to the unique geometry of the problem and complex relationships among the agents, the theoretical understanding of MARL is still limited, with most works showing slower convergence rates when compared to their single-agent counterparts (see Table 1).
Contributions We study the independent NPG algorithm in multi-agent systems and provide a novel technical analysis that guarantees a provably fast convergence rate. We start our analysis with potential games in Section 3.1 and then generalize the findings and provide a convergence guarantee for Markov potential games in Section 3.2. We show that under mild assumptions, the ergodic (i.e., temporal average of) NE-gap converges with iteration complexity of  $\mathcal{O}(1 / \epsilon)$  after a finite threshold (Theorem 3.6). This result provides a substantial improvement over the best known rate of  $\mathcal{O}\big(1 / \epsilon^2\big)$  in [38]. Our main theorem also reveals mild or improved dependence on multiple critical factors, including the number of agents  $n$ , the initialization dependent factor  $c$ , the distribution mismatch coefficient  $M$ , and the discount factor  $\gamma$ , discussed in Section 3. We dedicate Section 3.3 to discuss the impact of the asymptotic suboptimality gap  $\delta^{*}$ , which is a new factor in this work.
In addition to our theoretical results, two numerical experiments are also conducted in Section 4 for verification of the analysis. We consider a synthetic potential game similar to [38] and a congestion game from [19]. The omitted proofs of our theoretical results can be found in the appendix.
Markov Potential Games Since many properties of single-agent RL do not hold in MARL, the analysis of MARL presents several challenges. Various settings have been addressed for MARL in recent works. A major distinction between these works stems from whether the agents are competitive or cooperative [11]. In this paper we consider MPGs introduced in stochastic control [7]. MPGs are a generalized formulation of identical-reward cooperative games. Markov cooperative games have been studied in the early work of [34], and more recently by [23, 24, 32, 10]. The work [36]
${}^{2}c,M$  and  ${\delta }^{ * }$  will be formally defined in Section 3. Note that [8] has a different definition of  $M$  . For a fair comparison,we convert the results of [8] into the current definition of  $M$  .
This iteration complexity holds after a finite threshold.
also offered extensive empirical results for cooperative games using multi-agent proximal policy optimization. The work by [27] established polynomial convergence for MPGs under both Q-update as well as actor-critic update. [25] studied an independent Q-update in MPGs with perturbation, which converges to a stationary point with probability one. Conversely, [17, 13] studied potential games, a special static case of simplified MPGs with no state transition.
Policy Gradient in Games Policy gradient methods for centralized MDPs have drawn much attention thanks to recent advancements in RL theory [1, 26]. The extension of PG methods to multi-agent settings is quite natural. [6, 35, 5] studied two-player zero-sum competitive games. The general-sum linear-quadratic game was studied in [12]. [14] studied general-sum Markov games and provided convergence of V-learning in two-player zero-sum games.
Of particular relevance to our work are the works [19, 39, 38, 8] which focus on the MPG setting and propose adaptations of PG and NPG-based methods from single-agent problems to the MARL setting. Table 1 provides a detailed comparison between these works. The previous theoretical results in multi-agent systems have provided convergence rates dependent on different parameters of the system. However, the best-known iteration complexity to reach an  $\epsilon$ -NE in MARL is  $\mathcal{O}\big(1 / \epsilon^2\big)$ . Therefore, there still exists a rate discrepancy between MARL methods where  $\mathcal{O}(1 / \epsilon)$  complexity has been established in centralized RL algorithms [1]. Our main contribution is to close this gap by establishing an iteration complexity of  $\mathcal{O}(1 / \epsilon)$  in this work.
We consider a stochastic game  $\mathcal{M} = (n, S, \mathcal{A}, P, \{r_i\}_{i \in [n]}, \gamma, \rho)$  consisting of  $n$  agents denoted by a set  $[n] = \{1, \dots, n\}$ . The global action space  $\mathcal{A} = \mathcal{A}_1 \times \dots \times \mathcal{A}_n$  is the product of individual action spaces, with the global action defined as  $\pmb{a} := (a_1, \dots, a_n)$ . The global state space is represented by  $\mathcal{S}$ , and the system transition model is captured by  $P: S \times \mathcal{A} \to \Delta(S)$ . Furthermore, each agent is equipped with an individual reward function  $r_i: S \times \mathcal{A} \to [0, 1]$ . We use  $\gamma \in (0, 1)$  to denote the discount factor and  $\rho \in \Delta(S)$  to denote the initial state distribution.
The system policy is denoted by  $\pi : \mathcal{S} \to \Delta(\mathcal{A}_1) \times \dots \times \Delta(\mathcal{A}_n) \subset \Delta(\mathcal{A})$ , where  $\Delta(\mathcal{A})$  is the probability simplex over the global action space. In the multi-agent setting, all agents make decisions independently given the observed state, often referred to as a decentralized stochastic policy [38]. Under this setup, we have  $\pi(\boldsymbol{a}|s) = \prod_{i \in [n]} \pi_i(a_i|s)$ , where  $\pi_i : \mathcal{S} \to \Delta(\mathcal{A}_i)$  is the local policy for agent  $i$ . For the ease of notation, we denote the joint policy over the set  $[n] \setminus \{i\}$  by  $\pi_{-i} = \prod_{j \in [n] \setminus \{i\}} \pi_j$  and use the notation  $a_{-i}$  analogously.
We define the state value function  $V_{i}^{\pi}(s)$  with respect to the reward  $r_i(s, \mathbf{a})$  as  $V_{i}^{\pi}(s) \coloneqq \mathbb{E}^{\pi}[\sum_{t=0}^{\infty} \gamma^{t} r_{i}(s^{t}, \mathbf{a}^{t}) | s^{0} = s]$ , where  $(s^{t}, \mathbf{a}^{t})$  denotes the global state-action pair at time  $t$ , and we denote the expected value of the state value function over the initial state distribution  $\rho$  as  $V_{i}^{\pi}(\rho) \coloneqq \mathbb{E}_{s \sim \rho} [V_{i}^{\pi}(s)]$ . We can similarly define the state visitation distribution under  $\rho$  as  $d_{\rho}^{\pi}(s) \coloneqq (1 - \gamma) \mathbb{E}^{\pi}[\sum_{t=0}^{\infty} \gamma^{t} \mathbb{1}(s_{t} = s) | s_{0} \sim \rho]$ , where  $\mathbb{1}$  is the indicator function. The state-action value function and advantage function are, respectively, given by
$$
Q _ {i} ^ {\pi} (s, \boldsymbol {a}) = \mathbb {E} ^ {\pi} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r _ {i} \left(s ^ {t}, \boldsymbol {a} ^ {t}\right) \mid s ^ {0} = s, \boldsymbol {a} ^ {0} = \boldsymbol {a} \right], A _ {i} ^ {\pi} (s, \boldsymbol {a}) = Q _ {i} ^ {\pi} (s, \boldsymbol {a}) - V _ {i} ^ {\pi} (s). \tag {1}
$$
For the sake of analysis, we further define the marginalized Q-function and advantage function  $\bar{Q}_i: S \times \mathcal{A}_i \to \mathbb{R}$  and  $\bar{A}_i: S \times \mathcal{A}_i \to \mathbb{R}$  as:
$$
\bar {Q} _ {i} ^ {\pi} (s, a _ {i}) := \sum_ {a _ {- i}} \pi_ {- i} (a _ {- i} | s) Q _ {i} ^ {\pi} (s, a _ {i}, a _ {- i}), \bar {A} _ {i} ^ {\pi} (s, a _ {i}) := \sum_ {a _ {- i}} \pi_ {- i} (a _ {- i} | s) A _ {i} ^ {\pi} (s, a _ {i}, a _ {- i}). \tag {2}
$$
Definition 2.1 ([39]). The stochastic game  $\mathcal{M}$  is a Markov potential game if there exists a bounded potential function  $\phi : S \times \mathcal{A} \to \mathbb{R}$  such that for any agent  $i$ , initial states and any set of policies  $\pi_i, \pi_i', \pi_{-i}$ :
$$
V _ {i} ^ {\pi_ {i} ^ {\prime}, \pi_ {- i}} (s) - V _ {i} ^ {\pi_ {i}, \pi_ {- i}} (s) = \Phi^ {\pi_ {i} ^ {\prime}, \pi_ {- i}} (s) - \Phi^ {\pi_ {i}, \pi_ {- i}} (s),
$$
where  $\Phi^{\pi}(s) := \mathbb{E}^{\pi}[\sum_{k=0}^{\infty} \gamma^{k} \phi(s^{k}, \boldsymbol{a}^{k}) | s^{0} = s]$ .
We assume that an upper bound exists for the potential function, i.e.,  $0 \leq \phi(s, \boldsymbol{a}) \leq \phi_{\max}, \forall s \in S, \boldsymbol{a} \in \mathcal{A}$ , and consequently,  $\Phi^{\pi}(s) \leq \frac{\phi_{\max}}{1 - \gamma}$ .
It is common in policy optimization to parameterize the policy for easier computations. In this paper, we focus on the widely used softmax parameterization [1, 26], where a global policy  $\pi(\boldsymbol{a}|s) = \prod_{i\in [n]}\pi_i(a_i|s)$  is parameterized by a set of parameters  $\{\theta_1,\dots,\theta_n\}, \theta_i\in \mathbb{R}^{|S|\times |\mathcal{A}_i|}$  in the following form
$$
\pi_ {i} (a _ {i} | s) = \frac {\exp \{[ \theta_ {i} ] _ {s , a _ {i}} \}}{\sum_ {a _ {j} \in \mathcal {A} _ {i}} \exp \{[ \theta_ {i} ] _ {s , a _ {j}} \}}, \forall (s, a _ {i}) \in \mathcal {S} \times \mathcal {A} _ {i}.
$$
2.1 Optimality Conditions
In the MPG setting, there may exist multiple stationary points, a set of policies that has zero policy gradients, for the same problem; therefore, we need to introduce notions of solutions to evaluate policies. The term Nash equilibrium is used to define a measure of "stationarity" in strategic games.
Definition 2.2. A joint policy  $\pi^{*}$  is called a Nash equilibrium if for all  $i\in [n]$ , we have
$$
V _ {i} ^ {\pi_ {i} ^ {*}, \pi_ {- i} ^ {*}} (\rho) \geq V _ {i} ^ {\pi_ {i} ^ {\prime}, \pi_ {- i} ^ {*}} (\rho) f o r a l l \pi_ {i} ^ {\prime}.
$$
For any given joint policy that does not necessarily satisfy the definition of NE, we provide the definition of NE-gap as follows [39]:
$$
\mathrm {N E - g a p} (\pi) := \max  _ {i \in [ n ], \pi_ {i} ^ {\prime} \in \Delta (\mathcal {A} _ {i})} \left[ V _ {i} ^ {\pi_ {i} ^ {\prime}, \pi_ {- i}} (\rho) - V _ {i} ^ {\pi_ {i}, \pi_ {- i}} (\rho) \right].
$$
Furthermore, we refer to a joint policy  $\pi$  as  $\epsilon$ -NE when its NE-gap  $(\pi) \leq \epsilon$ . The NE-gap satisfies the following inequalities based on the performance difference lemma [15, 38],
$$
\mathrm {N E - g a p} (\pi) \leq \frac {1}{1 - \gamma} \sum_ {i, s, a _ {i}} d _ {\rho} ^ {\pi_ {i} ^ {*}, \pi_ {- i}} (s) \pi_ {i} ^ {*} (a _ {i} | s) \bar {A} _ {i} ^ {\pi} (s, a _ {i}) \leq \frac {1}{1 - \gamma} \sum_ {i, s} d _ {\rho} ^ {\pi_ {i} ^ {*}, \pi_ {- i}} (s) \max _ {a _ {i}} \bar {A} _ {i} ^ {\pi} (s, a _ {i}).
$$
In the tabular single-agent RL, most works consider the optimality gap as the difference between the expectations of the value functions of the current policy and the optimal policy, defined as  $V^{\pi^k}(\rho) - V^{\pi^\star}(\rho)$ . However, this notion does not extend to multi-agent systems. Even in a fully cooperative MPG where all agents share the same reward, the optimal policy of one agent is dependent on the joint policies of other agents. As a result, it is common for the system to have multiple "best" policy combinations (or stationary points), which all constitute Nash equilibria. Additionally, it has also been addressed by previous works that any NE point in an MPG is first order stable [39]. Given that this work addresses a MARL problem, we focus our analysis on the NE-gap.
3.1 Warm-Up: Potential Games
In this section, we first consider the instructive case of static potential games, where the state does not change with time. Potential games are an important class of games that admit a potential function  $\phi$  to capture differences in each agent's reward function caused by unilateral bias [28, 13], which is defined as
$$
r _ {i} \left(a _ {i}, a _ {- i}\right) - r _ {i} \left(a _ {i} ^ {\prime}, a _ {- i}\right) = \phi \left(a _ {i}, a _ {- i}\right) - \phi \left(a _ {i} ^ {\prime}, a _ {- i}\right), \quad \forall a _ {i}, a _ {i} ^ {\prime}, a _ {- i}. \tag {3}
$$
Algorithm Update In the potential games setting, the policy update using natural policy gradient is [4]:
$$
\pi_ {i} ^ {k + 1} \left(a _ {i}\right) \propto \pi_ {i} ^ {k} \left(a _ {i}\right) \exp \left(\eta \bar {r} _ {i} ^ {k} \left(a _ {i}\right)\right), \tag {4}
$$
where the exact independent gradient over policy  $\pi_{i}$ , also referred to as oracle, is captured by the marginalized reward  $\bar{r}_i(a_i) = \mathbb{E}_{a_{-i} \sim \pi_{-i}}[r_i(a_i, a_{-i})]$ . By definition, the NE-gap for potential games
is calculated as  $\max_{i\in [n]}\langle \pi_i^{*k} - \pi_i^k,\bar{r}_i^k\rangle$ , where  $\pi_i^{*k}\in \arg \max_{\pi_i}V_i^{\pi_i,\pi_{-i}^k}$  is the optimal solution for agent  $i$  when the rest of the agents use the joint policy  $\pi_{-i}^{k}$ .
The local marginalized reward  $\bar{r}_i(a_i)$  is calculated based on other agents' policies; hence, for any two sets of policies, the difference in marginalized reward can be bounded using the total variation distance of the two probability measures [4]. Using this property, we can also show that there is a "smooth" relationship between the marginalized rewards and their respective policies. We note that this relationship holds for stochastic games in general. It does not depend on the nature of the policy update or the potential game assumption.
We now introduce a lemma that is specific to the potential game formulation and the NPG update:
Lemma 3.1. Given policy  $\pi^k$  and marginalized reward  $\bar{r}_i^k (a_i)$ , for  $\pi^{k + 1}$  generated using an NPG update in (4), we have the following inequality for any  $\eta < \frac{1}{\sqrt{n}}$ ,
$$
\phi (\pi^ {k + 1}) - \phi (\pi^ {k}) \geq (1 - \sqrt {n} \eta) \sum_ {i = 1} ^ {n} \langle \pi_ {i} ^ {k + 1} - \pi_ {i} ^ {k}, \bar {r} _ {i} ^ {k} \rangle .
$$
Lemma 3.1 provides a lower bound on the difference of potential functions between two consecutive steps. This implies that at each time step, the potential function value is guaranteed to be monotonically increasing, as long as the learning rate satisfies  $\eta < \frac{1}{\sqrt{n}}$ .
Note that the lower bound of Lemma 3.1 involves  $\langle \pi_i^{k + 1} - \pi_i^k,\bar{r}_i^k\rangle$ , which resembles the form of NE-gap  $\langle \pi_i^{*k} - \pi_i^k,\bar{r}_i^k\rangle$ . Assuming we can establish a lower bound for the right-hand side of Lemma 3.1 using NE-gap, the next step is to show that the sum of all NE-gap iterations is upper bounded by a telescoping sum of the potential function, thus obtaining an upper bound on the NE-gap.
We start by introducing a function  $f^k: \mathbb{R} \to \mathbb{R}$  defined as follows,
$$
f ^ {k} (\alpha) = \sum_ {i = 1} ^ {n} \left\langle \pi_ {i, \alpha} - \pi_ {i} ^ {k}, \bar {r} _ {i} ^ {k} \right\rangle , \text {w h e r e} \pi_ {i, \alpha} (\cdot) \propto \pi_ {i} ^ {k} (\cdot) \exp \left\{\alpha \bar {r} _ {i} ^ {k} (\cdot) \right\}. \tag {5}
$$
It is obvious that  $f^k(0) = 0$ ,  $f^k(\eta) = \sum_i \langle \pi_i^{k+1} - \pi_i^k, \vec{r}_i^k \rangle \geq 0$ , and  $\lim_{\alpha \to \infty} f^k(\alpha) = \sum_i \langle \pi_i^{*k} - \pi_i^k, \vec{r}_i^k \rangle$ .
Without loss of generality, for agent  $i$  at iteration  $k$ , define  $a_{i_p}^k \in \arg \max_{a_j \in \mathcal{A}_i} \bar{r}_i^k(a_j) =: \mathcal{A}_{i_p}^k$  and  $a_{i_q}^k \in \arg \max_{a_j \in \mathcal{A}_i \setminus \mathcal{A}_{i_p}^k} \bar{r}_i^k(a_j)$ , where  $\mathcal{A}_{i_p}^k$  denotes the set of the best possible actions for agent  $i$  at iteration  $k$ . Similar to [38], we define
$$
c ^ {k} := \min  _ {i \in [ n ]} \sum_ {a _ {j} \in \mathcal {A} _ {i _ {p}} ^ {k}} \pi_ {i} ^ {k} (a _ {j}) \in (0, 1), \quad \delta^ {k} := \min  _ {i \in [ n ]} \left[ \bar {r} _ {i} ^ {k} \left(a _ {i _ {p}} ^ {k}\right) - \bar {r} _ {i} ^ {k} \left(a _ {i _ {q}} ^ {k}\right) \right] \in (0, 1). \tag {6}
$$
Additionally, we denote  $c_{K} \coloneqq \min_{k \in [K]} c^{k}$ ;  $c \coloneqq \inf_{K} c_{K} > 0$ ;  $\delta_{K} \coloneqq \min_{k \in [K]} \delta^{k}$ .
We provide the following lemma on the relationship between  $f^{k}(\alpha)$  and  $f^{k}(\infty) \coloneqq \lim_{\alpha \to \infty} f^{k}(\alpha)$ , which lays the foundation to obtain sharper results than those in the existing work.
Lemma 3.2. For function  $f^{k}(\alpha)$  defined in (5) and any  $\alpha > 0$ , we have the following inequality.
$$
f ^ {k} (\alpha) \geq f ^ {k} (\infty) \left[ 1 - \frac {1}{c \left(\exp \left(\alpha \delta_ {K}\right) - 1\right) + 1} \right].
$$
We refer to  $\delta_K$  as the minimal suboptimality gap of the system for the first  $K$  iterations, the effect of which will be discussed later in Section 3.3. Using the two lemmas above and the definitions of  $c$  and  $\delta_K$ , we establish the following theorem on the convergence of the NPG algorithm in potential games.
Theorem 3.3. Consider a potential game with NPG update using (4). For any  $K \geq 1$ , choosing  $\eta = \frac{1}{2\sqrt{n}}$ , we have
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} N E - g a p \left(\pi^ {k}\right) \leq \frac {2 \phi_ {m a x}}{K} \left(1 + \frac {2 \sqrt {n}}{c \delta_ {K}}\right).
$$
Proof. Choose  $\alpha = \eta = \frac{1}{2\sqrt{n}}$  in Lemma 3.2,
$$
\begin{array}{l} \mathrm {N E - g a p} (\pi^ {k}) \leq \lim _ {\alpha \to \infty} f ^ {k} (\alpha) \leq \frac {1}{1 - \frac {1}{c (\exp (\delta_ {K} \eta) - 1) + 1}} f ^ {k} (\eta) \leq \frac {1}{1 - \frac {1}{c \delta_ {K} \eta + 1}} f ^ {k} (\eta) \\ \leq \frac {1}{(1 - \sqrt {n} \eta) \frac {c \delta_ {K} \eta}{c \delta_ {K} \eta + 1}} [ \phi (\pi^ {k + 1}) - \phi (\pi^ {k}) ] = 2 [ \phi (\pi^ {k + 1}) - \phi (\pi^ {k}) ] (1 + \frac {2 \sqrt {n}}{c \delta_ {K}}). \\ \end{array}
$$
Then, we have  $\frac{1}{K}\sum_{k = 0}^{K - 1}\mathrm{NE - gap}(\pi^k)\leq \frac{2(\phi(\pi^K) - \phi(\pi^0))}{K} (1 + \frac{2\sqrt{n}}{c\delta_K})\leq \frac{2\phi_{max}}{K} (1 + \frac{2\sqrt{n}}{c\delta_K}).$
One challenge in MARL is that the NE-gap is not monotonic, so we must seek ergodic convergence results, which characterize the behavior of the temporal average of the NE-gap. Theorem 3.3 shows that for potential games with NPG policy update, the ergodic NE-gap of the system converges to zero with a rate  $\mathcal{O}(1 / (K\delta_K))$ . When  $\delta_K$  is uniformly lower bounded, Theorem 3.3 provides a significant speed up compared to previous convergence results. Apart from the iteration complexity, the NE-gap is also dependent linearly on  $1 / c$  and  $1 / \delta_K$ . We address the effect of  $c$  in the analysis here and defer the discussion of  $\delta_K$  to Section 3.3. Under some mild assumptions, we can show that the system converges with a rate of  $\mathcal{O}(1 / K)$ .
The Effect of  $c$  The convergence rate given by Theorem 3.3 scales with  $1 / c$ , where  $c$  might potentially be arbitrarily small. A small value for  $c$  generally describes a policy that is stuck at some regions far from a NE, yet the policy gradient is small. It has been shown in [20] that these ill-conditioned problems could take exponential time to solve even in single-agent settings for policy gradient methods. The same issue also occurs to NPG in MARL, since the local Fisher information matrix can not cancel the occupancy measure and action probability of other agents. A similar problem has also been reported in the analysis of [38, 26] for the MPG setting. [38] proposed the addition of a log-barrier regularization to mitigate this issue. However, that comes at the cost of an  $\mathcal{O}(1 / (\lambda K))$  convergence rate to a  $\lambda$ -neighborhood solution, which is only reduced to the exact convergence rate of  $\mathcal{O}(1 / \sqrt{K})$  when  $\lambda = 1 / \sqrt{K}$ . Therefore, this limitation may not be effectively avoided without impacting the convergence rate.
3.2 General Markov Potential Games
We now extend our analysis to MPGs. The analysis mainly follows a similar framework as potential games. However, the introduction of state transitions and the discount factor  $\gamma$  add an additional layer of complexity to the problem, making it far from trivial. As pointed out in [19], we can construct MDPs that are potential games for every state, yet the entire system is not a MPG. Thus, the analysis of potential games does not directly apply to MPGs.
We first provide the independent NPG update for MPGs.
Algorithm Update For MPGs at iteration  $k$ , the independent NPG updates the policy as follows [38]: