Title: Provably Fast Convergence of Independent Natural Policy Gradient for Markov Potential Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi Agent Reinforcement Learning

Markov Potential Games

Natural Policy Gradient

Nash Equilibrium
Abstract: This work studies an independent natural policy gradient (NPG) algorithm for the multi-agent reinforcement learning problem in Markov potential games. It is shown that, under mild technical assumptions and the introduction of the \textit{suboptimality gap}, the independent NPG method with an oracle providing exact policy evaluation asymptotically reaches an $\epsilon$-Nash Equilibrium (NE) within $\mathcal{O}(1/\epsilon)$ iterations. This improves upon the previous best result of $\mathcal{O}(1/\epsilon^2)$ iterations and is of the same order, $\mathcal{O}(1/\epsilon)$, that is achievable for the single-agent case. Empirical results for a synthetic potential game and a congestion game are presented to verify the theoretical bounds.

Provably Fast Convergence of Independent Natural Policy Gradient for Markov Potential Games

Abstract

This work studies an independent natural policy gradient (NPG) algorithm for the multi-agent reinforcement learning problem in Markov potential games. It is shown that, under mild technical assumptions and the introduction of the suboptimality gap , the independent NPG method with an oracle providing exact policy evaluation asymptotically reaches an ϵ -Nash Equilibrium (NE) within O (1 /ϵ ) iterations. This improves upon the previous best result of O (1 /ϵ 2 ) iterations and is of the same order, O (1 /ϵ ) , that is achievable for the single-agent case. Empirical results for a synthetic potential game and a congestion game are presented to verify the theoretical bounds.

1 Introduction

Reinforcement learning (RL) is often impacted by the presence and interactions of several agents in a multi-agent system. This challenge has motivated recent studies of multi-agent reinforcement learning (MARL) in stochastic games [37, 3]. Applications of MARL include robotics [30], modern production systems [2], economic decision making [33], and autonomous driving [31]. Among the various types of stochastic games, we focus on a commonly studied model for MARL, known as Markov Potential Games (MPGs). MPGs are seen as a generalization of canonical Markov Decision Processes (MDPs) in the multi-agent setting. In MPGs, there exists a potential function that can track the value changes of all agents. Unlike single-agent systems, where the goal is to find the optimal policy, the objective in this paper is to find a global policy, formed by the joint product of a set of local policies, that leads the system to reach a Nash equilibrium (NE) [29], which is precisely defined in Section 2.

A major challenge in the analysis of multi-agent systems is the restriction on joint policies of agents. For single-agent RL, policy updates are designed to increase the probability of selecting the action with the highest reward. However, in multi-agent systems, the global policy is constructed by taking the product of local agents' policies, which makes MARL algorithms suffer a greater risk of being trapped near undesirable stationary points. Consequently, finding a NE in MARL is more challenging than finding the global optimum in the single-agent case, and it is therefore difficult for MPGs to recover the convergence rates of single-agent Markov decision processes (MDPs).

Additionally, the global action space in MPGs scales exponentially with the number of agents within the system, making it crucial to find an algorithm that scales well for a large number of agents. Recent

studies [8, 14] addressed the issue by an approach called independent learning, where each agent performs policy update based on local information without regard to policies of the other agents. Independent learning algorithms scale only linearly with respect to the number of agents and are therefore preferred for large-scale multi-agent problems.

Using algorithms such as policy gradient (PG) and natural policy gradient (NPG), single-agent RL can provably converge to the global optimal policy [1]. However, extending these algorithms from single-agent to multi-agent settings presents natural challenges as discussed above. Multiple recent works have analyzed PG and NPG in multi-agent systems. However, due to the unique geometry of the problem and complex relationships among the agents, the theoretical understanding of MARL is still limited, with most works showing slower convergence rates when compared to their single-agent counterparts (see Table 1).

Contributions We study the independent NPG algorithm in multi-agent systems and provide a novel technical analysis that guarantees a provably fast convergence rate. We start our analysis with potential games in Section 3.1 and then generalize the findings and provide a convergence guarantee for Markov potential games in Section 3.2. We show that under mild assumptions, the ergodic (i.e., temporal average of) NE-gap converges with iteration complexity of O (1 /ϵ ) after a finite threshold (Theorem 3.6). This result provides a substantial improvement over the best known rate of O ( 1 /ϵ 2 ) in [38]. Our main theorem also reveals mild or improved dependence on multiple critical factors, including the number of agents n , the initialization dependent factor c , the distribution mismatch coefficient M , and the discount factor γ , discussed in Section 3. We dedicate Section 3.3 to discuss the impact of the asymptotic suboptimality gap δ ∗ , which is a new factor in this work.

In addition to our theoretical results, two numerical experiments are also conducted in Section 4 for verification of the analysis. We consider a synthetic potential game similar to [38] and a congestion game from [19]. The omitted proofs of our theoretical results can be found in the appendix.

1.1 Related Literature

Markov Potential Games Since many properties of single-agent RL do not hold in MARL, the analysis of MARL presents several challenges. Various settings have been addressed for MARL in recent works. A major distinction between these works stems from whether the agents are competitive or cooperative [11]. In this paper we consider MPGs introduced in stochastic control [7]. MPGs are a generalized formulation of identical-reward cooperative games. Markov cooperative games have been studied in the early work of [34], and more recently by [23, 24, 32, 10]. The work [36]

also offered extensive empirical results for cooperative games using multi-agent proximal policy optimization. The work by [27] established polynomial convergence for MPGs under both Q-update as well as actor-critic update. [25] studied an independent Q-update in MPGs with perturbation, which converges to a stationary point with probability one. Conversely, [17, 13] studied potential games, a special static case of simplified MPGs with no state transition.

Policy Gradient in Games Policy gradient methods for centralized MDPs have drawn much attention thanks to recent advancements in RL theory [1, 26]. The extension of PG methods to multi-agent settings is quite natural. [6, 35, 5] studied two-player zero-sum competitive games. The general-sum linear-quadratic game was studied in [12]. [14] studied general-sum Markov games and provided convergence of V-learning in two-player zero-sum games.

Of particular relevance to our work are the works [19, 39, 38, 8] which focus on the MPG setting and propose adaptations of PG and NPG-based methods from single-agent problems to the MARL setting. Table 1 provides a detailed comparison between these works. The previous theoretical results in multi-agent systems have provided convergence rates dependent on different parameters of the system. However, the best-known iteration complexity to reach an ϵ -NE in MARL is O ( 1 /ϵ 2 ) . Therefore, there still exists a rate discrepancy between MARL methods where O (1 /ϵ ) complexity has been established in centralized RL algorithms [1]. Our main contribution is to close this gap by establishing an iteration complexity of O (1 /ϵ ) in this work.

2 Problem Formulation

We consider a stochastic game M = ( n, S , A , P, { r i } i ∈ [ n ] , γ, ρ ) consisting of n agents denoted by a set [ n ] = { 1 , ..., n } . The global action space A = A 1 × ... ×A n is the product of individual action spaces, with the global action defined as a := ( a 1 , ..., a n ) . The global state space is represented by S , and the system transition model is captured by P : S × A → ∆( S ) . Furthermore, each agent is equipped with an individual reward function r i : S × A → [0 , 1] . We use γ ∈ (0 , 1) to denote the discount factor and ρ ∈ ∆( S ) to denote the initial state distribution.

The system policy is denoted by π : S → ∆( A 1 ) × · · · × ∆( A n ) ⊂ ∆( A ) , where ∆( A ) is the probability simplex over the global action space. In the multi-agent setting, all agents make decisions independently given the observed state, often referred to as a decentralized stochastic policy [38]. Under this setup, we have π ( a | s ) = ∏ i ∈ [ n ] π i ( a i | s ) , where π i : S → ∆( A i ) is the local policy for agent i . For the ease of notation, we denote the joint policy over the set [ n ] \{ i } by π -i = ∏ j ∈ [ n ] \{ i } π j and use the notation a -i analogously.

We define the state value function V π i ( s ) with respect to the reward r i ( s, a ) as V π i ( s ) := E π [ ∑ ∞ t =0 γ t r i ( s t , a t ) | s 0 = s ] , where ( s t , a t ) denotes the global state-action pair at time t , and we denote the expected value of the state value function over the initial state distribution ρ as V π i ( ρ ) := E s ∼ ρ [ V π i ( s )] . We can similarly define the state visitation distribution under ρ as d π ρ ( s ) := (1 -γ ) E π [ ∑ ∞ t =0 γ t ✶ ( s t = s ) | s 0 ∼ ρ ] , where ✶ is the indicator function. The state-action value function and advantage function are, respectively, given by

For the sake of analysis, we further define the marginalized Q-function and advantage function ¯ Q i : S × A i → R and ¯ A i : S × A i → R as:

Definition 2.1 ([39]) . The stochastic game M is a Markov potential game if there exists a bounded potential function ϕ : S × A → R such that for any agent i , initial state s and any set of policies π i , π ′ i , π -i :

We assume that an upper bound exists for the potential function, i.e., 0 ≤ ϕ ( s, a ) ≤ ϕ max , ∀ s ∈ S , a ∈ A , and consequently, Φ π ( s ) ≤ ϕ max 1 -γ .

It is common in policy optimization to parameterize the policy for easier computations. In this paper, we focus on the widely used softmax parameterization [1, 26], where a global policy π ( a | s ) = ∏ i ∈ [ n ] π i ( a i | s ) is parameterized by a set of parameters { θ 1 , ..., θ n } , θ i ∈ R |S|×|A i | in the following form

2.1 Optimality Conditions

In the MPG setting, there may exist multiple stationary points, a set of policies that has zero policy gradients, for the same problem; therefore, we need to introduce notions of solutions to evaluate policies. The term Nash equilibrium is used to define a measure of "stationarity" in strategic games.

Definition 2.2. A joint policy π ∗ is called a Nash equilibrium if for all i ∈ [ n ] , we have

For any given joint policy that does not necessarily satisfy the definition of NE, we provide the definition of NE-gap as follows [39]:

Furthermore, we refer to a joint policy π as ϵ -NE when its NE-gap ( π ) ≤ ϵ . The NE-gap satisfies the following inequalities based on the performance difference lemma [15, 38],

In the tabular single-agent RL, most works consider the optimality gap as the difference between the expectations of the value functions of the current policy and the optimal policy, defined as V π k ( ρ ) -V π ⋆ ( ρ ) . However, this notion does not extend to multi-agent systems. Even in a fully cooperative MPG where all agents share the same reward, the optimal policy of one agent is dependent on the joint policies of other agents. As a result, it is common for the system to have multiple "best" policy combinations (or stationary points), which all constitute Nash equilibria. Additionally, it has also been addressed by previous works that any NE point in an MPG is first order stable [39]. Given that this work addresses a MARL problem, we focus our analysis on the NE-gap.

3 Main Results

3.1 Warm-Up: Potential Games

In this section, we first consider the instructive case of static potential games, where the state does not change with time. Potential games are an important class of games that admit a potential function ϕ to capture differences in each agent's reward function caused by unilateral bias [28, 13], which is defined as

Algorithm Update In the potential games setting, the policy update using natural policy gradient is [4]:

where the exact independent gradient over policy π i , also referred to as oracle, is captured by the marginalized reward ¯ r i ( a i ) = E a -i ∼ π -i [ r i ( a i , a -i )] . By definition, the NE-gap for potential games

is calculated as max i ∈ [ n ] ⟨ π ∗ k i -π k i , ¯ r k i ⟩ , where π ∗ k i ∈ arg max π i V π i ,π k -i i is the optimal solution for agent i when the rest of the agents use the joint policy π k -i .

The local marginalized reward ¯ r i ( a i ) is calculated based on other agents' policies; hence, for any two sets of policies, the difference in marginalized reward can be bounded using the total variation distance of the two probability measures [4]. Using this property, we can also show that there is a "smooth" relationship between the marginalized rewards and their respective policies. We note that this relationship holds for stochastic games in general. It does not depend on the nature of the policy update or the potential game assumption.

We now introduce a lemma that is specific to the potential game formulation and the NPG update:

Lemma 3.1. Given policy π k and marginalized reward ¯ r k i ( a i ) , for π k +1 generated using an NPG update in (4) , we have the following inequality for any η < 1 √ n ,

Lemma 3.1 provides a lower bound on the difference of potential functions between two consecutive steps. This implies that at each time step, the potential function value is guaranteed to be monotonically increasing, as long as the learning rate satisfies η < 1 √ n .

Note that the lower bound of Lemma 3.1 involves ⟨ π k +1 i -π k i , ¯ r k i ⟩ , which resembles the form of NE-gap ⟨ π ∗ k i -π k i , ¯ r k i ⟩ . Assuming we can establish a lower bound for the right-hand side of Lemma 3.1 using NE-gap, the next step is to show that the sum of all NE-gap iterations is upper bounded by a telescoping sum of the potential function, thus obtaining an upper bound on the NE-gap.

We start by introducing a function f k : R → R defined as follows,

It is obvious that f k (0) = 0 , f k ( η ) = ∑ i ⟨ π k +1 i -π k i , ¯ r k i ⟩ ≥ 0 , and lim α →∞ f k ( α ) = ∑ i ⟨ π ∗ k i -π k i , ¯ r k i ⟩ .

Without loss of generality, for agent i at iteration k , define a k i p ∈ arg max a j ∈A i ¯ r k i ( a j ) =: A k i p and a k i q ∈ arg max a j ∈A i \A k i p ¯ r k i ( a j ) , where A k i p denotes the set of the best possible actions for agent i at iteration k . Similar to [38], we define

Additionally, we denote c K := min k ∈ [ K ] c k ; c := inf K c K > 0; δ K := min k ∈ [ K ] δ k .

We provide the following lemma on the relationship between f k ( α ) and f k ( ∞ ) := lim α →∞ f k ( α ) , which lays the foundation to obtain sharper results than those in the existing work.

Lemma 3.2. For function f k ( α ) defined in (5) and any α > 0 , we have the following inequality.

We refer to δ K as the minimal suboptimality gap of the system for the first K iterations, the effect of which will be discussed later in Section 3.3. Using the two lemmas above and the definitions of c and δ K , we establish the following theorem on the convergence of the NPG algorithm in potential games.

Theorem 3.3. Consider a potential game with NPG update using (4). For any K ≥ 1 , choosing η = 1 2 √ n , we have

Proof. Choose α = η = 1 2 √ n in Lemma 3.2,

One challenge in MARL is that the NE-gap is not monotonic, so we must seek ergodic convergence results, which characterize the behavior of the temporal average of the NE-gap. Theorem 3.3 shows that for potential games with NPG policy update, the ergodic NE-gap of the system converges to zero with a rate O (1 / ( Kδ K )) . When δ K is uniformly lower bounded, Theorem 3.3 provides a significant speed up compared to previous convergence results. Apart from the iteration complexity, the NE-gap is also dependent linearly on 1 /c and 1 /δ K . We address the effect of c in the analysis here and defer the discussion of δ K to Section 3.3. Under some mild assumptions, we can show that the system converges with a rate of O (1 /K ) .

The Effect of c The convergence rate given by Theorem 3.3 scales with 1 /c , where c might potentially be arbitrarily small. A small value for c generally describes a policy that is stuck at some regions far from a NE, yet the policy gradient is small. It has been shown in [20] that these ill-conditioned problems could take exponential time to solve even in single-agent settings for policy gradient methods. The same issue also occurs to NPG in MARL, since the local Fisher information matrix can not cancel the occupancy measure and action probability of other agents. A similar problem has also been reported in the analysis of [38, 26] for the MPG setting. [38] proposed the addition of a log-barrier regularization to mitigate this issue. However, that comes at the cost of an O (1 / ( λK )) convergence rate to a λ -neighborhood solution, which is only reduced to the exact convergence rate of O (1 / √ K ) when λ = 1 / √ K . Therefore, this limitation may not be effectively avoided without impacting the convergence rate.

3.2 General Markov Potential Games

We now extend our analysis to MPGs. The analysis mainly follows a similar framework as potential games. However, the introduction of state transitions and the discount factor γ add an additional layer of complexity to the problem, making it far from trivial. As pointed out in [19], we can construct MDPs that are potential games for every state, yet the entire system is not a MPG. Thus, the analysis of potential games does not directly apply to MPGs.

We first provide the independent NPG update for MPGs.

Algorithm Update For MPGs at iteration k , the independent NPG updates the policy as follows [38]:

where ¯ A i is defined in (2).

Different agents in a MPG do not share reward functions in general, which makes it difficult to compare evaluations of gradients across agents. However, with the introduction of Lemma 3.4, we find that MPGs have similar properties as fully cooperative games with a shared reward function. This enables us to establish relationships between policy updates of all agents. We first define h i ( s, a ) := r i ( s, a ) -ϕ ( s, a ) , which implies V π i ( s ) = Φ π ( s ) + V π h i ( s ) .

Lemma 3.4. Define ¯ A π h i ( s, a i ) with respect to h i similar to (2) . We then have

Lemma 3.4 shows a unique property of function h i , where the expectation of the marginalized advantage function over every local policy π ′ i yields the same effect. This property is directly associated with the MPG problem structure and is later used in Lemma 3.5. Next, we introduce the following assumption on the state visitation distribution, which is crucial and standard for studying the Markov dynamics of the system.

Assumption 3.1 ([1, 38, 8]) . The Markov potential game M satisfies: inf π min s d π ρ ( s ) > 0 .

Similar to potential games, when the potential function ϕ of a MPG is bounded, the marginalized advantage function ¯ A i for two policies can be bounded by the total variation between the policies.

Additionally, similar to Lemma 3.1, we present a lower bound in the following lemma for the potential function difference in two consecutive rounds.

Lemma 3.5. Given policy π k and marginalized advantage function ¯ A π k i ( s, a i ) , for π k +1 generated using NPG update in (7) , we have the following inequality,

Thus, using a function f adapted from (5) as a connection, we are able to establish the convergence of NE-gap for MPGs in the following theorem.

Theorem 3.6. Consider a MPG with isolated stationary points and the policy update following NPG update (7). For any K ≥ 1 , choosing η = (1 -γ ) 2 2 √ nϕ max , we have

similar to (6) .

Here, isolated implies that no other stationary points exist in any sufficiently small open neighborhood of any stationary point. This convergence result is similar to that provided in Theorem 3.3 for potential games. We note that our theorem also applies to an alternate definition for MPGs in works such as [8], which we discuss in Appendix C. Compared to potential games, the major difference is the introduction of M , which measures the distribution mismatch in the system. Generally, Assumption 3.1 implies a finite value for M in the MPG setup.

Discussion and Comparison on Convergence Rate Compared to the iteration complexity of previous works listed in Table 1, the convergence rate in Theorem 3.6 presents multiple improvements. Most importantly, the theorem guarantees that the averaged NE-gap reaches ϵ in O (1 /ϵ ) iterations, improving the best previously known result of O ( 1 /ϵ 2 ) in Table 1. Furthermore, this rate of convergence does not depend on the size of action space |A i | , and it has milder dependence on other system parameters, such as the distribution mismatch coefficient M and (1 -γ ) . Note that many of the parameters above could be arbitrarily large in practice (e.g., 1 -γ = 0 . 01 in our congestion game experiment). Theorem 3.6 indicates a tighter convergence bound with respect to the discussed factors in general.

3.3 The Consideration of Suboptimality Gap

In Theorems 3.3 and 3.6, we established the ergodic convergence rates of NPG in potential game and Markov potential game settings, which depend on 1 /δ K . In these results, δ k generally encapsulates where

the difference between the gradients evaluated at the best and second best policies, and δ K is a lower bound on δ k . We refer to δ k as the suboptimality gap at iteration k . In our analysis, the suboptimality gap provides a vehicle to establish the improvement of the potential function in two consecutive steps. In particular, it enables us to draw a connection between Φ π k +1 -Φ π k and NE-gap( π k ) using Lemma 3.2 and Lemma B.3 in the appendix. On the contrary, most of the existing work does not rely on this approach and generally studies the relationship between Φ π k +1 -Φ π k and the squared NE-gap( π k ), which suffers a slower convergence rate of O (1 / √ K ) [38].

The analysis leveraging the suboptimality gap, though has not been adopted in the MARL studies, was considered in the single-agent scenario. Khodadadian et al. [16] proved asymptotic geometric convergence of single-agent RL with the introduction of optimal advantage function gap ∆ k , which shares similar definition as the gap δ k studied in this work. Moreover, the notion of suboptimality gap is commonly used in the multi-armed bandit literature [18], so as to give the instance-dependent analysis.

While our convergence rate is sharper, a lower bound on the suboptimality gap is generally not guaranteed, and scenarios with zero optimality gap can be constructed in MPGs. In practice, we find that even if δ k approaches zero in certain iterations, it may not converge to zero, and in these scenarios the system still converges without a slowdown. We provide a numerical example in Section 4.1 (Fig. 1a) to support our claim. Nevertheless, in what follows, we further identify a sufficient condition that allows us to alleviate this problem altogether by focusing on the asymptotic profile of the sub-optimality gap.

Theoretical Relaxation The following proposition guarantees the asymptotic results of independent NPG.

Proposition 3.1 ([38]) . Suppose that Assumption 3.1 holds and that the stationary policies are isolated. Then independent NPG with η ≤ (1 -γ ) 2 2 √ nϕ max guarantees that lim k →∞ π k = π ∞ , where π ∞ is a Nash policy.

Since asymptotic convergence of policy is guaranteed by Proposition 3.1, the suboptimality gap δ k is also guaranteed to converge to some δ ∗ . We make the following assumption about the asymptotic suboptimality gap, which only depends on the property of the game itself.

Assumption 3.2. Assume that lim k →∞ δ k = δ ∗ > 0 .

Assumption 3.2 provides a relaxation in the sense that instead of requiring a lower bound inf δ k for all k , we only need δ ∗ > 0 , a lower bound on the limit as the agents approach some Nash policies. This will allow us to disregard the transition behavior of the system and focus on the rate for large enough k .

By the definition of δ ∗ , we know that there exists finite K ′ such that ∀ k > K ′ , | δ k -δ ∗ | ≤ δ ∗ 2 , δ k ≥ δ ∗ 2 . Using these results, we can rework the proofs of Theorems 3.3 and 3.6 to get the following corollary.

Corollary 3.6.1. Consider a MPG that satisfies Assumption 3.2 with NPG update using algorithm 7. There exists K ′ , such that for any K ≥ 1 , choosing η = (1 -γ ) 2 √ , we have

2 nϕ max

where M,c are defined as in Theorem 3.6.

4 Experimental Results

In previous sections, we established the theoretical convergence of NPG in MPGs. In order to verify the results, we construct two experimental settings for the NPG update and compare our empirical results with existing algorithms. We consider a synthetic potential game scenario with randomly generated rewards and a congestion problem studied in [19] and [8]. We also provide the source code 4 for all experiments.

0

25

50

75

100

125

150

175

200

Iterations

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Value

NE-gap

Suboptimality Gap

ck

(a)

0

1000

2000

3000

4000

5000

Iterations

0.00

0.05

0.10

0.15

0.20

NE-gap

Direct Projected Q Ascent

Softmax PG

Softmax NPG

Softmax NPG Entropy Reg.

(b)

0

50

100

150

200

250

Iterations

0.0

0.5

1.0

1.5

2.0

2.5

3.0

L1-accuracy

Direct Projected Q Ascent

Direct Projected Gradient Ascent

Softmax PG

Softmax NPG

(c)

4.1 Synthetic Potential Games

We first construct a potential game with a fully cooperative objective, where the reward tensor r ( a ) is randomly generated from a uniform distribution. We set the agent number to n = 3 , each with different action space |A 1 | = 3 , |A 2 | = 4 , |A 3 | = 5 . At the start of the experiment, all agents are initialized with uniform policies. We note that the experimental setting in [38] falls under this broad setting, although the experiment therein was a two-player game with carefully designed rewards.

The results are shown in Figure 1b. We compare the performance of independent NPG with other commonly studied policy updates, such as projected Q ascent [8], entropy regularized NPG [38] as well as log-barrier regularized NPG method [38]. We set the learning rate of all algorithms as η = 0 . 1 . We also fine-tune regularization parameters to find the entropy regularization factor τ = 0 . 05 and the log-barrier regularization factor λ = 0 . 005 . As discussed in Section 3.1 (the effect of c ), we observe in Figure 1b that the regularized algorithms fail to reach an exact Nash policy despite exhibiting good convergence performance at the start. The empirical results here align with our theoretical findings in Section 3.

Impact of c and δ We demonstrate the impact of the initialization dependent factor c k and suboptimality gap δ k in MPGs with the same experimental setup. Figure 1a depicts the change in value for c k , δ k , and the NE-gap. We can see from the figure that as the algorithm updates over iterations, the value of c k increases and approaches one, while δ k approaches some non-zero constant. Figure 1a shows that although δ k could be arbitrarily small in theory if we do not impose technical assumptions, it is not the case in general. Therefore, one should not automatically assume that the suboptimality gap diminishes with respect to iteration, as the limit entirely depends on the problem environment.

The effect of suboptimality gap δ ∗ in PG is illustrated in Section E of the appendix under a set of carefully constructed numerical examples. We verify that a larger gap indicates a faster convergence, which corroborates our theory.

4.2 Congestion Game

We now consider a class of MDPs where each state defines a congestion game. We borrow the specific settings for this experiment from [19] [8].

For the congestion game experiments, we consider the agent number n = 8 with the number of facilities |A i | = 4 , where A i = { A,B,C,D } as the corresponding individual action spaces. There are two states defined as S = { safe , distancing } . In each state, all agents prefer to be taking the same action with as many agents as possible. The reward for an agent selecting action k is defined by predefined weights w k s multiplied by the number of other agents taking the same action. Additionally, we set w A s < w B s < w C s < w D s and the reward in distancing state is reduced by some constant compared to the safe state. The state transition depends on the joint actions of all agents. If more than half of all agents take the same action, the system enters a distancing state with lower rewards. If the agents are evenly distributed over all actions, the system enters safe state with higher rewards.

We use episodic updates with T = 20 steps and collect 20 trajectories in each mini-batch and estimate the value function and Q-functions as well as the discounted visitation distribution. We use a discount factor of γ = 0 . 99 . We adopt the same step size used in [19, 8] and determine optimal step-sizes of

softmax PG and NPG with grid-search. Since regularized methods in Section 4.1 generally do not converge to Nash policies, they are excluded in this experiment. To make the experiment results align with previous works, we provide the L 1 distance between the current-iteration policies compared to Nash policies. We plot the mean and variance of L 1 distance across multiple runs in Figure 1c. Compared to the direct parameterized algorithms, the two softmax parameterized algorithms exhibit faster convergence, and softmax parameterized NPG has the best performance across all tested algorithms.

5 Conclusion and Discussion

In this paper, we studied Markov potential games in the context of multi-agent reinforcement learning. We focused on the independent natural policy gradient algorithm and studied its convergence rate to the Nash equilibrium. The main theorem of the paper shows that the convergence rate of NPG in MPGs is O (1 /K ) , which improves upon the previous results. Additionally, we provided detailed discussions on the impact of some problem factors (e.g., c and δ ) and compared our rate with the best known results with respect to these factors. Two empirical results were presented as a verification of our analysis.

Despite our newly proposed results, there are still many open problems that need to be addressed. One of the limitations of this work is the assumption of Markov potential games, the relaxation of which could extend our analysis to more general stochastic games. As a matter of fact, the gradient-based algorithm studied in this work will fail for a zero-sum game as simple as Tic-Tac-Toe. A similar analysis could also be applied to regularized games and potentially sharper bounds could be obtained. The agents are also assumed to receive gradient information from an oracle in this paper. When such oracle is unavailable, the gradient can be estimated via trajectory samples, which we leave as a future work. Other future directions are the convergence analysis of policy gradient-based algorithms in safe MARL and robust MARL, following the recent exploration of safe single-agent RL [9, 22, 41] and robust single-agent RL [21, 40].

Acknowledgments and Disclosure of Funding

This material is based upon work partially supported by the US Army Contracting Command under W911NF-22-1-0151 and W911NF2120064, US National Science Foundation under CMMI-2038625, and US Office of Naval Research under N00014-21-1-2385. The views expressed herein and conclusions contained in this document are those of the authors and should not be interpreted as representing the views or official policies, either expressed or implied, of the U.S. NSF, ONR, ARO, or the United States Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

References

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research , 22(1):4431-4506, 2021.

Jupiter Bakakeu, Schirin Baer, Hans-Henning Klos, Joern Peschke, Matthias Brossog, and Joerg Franke. Multi-agent reinforcement learning for the energy optimization of cyber-physical production systems. Artificial Intelligence in Industry 4.0: A Collection of Innovative Research Case-studies that are Reworking the Way We Look at Industry 4.0 Thanks to Artificial Intelligence , pages 143-163, 2021.

Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) , 38(2):156-172, 2008.

Shicong Cen, Fan Chen, and Yuejie Chi. Independent natural policy gradient methods for potential games: Finite-time global convergence with entropy regularization. In 2022 IEEE 61st Conference on Decision and Control (CDC) , pages 2833-2838. IEEE, 2022.

Shicong Cen, Yuejie Chi, Simon Shaolei Du, and Lin Xiao. Faster last-iterate convergence of policy optimization in zero-sum markov games. In The Eleventh International Conference on Learning Representations , 2022.

Constantinos Daskalakis, Dylan J Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. Advances in neural information processing systems , 33:5527-5540, 2020.

WDavis Dechert and SI O'Donnell. The stochastic lake game: A numerical solution. Journal of Economic Dynamics and Control , 30(9-10):1569-1587, 2006.

Dongsheng Ding, Chen-Yu Wei, Kaiqing Zhang, and Mihailo Jovanovic. Independent policy gradient for large-scale markov potential games: Sharper rates, function approximation, and game-agnostic convergence. In International Conference on Machine Learning , pages 51665220. PMLR, 2022.

Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems , 33:8378-8390, 2020.

Roy Fox, Stephen M Mcaleer, Will Overman, and Ioannis Panageas. Independent natural policy gradient always converges in markov potential games. In International Conference on Artificial Intelligence and Statistics , pages 4414-4425. PMLR, 2022.

Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review , pages 1-49, 2022.

Ben Hambly, Renyuan Xu, and Huining Yang. Policy gradient methods find the nash equilibrium in n-player general-sum linear-quadratic games. Journal of Machine Learning Research , 24(139), 2023.

Amélie Heliou, Johanne Cohen, and Panayotis Mertikopoulos. Learning with bandit feedback in potential games. Advances in Neural Information Processing Systems , 30, 2017.

Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning-a simple, efficient, decentralized algorithm for multiagent rl. In ICLR 2022 Workshop on Gamification and Multiagent Solutions , 2022.

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning , ICML '02, page 267-274, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc.

Sajad Khodadadian, Prakirt Raj Jhunjhunwala, Sushil Mahavir Varma, and Siva Theja Maguluri. On linear and super-linear convergence of natural policy gradient algorithm. Systems & Control Letters , 164:105214, 2022.

Robert Kleinberg, Georgios Piliouras, and Éva Tardos. Multiplicative updates outperform generic no-regret learning in congestion games. In Proceedings of the forty-first annual ACM symposium on Theory of computing , pages 533-542, 2009.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Stefanos Leonardos, Will Overman, Ioannis Panageas, and Georgios Piliouras. Global convergence of multi-agent policy gradient in markov potential games. In International Conference on Learning Representations , 2022.

Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Softmax policy gradient methods can take exponential time to converge. In Conference on Learning Theory , pages 3107-3110. PMLR, 2021.

Yan Li, Guanghui Lan, and Tuo Zhao. First-order policy optimization for robust markov decision process. arXiv preprint arXiv:2209.10579 , 2022.

Tao Liu, Ruida Zhou, Dileep Kalathil, PR Kumar, and Chao Tian. Policy optimization for constrained mdps with provable fast global convergence. arXiv preprint arXiv:2111.00552 , 2021.

Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems , 30, 2017.

Sergio Valcarcel Macua, Javier Zazo, and Santiago Zazo. Learning parametric closed-loop policies for markov potential games. In International Conference on Learning Representations , 2018.

Chinmay Maheshwari, Manxi Wu, Druv Pai, and Shankar Sastry. Independent and decentralized learning in markov potential games. arXiv preprint arXiv:2205.14590 , 2022.

Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning , pages 6820-6829. PMLR, 2020.

David H Mguni, Yutong Wu, Yali Du, Yaodong Yang, Ziyi Wang, Minne Li, Ying Wen, Joel Jennings, and Jun Wang. Learning in nonzero-sum stochastic games with potentials. In International Conference on Machine Learning , pages 7688-7699. PMLR, 2021.

Dov Monderer and Lloyd S Shapley. Potential games. Games and economic behavior , 14(1):124143, 1996.

John F Nash Jr. Equilibrium points in n-person games. Proceedings of the national academy of sciences , 36(1):48-49, 1950.

Guillaume Sartoretti, Yue Wu, William Paivine, TK Satish Kumar, Sven Koenig, and Howie Choset. Distributed reinforcement learning for multi-robot decentralized collective construction. In Distributed Autonomous Robotic Systems: The 14th International Symposium , pages 35-49. Springer, 2019.

Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295 , 2016.

Ziang Song, Song Mei, and Yu Bai. When can we learn general-sum markov games with a large number of players sample-efficiently? In International Conference on Learning Representations , 2022.

Alexander Trott, Sunil Srinivasa, Douwe van der Wal, Sebastien Haneuse, and Stephan Zheng. Building a foundation for data-driven, interpretable, and robust policy design using the ai economist. Interpretable, and Robust Policy Design using the AI Economist (August 5, 2021) , 2021.

Xiaofeng Wang and Tuomas Sandholm. Reinforcement learning to play an optimal nash equilibrium in team markov games. Advances in neural information processing systems , 15, 2002.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive markov games. In Conference on learning theory , pages 4259-4299. PMLR, 2021.

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems , 35:24611-24624, 2022.

Kaiqing Zhang, Zhuoran Yang, and Tamer Ba¸ sar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control , pages 321-384, 2021.

Runyu Zhang, Jincheng Mei, Bo Dai, Dale Schuurmans, and Na Li. On the global convergence rates of decentralized softmax gradient play in markov potential games. Advances in Neural Information Processing Systems , 35:1923-1935, 2022.

Runyu Zhang, Zhaolin Ren, and Na Li. Gradient play in stochastic games: stationary points, convergence, and sample complexity. 2021.

Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, PR Kumar, and Chao Tian. Natural actorcritic for robust reinforcement learning with function approximation. Advances in neural information processing systems , 2023.

Ruida Zhou, Tao Liu, Dileep Kalathil, PR Kumar, and Chao Tian. Anchor-changing regularized natural policy gradient for multi-objective reinforcement learning. Advances in Neural Information Processing Systems , 2022.

A Proof for Potential Games in Section 3.1

We first find the upper bound on the difference in marginalized rewards under two policies. The following lemma establishes a "smooth" relationship between the marginalized rewards and their respective policies. For ease of notation, we denote r ( π ) := E a ∼ π [ r ( a )] and ϕ ( π ) := E a ∼ π [ ϕ ( a )] , where these function values are taken expectation over policy π .

LemmaA.1. For any two sets of policies π, π ′ ∈ ∆( A 1 ) × ... × ∆( A n ) , the difference in marginalized reward of any agent i is bounded by the total variation distance of policies,

Proof. Given any π, π ′ ∈ ∆( A ) , we have

Next, we provide the proof of Lemma 3.1, which demonstrates the lower bound of the potential function difference.

Lemma A.2 (Restatement of Lemma 3.1) . Given policy π k and marginalized reward ¯ r k i ( a i ) , for π k +1 generated using the NPG update in (4) , we have the following inequality for any η < 1 √ n ,

Proof of Lemma 3.1. We introduce

to denote the mixed strategy where any agent with index j < i follows π k j and any agent with index ℓ > i follows π k +1 ℓ instead. Let ˜ r k i be the associated marginalized reward function, i.e.,

It follows that

We now decompose ϕ k +1 -ϕ k as follows:

The third equality follows from the definition (3). Since

we have

Additionally, we provide the proof of Lemma 3.2, which builds the relationship between potential function difference and NE-gap.

Lemma A.3 (Restatement of Lemma 3.2) . For function f k ( α ) defined in (5) and any α > 0 , we have the following inequality.

Proof. Recall that we define a k i p ∈ arg max a j ∈A i ¯ r k i ( a j ) =: A k i p and a k i q ∈ arg max a j ∈A i \A k i p ¯ r k i ( a j ) , where A k i p denotes the set of the best possible actions for agent i . For any i ∈ [ n ] , we have

where the first inequality holds due to Lemma D.1. By taking sum over all agents i ∈ [ n ] , we get

We next provide the following lemma on the structure of potential games. This lemma is not directly used in the analysis of this paper but instead provides a discussion on the function class of potential games, which could be of separate interest.

Lemma A.4 (Structure of potential game reward function class) . For any agent i , the set of reward functions for potential games, R i , has the following structure within the ambient space of R ∏ j |A j | .

̸

̸

Proof. The reward vector r i ∈ R ∏ n i =1 |A i | can be viewed as an ∏ n i =1 |A i | -dimensional linear vector. There are in total ( |A i | -1) ∏ j = i |A j | = ∏ n i =1 |A i | -∏ j = i |A j | independent constraints w.r.t r i and potential function ϕ . Since R i is a linear space with dimension ∏ j = i |A j | , R i is the space of valid reward functions for potential games.

̸

B Proof for Markov Potential Games in Section 3.2

We first introduce the following lemma to fill the gap between general MPGs and fully-cooperative MPGs. Define h i ( s, a ) := r i ( s, a ) -ϕ ( s, a ) , which implies V π i ( s ) = Φ π ( s ) + V π h i ( s ) .

Lemma B.1 (Restatement of Lemma 3.4) . Define ¯ A π h i ( s, a i ) with respect to h i similar to (2) . We then have

Proof of Lemma 3.4. Due to the definition of Markov potential game, we have

for any µ , π ′ i and π i . It then follows that

Denote [ ˜ P ] s,s ′ = E a ∼ π ′ i × π -i [ P ( s ′ | s, a )] as the transition matrix generated by policy π ′ i × π -i . Then

where d π ′ i ,π -i µ and µ are row vectors and ( I -γ ˜ P ) -1 is clearly full rank. Since the above relationship holds for any µ ∈ ∆ S , we get 0 = ∑ a i ( π ′ i ( a i | s ) -π i ( a i | s )) ¯ A π h i ( s, a i ) , ∀ s ∈ S .

Then, we analyze the lower bound of the potential value function difference.

̸

Lemma B.2 (Restatement of Lemma 3.5) . Given policy π k and marginalized advantage function ¯ A π k i ( s, a i ) , for π k +1 generated using NPG update in (7) , we have the following inequality,

Proof of Lemma 3.5. We introduce

to denote the mixed strategy where each agent with index j < i follows π k +1 j and each agent with index ℓ > i follows π k ℓ instead. Let ¯ A ˜ π k i,ϕ be the associated marginalized advantage value based on potential function, i.e.,

We now decompose Φ π k +1 -Φ π k as follows:

Since

Since we have

Next, we introduce the following lemma, which is the MPG version of Lemma 3.2. Lemma B.3. At iteration k , define

Then, for any α > 0

with c and δ k defined in Theorem 3.6.

Proof. For any i ∈ [ n ] , we have

where the first inequality holds due to Lemma D.1.

Combining the above lemmas, we show the proof of Theorem 3.6.

Proof of Theorem 3.6. (1 -γ ) 2

Then

Finally, we show a corollary of Theorem 3.6.

Corollary B.3.1 (Restatement of Corollary 3.6.1) . Consider a MPG that satisfies Assumption 3.2 with NPG update using algorithm 7. There exists K ′ , such that for any K ≥ 1 , choosing η = (1 -γ ) 2 2 √ nϕ max , we have

where M,c are defined as in Theorem 3.6.

Proof of Corollary 3.6.1. Following from the main paper, we know that there exists K ′ such that for all k ≥ K ′ , δ k ≥ δ ∗ 2 .

From the proof of Theorem 3.6, we know that for MPGs, we get the following relationship:

Therefore, for all k ≥ K ′ ,

Then, taking summation from K ′ to K -1 , we get

Additionally, we know that NE-gap ( π k ) ≤ ϕ max 1 -γ for all k by the definition of MPG. Combining the equations above, we have

C Extension to General MPG Formulation

In this section, we extend our results to a more general form of potential function.

Note that the detailed analysis is omitted for brevity. Proofs of lemmas and theorem in this section either exist in previous works or closely follow our analysis in Section B.

Definition C.1 (Alternate definition of MPG [8]) . For a MPG, at any state s , there exists a global function Φ π ( s ) : Π ×S → R such that

is true for any policies π i , π ′ i ∈ Π i .

This definition of MPG differs from Definition 2.1 and does not enforce the existence of additional structure in function ϕ . However, whether the two definitions are equivalent, similar to Lemma A.4, is still an open question to the best of the authors' knowledge.

We present the following Lemmas from [8], using which we provide a similar result to Lemma 3.5.

Lemma C.1 ([8]) . For any function Ψ π : Π → R , and any two policies π, π ′ ∈ Π ,

Lemma C.2 ([8]) . Consider a two-player game (group of players) with a common-payoff Markov game with state space S and action sets A 1 , A 2 , with reward function r , transition function P and policy sets Π 1 , Π 2 be defined with respect to general MDPs. Then for any π 1 , π ′ 1 ∈ Π 1 , π 2 , π ′ 2 ∈ Π 2 , the for value function V π ( µ ) ,

We refer to [8] for complete proof for Lemmas C.1 and C.2. By using Lemmas C.1 and C.2, we can replace Lemma 3.5 in our paper with the following lemma.

Lemma C.3. Given policy π k and marginalized advantage function ¯ A π k i ( s, a i ) , for π k +1 generated using independent NPG update, we have the following inequality,

The proof of Lemma C.3 closely follows proof of Lemma 3.5. First the potential function difference Φ π k +1 ( ρ ) -Φ π k ( ρ ) is decomposed following Lemma 3.5, with each term defined as a potential function that only differs by the local policy π i . Different from 3.5, the decomposition is based on Definition C.1 instead. Then each term can be bounded using Lemma C.1 with the addition of Lemma C.2 to handle the residual terms.

Combining Lemma C.3 and Lemma B.3, we can get a similar convergence guarantee without the explicit definition of ϕ , as stated in the following theorem:

Theorem C.4. Consider a MPG in the sense of Definition C.1 with isolated stationary points, where the policy update follows NPG update in (7). For any K ≥ 1 , choosing η = (1 -γ ) 2 4 nM 3 max i |A i | , we have

The proof of Theorem C.4 follows Theorem 3.6 and Corollary 3.6.1, and the only difference in the process is the different choice of η from Lemma C.3. Compared to Corollary 3.6.1, Theorem C.4 incorporates additional terms on M (distribution mismatch coefficient) and |A i | (size of action space), but the O (1 /ϵ ) order is still maintained.

D Supporting Lemmas

Lemma D.1. Given a reward vector r = [ r 1 , r 2 , · · · , r n ] where r 1 > r 2 > r 3 > · · · > r n with weights π 1 e αr 1 , π 2 e αr 2 , · · · , π n e αr n , we have

Proof. The above claim can be proved by induction. We first have

This gives a new reward vector

with weights

0

200

400

600

800

1000

Iterations

0.0

0.5

1.0

1.5

2.0

2.5

3.0

NE-Gap

Natrual Policy Gradient: Various

*

* =10

* =1

* =0.1

* =0.01

* =0.001

0

200

400

600

800

1000

Iterations

0.0

0.2

0.4

0.6

0.8

1.0

1.2

L1-accuracy

Natrual Policy Gradient: Various

*

* =10

* =1

* =0.1

* =0.01

* =0.001

Define π ′ n -1 := π n -1 + π n and by similar argument as above,

This gives a reward vector

with weights

By induction, we have

E Additional Numerical Experiments

We illustrate the impact of δ ∗ on the algorithm in the following example. We consider a 2-by-2 matrix game with the reward matrix

We run the NPG algorithm with the same initial policy under various values of δ ∗ ranging from 1 e -3 to 10 . The NE-gap and the L 1 accuracy over number of running steps are shown in Figures 2a-2b. As illustrated in the figure, δ ∗ plays an important role in the convergence of the algorithm. Larger δ ∗ results in faster convergence, which corroborates our theoretical findings.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an analysis of the independent natural policy gradient algorithm for Markov Potential games. Under technical assumptions on a sub-optimality gap problem dependent quantity and supposing access to exact (averaged) advantage functions, this paper provides a novel $O(1/\\epsilon)$ iteration complexity to guarantee that the average Nash gap along iterations is smaller than the accuracy \\epsilon, improving over the previously known $O(1/\\epsilon^2)$ iteration complexity in the same independent learning setting. After discussing the potential game setting as a warm-up and generalizing to the MPG setting, the paper provides simulations in a synthetic potential game and a congestion game.\n\nSTRENGTHS:\n- The convergence analysis improves over the $O(1/\\epsilon^2)$ iteration complexity in prior work under some technical assumptions.\n\n- To the best of my knowledge, the analysis provided in this paper is new and the proofs seem to be correct (I went through the appendix, please see some additional comments below). However, please see some comments below regarding related works and quantities/notations to be precised for clarifications. In my opinion, the main technical novelty is Lemma 3.2 (and Lemma B.3 relying on the technical lemma C.1) which is key to obtain the result in Theorem 3.3. This result is interesting and connects the sum of the ‘gaps’ (over agents) for a fixed temperature parameter $\\eta$ and for $\\eta$ going to infinity. \n\n- The paper is well-organized.\n\nWEAKNESSES:\n**1. About the suboptimality gap \\delta_K and the $O(1/K)$ convergence rate**:  \nTheorem 3.3 provides an upper-bound on the average NE-gap which depends on the K-dependent quantity $\\delta_K$ without further assumption, this provides a $O(1/K\\delta_K)$ convergence rate as mentioned in the paper. \nI think it should be clearly stated that the $O(1/K)$ convergence rate which is claimed in the abstract and in the contributions under ‘mild assumptions’ (as it is stated) is actually “asymptotic”.  Indeed, under assumption 3.2, the bound in Corollary 3.6.1 features an unknown constant (number of iterations) $K$’ that is not explicit in the problem parameters (such a constant is only guaranteed to exist under assumption 3.2). This is due to the fact that the NE gap is only controlled (independently of $\\delta_k$) for a large enough number of iterations as it appears in l. 500 in the appendix (p. 17) under the chosen assumption.  The results that the paper compares to in Table 1 have stronger guarantees in the sense that they are not asymptotic. This is not clear in the presentation and the comparison to prior work. \n\n**2. About Assumption 3.2**: this technical assumption guarantees that $\\delta_K$ is uniformly lower bounded away from zero but it seems hard to interpret or give a meaning to this assumption although the ‘sub-optimality gap’ $\\delta_k$ is a standard quantity in the bandit literature for instance. I am also not sure if this assumption is ‘mild’ as it is formulated. See the ‘Questions’ section for clarification questions regarding this assumption. \n\n**3. Discussion of related works**: Some relevant related works are missing in the discussion. \n\n**(a)** While Song et al. 2021 [27] is cited, it is not mentioned in the discussion that a $O(1/\\epsilon)$ iteration complexity has been achieved in that work for MPGs with a Nash-coordinate ascent algorithm (see Section 5 Algorithm 7, note that sample complexity is given and the $O(1/\\epsilon)$ iteration complexity appears in the proof when discarding the $O(1/\\epsilon^2)$ sample complexity needed for policy evaluation). However, this algorithm is ‘turn-based’ and requires coordination and hence is not independent as the present work. \n\n**(b)** Fox et al. 2022 derived an asymptotic convergence result for the independent natural policy gradient algorithm considered in this work that seemed to be later used by Zhang et al. 2022 [33] but this reference does not appear in related works. This same result seems also to be used in Proposition 3.1 of the paper. \n\n**(c)** The results shown in this paper seem to have some similarities with the asymptotic convergence analysis provided by Khodadadian et al. 2022 in the single agent setting. For instance, the analysis in that paper introduces the optimal advantage function gap (see $\\Delta$ as defined in Definition 2), a quantity similar to the sub-optimality gap $\\delta_k$ in the present paper (up to the multi-agent setting). However, I would like to point out that this is just for the purpose of comparison and the present work has to overcome many difficulties related to the game theoretic setting and the multi-agent nature of the problem that make this work very different from Khodadadian et al. 2022. The abstract precises that the result improves over “ $O(1/\\epsilon)$, that is achievable for the single-agent case”. Actually, Khodadadian et al. 2022 provide an asymptotic geometric convergence rate. Other recent related works even prove a global linear rate with increasing step sizes for the natural policy gradient algorithm (see for e.g. Xiao 2022, Section 4.2).  \n\nFox et al. Independent natural policy gradient always converges in Markov potential games, AISTATS 2022.\n\nKhodadadian et al. On linear and super-linear convergence of Natural Policy Gradient algorithm,  Systems and Control letters 2022.  \n\nXiao. On the convergence rates of policy gradient methods. JMLR 2022\n\n**(d)** minor remark: you might want to give reference to [Monderer and Shapley 1996, Potential Games, Games and Economic Behaviour 14, 124-143] which introduced this class of games in section 3.1 when you mention [12] (l. 128) which is much more recent. \n\n**4. Regarding the definition of Markov potential games** (Definition 2.1), for a fair comparison in Table 1, it might be worth mentioning that this definition differs from the one considered in [8,16] although it matches the definition used in [33, 34]. Indeed, the potential function in Definition 2.1 is supposed to have a discounted cumulative structure whereas such a structure is not available in the more general definition considered in [8,16]. As a matter of fact, the analysis becomes more challenging in [8] for instance since showing the potential function improvement is then more involved in that case, another decomposition different from the decomposition in Lemma B.2 l. 440 is then used to guarantee policy improvement (see Lemma in [8]). Also the dependence with respect to some parameters such as $(1-\\gamma)$ and the state action space sizes are usually improved with this additional structure. \n\n**5. Originality of the analysis:** While Lemma 3.2 and its use is indeed new, the potential function improvement lemmas (Lemma 3.1, Lemma 3.5) and their proofs in appendix follow prior techniques used in [4, 33]. I suggest that authors mention this somewhere in the main part or in the appendix and emphasize the novelty of the analysis (Lemma 3.2). For instance Lemma A.1 was proved in [4], the proof of Lemma A.2 is almost identical to the proof in [4] while the proof of Lemma B.2 is very similar to the proofs in [33].   \n\n**6. Clarity**: Overall, writing can be substantially improved in my opinion. Few minor details below: \n\n— l. 108: what do you mean by ‘multiple stationary points for the problem’? Stationary points if the potential function?\n\n— l. 134-135: not very clear, see the ‘Questions’ section below. \n\n— l. 136-137: ‘for any two sets of policies’. I guess you mean any two joint policies (in the product of the individual simplices) when you say sets.\n\n— l. 158: ‘They are related by the following lemma’, the constants you just defined in l. 157 or the quantities defined few lines above in l. 153?\n\n— notation $f(\\infty)$ is used in Lemma 3.2 and does not seem to be defined before although it is quite obvious this corresponds to $\\lim_{\\alpha \\to \\infty} f(\\alpha)$ as used in l. 153. \n\n— Lemma A.4 in the appendix: the statement and the proof are not very precise. What is $\\mathcal{R}_i$ (it seems only defined in the end of the proof of this lemma, or we can guess it from the title of the lemma)? I guess this is the set of reward functions which is known to have a particular structure for MPGs as you write it. $\\mathcal{R}_i$ is a linear space in which ambient space? Please precise the proof even if we can guess the idea. Also, where is this result used? \n\n—  Lemma C.1: if rewards are vectors, please precise somewhere in notations that inequalities hold for all the entries of the vectors.  \n\n**Typos (main part and Appendix)**: \n l. 91: $V_i(s)$; do you mean $V_i^{\\pi}(s)$? \n\nl. 476-477 (proof of Lemma B.2 in appendix): $\\bar{A}_{f_i}^{\\pi^k}$, what is $f_i$? I guess you mean $h_i$.\n \nl. 479 to l. 480: $\\pi_i^{k+1}(\\cdot|s)$, $s$ is missing in the first term of the last equation and in the KL divergences.\n\nl. 503 (proof of corollary 3.6.1): is there a missing $\\phi_{\\max}$ in front of $K’$ in the first inequality of the page (given l. 501)?\n\nQUESTIONS:\n1. The assumption of ‘isolated stationary policies’ seems to be needed to guarantee that $c>0$ as stated in Theorem 3.6 (as it is also stated and used in [33]). However this assumption is not stated in the theorem. How does the theorem guarantee that $c> 0$? \n\n2. Please clarify the definition of ‘isolated stationary policies’. What do we mean by ‘stationary’ policies in this context? Although such a terminology is used in [33], it is not clear to the reader what this means, especially that ‘stationary’ has also another meaning for policies (time-independent). The reference Fox et al. 2022 above may help with this. \n\n3. About assumption 3.2, isn’t it possible to use the assumption that the ‘stationary policies are isolated’ instead to obtain corollary 3.6.1? According to the comments l 248-250 in the paper (a short proof in the appendix may be useful then to state this implication properly), this assumption would guarantee that Assumption 3.2 hold. Moreover, the assumption that the ‘stationary policies are isolated’ seems to be already needed in Theorem 3.6 to guarantee that $c >0$. Would this mean that only the assumption that ‘stationary policies are isolated’ would be then sufficient for Theorem 3.6 to hold without needing Assumption 3.2? \n\n4. Could you please precise the mathematical definition of $\\pi_i^{*k}$? It seems to be only defined once with words in l. 134-135. Why should it be unique (‘the optimal solution …’)? Do you mean it is an optimal policy in the sense that it maximizes the averaged advantage function w.r.t. the policy of other agents but i being fixed to $\\pi_{-i}^k$ (or the averaged reward function in the potential games setting)? This is also important to clarify the limit statement in l. 153.  \n\n5. l. 153: why do we have that $f(\\eta) \\geq 0$?\n\nLIMITATIONS:\nThe paper mentions the MPG setting as a limitation in the conclusion. \n\n**Extension to the stochastic setting:** While the analysis in the deterministic setting is an important step towards understanding the more practical stochastic setting where exact advantage functions are not available and can only be estimated via sampled trajectories, this analysis does not seem to be easily extendable to the aforementioned stochastic setting. This seems to be related to the fact that showing that the constant c is positive in the stochastic setting seems to be much harder if not hopeless even in a single agent setting (see for e.g. Mei et al. 2021). However, this is a limitation that also applies to prior work in MPGs analyzing independent natural policy gradient such as Zhang et al. 2022 [33]. A comment on this or an additional remark in the paper would be welcome. For instance, [8] analyzes the sample complexity in the stochastic setting but their algorithm does not cover the case where the regularization in the policy mirror descent-like update rule is not euclidean, KL regularization (which leads to natural policy gradient) is not covered. \n\n   Mei et al. ‘Understanding the effect of stochasticity in policy optimization’ (Neurips 2021)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a new analysis for the (Markov) potential games with a $O(1/\\epsilon)$ convergence rate. The new results are problem-dependent and may be a tighter guarantee for certain classes of potential games. Asymptotic guarantees are also provided to elaborate on the problem-dependent nature of the results.\n\nSTRENGTHS:\nThe results it presents showcase an improvement over previous results on (markov) potential games. The new rates are now unaffected by the number of actions and are at the order of $O(1/\\epsilon)$. The problem-dependent nature of the results may provide a tighter guarantee on a certain class of potential games and the new analysis methods may lead to future works. The results are extensively discussed and empirical results are provided to corroborates the theoretical results.\n\nWEAKNESSES:\nThe new results do not seem to be directly comparable to the previous ones due to the use of a suboptimality gap. I encourage the authors to explicitly state this early in the paper to avoid confusion (e.g. in Table 1). I also encourage the authors to complement the work with more empirical results. For example, it would be interesting to see how the algorithm performs against previous ones when the suboptimal gap is very small. It may also be helpful to state when the suboptimality gap is very small, the results can degenerate into previous results.\n\nQUESTIONS:\n1. In Table 1, the results for PG + softmax [33] also include a c in the denominator. Is the c defined as the same as this paper?\n2. It seems that in [33], the c in the denominator can be alleviated by using log barrier regularization. Will log barrier regularization have the same effect on the results presented in this paper?\n3. In Definition 2.1, the potential function is assumed to take a specific form with $\\phi$. However, this form does not seem to be needed in previous analyses. How important is this specific form to the analysis and can it be lifted?\n\nLIMITATIONS:\nYes, it is discussed in the conclusion part.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article considers what the authors describe to be “natural policy gradient” learning in normal form and Markov potential games. This is the discrete time algorithm in which individual players' mixed strategies are multiplied by a softmax response to the opponent mixed strategies then normalised. Convergence rates are derived in both normal form and the Markovian potential games.\n\nSTRENGTHS:\nThe article is nicely written, and the claimed results are supported by the theory.\n\nThe convergence rate results are nice, given the general difficulty to provide convergence rates in anything multiagent, and especially in Markovian games - of course restricting to situations in which there is a potential function for the full Markovian game makes things much less impossible, but it is still a hard problem.\n\nThe paper is nicely self-contained, and a real pleasure to read.\n\nWEAKNESSES:\nI have some doubts over the claimed results which I would like to see clarified.\n\nThe authors claim that Thm3. Implies a 1/eps convergence rate. I don’t buy it I’m afraid. c and delta_K are not controlled. They could be arbitrarily small, at least without further work. I suspect c is okay, although a sudden switch in best response could easily result in a very small pi_i^k(br_i(pi_{-i}^k)) coming into c late in the process. And since delta is the optimality gap when playing a mixed strategy, I find it very difficult to see how to constrain it effectively. (I think line 158 tells us that c is the smallest ever pi_i^k(br(pi_{-i}^k)), and delta_K is the smallest optimality gap that occurs up to time K – if I have misinterpreted this then my objections may dissipate!)\n\nThe synchronous form of \"learning\", and the tight construction of Markovian potential games with average reward, means the step up to Markovian settings is much less then in a less restrictive setting.\n\nA more philosophical point is that the article assumes players can receive oracle information about the long term payoff of any action. While it makes for a nice compact paper, I think for NeurIPS the authors need to at least posit some suggestions for where learners might be able to access the advantage functions that are required to implement the method.\n\nFurthermore, the dynamic is well known as the multiplicative weights algorithm, and I would expect the authors to compare their results with those presented under the multiplicative weights description.\n\nQUESTIONS:\n1. In (5), the function f has dependence only on alpha. However there is also important dependence on k, and on pi_{-i}^k. This may seem like pedantry, but I think it’s important (see below).\n2. In lines 154/155 the definition of a^k_{i_q} is clumsy, due to possible multiplicities in the argmax for a^k_{i_p}. I think that a^i_q is only needed to define the optimality gap? If this is the case, why not just define delta^k directly max_a r^i(a) – max_{a\\notin \\argmax} r(a), or something like that? The current formulation taking a_i_q to be the max that is not a_i_p is incorrect, is noted to be incorrect in the text, but is used anyway.\n3. In (6) we begin to get bitten by the lack of care in defining what the functions depend on. In particular, since the r_I^k functions depend on mixed strategies pi_{-i}^k, so does the delta term. And so delta_k may well be extremely small. Have I missed something here?\n4. On line 159, we see that c_K and delta_K are the smallest such quantities that are observed in the first K iterations of the algorithm, and indeed that c is minimum c observed in the limit. The notation means that this path-dependency is somewhat obscured. The authors try to argue, later, that these quantities are not small. But I do not think the argument has been made strongly enough. Furthermore, it seems strange to take a limit of c_K values, but not take a limit of delta_K values in the theory that follows? This is clearly very relevant to my point made in \"weaknesses\".\n5. First inequality in the proof of Theorem 3.3, I think should be an equality?\n6. Line 259, I think we cannot claim that |delta^k-delta*| is small, since we are only assuming a lim inf?\n\nLIMITATIONS:\nNo discussion of limitations is presented. I don't feel such a discussion would add much to the paper, but the review form asks the question!", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies the convergence of Natural Policy Gradient method (NPG) in Markov Potential Game. Under stronger assumptions, the convergence result improves upon previous ones.

STRENGTHS:
Under the considered setting, NPG is shown to achieve a $1/K$ convergence rate for multi-agent Markov Potential Game, matching the convergence rate in single-agent setting. Discussion on the parameters that appear in the convergence rate is presented in detail.

WEAKNESSES:
The convergence result only makes sense under the assumption that $c>0, \delta>0$. However, such assumption could be too strong and unnecessary (as the table 1 indicates).

QUESTIONS:
Line 177: "A small value for c generally describes a policy that is stuck at some regions far from an NE, yet the policy gradient is small. It has been shown in [17] that these ill-conditioned problems could take exponential time to solve even in single-agent settings."

In my opinion, this justification is not suitable. In single-agent settings, NPG has a nice convergence that does not depend on $1/c$, while [17] shows that PG can take exponential many iterations to converge. Thus, taking [17] as an excuse, it is okay to introduce certain assumptions (e.g. concentrability) to ensure better convergence, but directly assuming $c>0$ is still too strong (and possibly unnecessary).

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06