Title: Multi-Player Zero-Sum Markov Games with Networked Separable Interactions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Markov Games

Local Interaction

PPAD-Hardness

Fictitious Play
Abstract: We study a new class of  Markov games, \textit{(multi-player) zero-sum Markov Games} with {\it Networked separable interactions} (zero-sum NMGs), to model the local interaction structure in non-cooperative multi-agent sequential decision-making. We define a zero-sum NMG as a model where {the payoffs of the auxiliary games associated with each state are zero-sum and} have some separable (i.e., polymatrix) structure across the neighbors over some interaction network. 
We first identify the necessary and sufficient conditions under which an MG can be presented as a zero-sum NMG, and show that the set of Markov coarse correlated equilibrium (CCE) collapses to the set of Markov Nash equilibrium (NE) in these games, in that the {product of} per-state marginalization of the former for all players yields the latter. Furthermore,  we show that finding approximate Markov \emph{stationary}  CCE  in infinite-horizon discounted zero-sum NMGs is \texttt{PPAD}-hard, unless the underlying network has a ``star topology''. Then, we propose fictitious-play-type dynamics, the classical learning dynamics in normal-form games, for zero-sum NMGs, and establish convergence guarantees to Markov stationary NE under a star-shaped network structure. Finally, in light of the hardness result, we focus on computing a Markov \emph{non-stationary} NE and provide finite-iteration guarantees for a series of value-iteration-based algorithms. We also provide numerical experiments to corroborate our theoretical results.

Multi-Player Zero-Sum Markov Games with Networked Separable Interactions

Abstract

We study a new class of Markov games, (multi-player) zero-sum Markov Games with Networked separable interactions (zero-sum NMGs), to model the local interaction structure in non-cooperative multi-agent sequential decision-making. We define a zero-sum NMG as a model where the payoffs of the auxiliary games associated with each state are zero-sum and have some separable (i.e., polymatrix) structure across the neighbors over some interaction network. We first identify the necessary and sufficient conditions under which an MG can be presented as a zero-sum NMG, and show that the set of Markov coarse correlated equilibrium (CCE) collapses to the set of Markov Nash equilibrium (NE) in these games, in that the product of per-state marginalization of the former for all players yields the latter. Furthermore, we show that finding approximate Markov stationary CCE in infinite-horizon discounted zero-sum NMGs is PPAD -hard, unless the underlying network has a 'star topology'. Then, we propose fictitious-play-type dynamics, the classical learning dynamics in normal-form games, for zero-sum NMGs, and establish convergence guarantees to Markov stationary NE under a star-shaped network structure. Finally, in light of the hardness result, we focus on computing a Markov non-stationary NE and provide finite-iteration guarantees for a series of value-iteration-based algorithms. We also provide numerical experiments to corroborate our theoretical results.

1 Introduction

Nash Equilibrium (NE) has been broadly used as a solution concept in game theory, since the seminal works of [1, 2]. Perhaps equally important, NE is also deeply rooted in the prediction and analysis of learning dynamics in multi-agent strategic environments: it may appear as a natural outcome of many non-equilibrating learning processes of multiple agents interacting with each other [3, 4]. A prominent example of such learning processes is fictitious play (FP) [5, 6], in which myopic agents estimate the opponents' play using history, and then choose a best-response action (based on their payoff matrix) against this estimate, as if the opponents use it as their stationary strategy. The focus of these studies has initially been on the convergence to NE in zero-sum games (see [5, 6], and also [7, 4]), and in games with aligned objective (identical-interest and potential games, see [8]). Since then, FP has been shown to converge to NE in more important classes of games, including 2x n games [9, 10], 'one-against-all' games [11], and zero-sum polymatrix games [12], justifying the prediction power of NE in learning in normal-form/matrix games.

Some of these results have recently been extended to the stochastic game (also known as the Markov game (MG)) setting, a model for multi-agent sequential decision-making with state transition dynamics, first introduced in [13]. In particular, [14, 15, 16, 17, 18] have studied best-response-type learning dynamics in two-player zero-sum MGs, and [19, 17] have studied that in multi-player

Asuman Ozdaglar MIT

identical-interest games. Following the same path as studying matrix games, one natural question arises: Are there other classes of MGs beyond two-player zero-sum and identical-interest cases that allow natural learning dynamics, e.g., fictitious play, to justify NE as the long-run emerging outcome?

On the other end of the spectrum, it is well-known that for general-sum normal-form games, the special case of MGs without the state transition dynamics, computing an NE is intractable [20, 21]. Relaxed solution concepts as (coarse) correlated equilibrium ((C)CE) have thus been favored when it comes to equilibrium computation for general-sum, multi-player games [22, 3]. Encouragingly, when the interactions among players have some networked separable structure, also known as being polymatrix , computing NE may be made tractable even in multi-player settings. This has been instantiated in the seminal works [23, 24] in the normal-form game setting, which showed that any CCE collapses to the NE in such games when the payoffs are zero-sum. Thus, any algorithms that can efficiently compute the CCE in such games will lead to the efficient computation of the NE.

In fact, besides being of theoretical interest, multi-player zero-sum games with networked separable interactions also find a range of applications, including security games [24], fashion games [25, 26, 27], and resource allocation problems [28]. These examples, oftentimes, naturally involve some state transition that captures the dynamics of the evolution of the environment in practice. For example, in the security game, the protection level or immunity of a target increases as a function of the number of past attacks, leading to a smaller probability of an attack on the target being successful. Hence, it is imperative to study such multi-player zero-sum games with state transitions. As eluded in the recent results [29, 30], such a transition from stateless to stateful cases may not always yield straightforward and expected results, e.g., computing stationary CCE can be computationally intractable in stochastic games, in stark contrast to the normal-form case where CCE can be efficiently computed. This naturally prompts another question: Are there other types of (multi-player) MGs that may circumvent the computational hardness of computing NE/CCE?

In an effort to address these two questions, we introduce a new class of Markov games (multi-player) zero-sum Markov games with networked separable interactions (zero-sum NMGs). We summarize our contributions as follows, and defer a more detailed literature review to Appendix A.

Contributions. First, we introduce a new class of non-cooperative Markov games: (multi-player) zero-sum MGs with Networked separable interactions (zero-sum NMGs), wherein the payoffs of the auxiliary-games associated with each state, i.e., the sum of instantaneous reward and expectation of any estimated state-value functions, possess the multi-player zero-sum and networked separable (i.e., polymatrix) structure as in [31, 28, 23, 24] for normal-form games, a strict generalization of the latter. We also provide structural results on the reward and transition dynamics of the game, as well as examples of this class of games. Specifically, for a Markov game to qualify as a zero-sum NMG, if and only if its reward function has the zero-sum polymatrix structure, and its transition dynamics is an ensemble of multiple single-controller transition dynamics that are sampled randomly at each state (see Remark 3 for more details). This transition dynamics covers the common ones in the MG literature, including the single-controller and turn-based dynamics. Second, we show that Markov CCE and Markov NE collapse in that the product of per-state marginal distributions of the former yields the latter, making it sufficient to focus on the former in equilibrium computation. We then show the PPAD -hardness [32] of computing the Markov stationary equilibrium, a natural solution concept in infinite-horizon discounted MGs, unless the underlying network has a star-topology. This is in contrast to the normal-form case where CCE is always computationally tractable. Third, we study the fictitious-play property [8] of zero-sum NMGs, showing that the fictitious-play dynamics [16, 19] converges to the Markov stationary NE, for zero-sum NMGs with a star-shaped network structure. Finally, in light of the hardness of computing stationary equilibria, we develop a series of value-iteration-based algorithms for computing a Markov non-stationary NE of zero-sum NMGs, with finite-iteration guarantees. We also provide numerical experiments to corroborate our theoretical results in Section G. We hope our results serve as a starting point for studying this networked separable interaction structure in non-cooperative Markov games.

Notation. For a real number c , we use ( c ) + to denote max { c, 0 } . For an event E , we use 1( E ) to denote the indicator function such that 1( E ) = 1 if E is true, and 1( E ) = 0 otherwise. We define multinomial distribution with probability ( w i ) i ∈N as Multinomial ( ( w i ) i ∈N ) . We denote the uniform distribution over a set S as Unif( S ). We denote the Bernoulli distribution with probability p as Bern( p ). The sgn function is defined as sgn ( x ) = 2 × 1( x ≥ 0) -1 . The KL-divergence between two probability distributions p, q is denoted as KL( p, q ) = E p [log( p/q )] . For a graph G = ( N , E ) , we denote the set of neighboring nodes of node i ∈ N as E i (without including i ). The maximum norm of a matrix X ∈ R m × n , denoted as ∥ X ∥ max , is defined as ∥ X ∥ max := max i ∈ [ m ] ,j ∈ [ n ] | X i,j | .

2 Preliminaries

2.1 Markov games

We define a Markov game as a tuple ( N , S , A , H, ( P h ) h ∈ [ H ] , ( r h,i ) h ∈ [ H ] ,i ∈N , γ ) , where N = [ n ] is the set of players, S is the state space with |S| = S , A i is the action space for player i with |A i | = A i and A = ∏ i ∈N A i , H ≤ ∞ is the length of the horizon, P h : S × A → ∆( S ) captures the state transition dynamics at timestep h , r h,i ∈ [0 , R ] is the reward function for player i at timestep h , bounded by some R > 0 , and γ ∈ (0 , 1] is a discount factor. An MG with a finite horizon ( H < ∞ ) is also referred to as an episodic MG, while an MG with an infinite horizon ( H = ∞ ) and γ < 1 is referred to as an infinite-horizon γ -discounted MG. When H = ∞ , we will consider the transition dynamics and reward functions, denoted by P and ( r i ) i ∈N , respectively, to be independent of h . Hereafter, we may use agent and player interchangeably.

Policy. Consider the stochastic Markov policy for player i , denoted by π i , as π i := { π h,i : S → ∆( A i ) } h ∈ [ H ] . A joint Markov policy is a policy π := { π h : S → ∆( A ) } h ∈ [ H ] , where π h : S → ∆( A ) decides the joint action of all players that can be potentially correlated. A joint Markov policy is a product Markov policy if π h : S → ∏ i ∈N ∆( A i ) for all h ∈ [ H ] , and is denoted as π = π 1 × π 2 ×··· × π n . When the policy is independent of h , the policy is called a stationary policy. We let a h denote the joint action of all agents at timestep h . Unless otherwise noted, we will work with Markov policies throughout. We denote π ( s ) ∈ ∆( A ) as the joint policy at state s ∈ S .

Value function. For player i , the value function under joint policy π , at timestep h and state s h is defined as V π h,i ( s h ) := E π [ ∑ H h ′ = h γ h ′ -h r h ′ ,i ( s h ′ , a h ′ ) ∣ ∣ s h ] , which denotes the expected cumulative reward for player i at step h if all players adhere to policy π . We also define V π h,i ( ρ ) := E s h ∼ ρ [ V π h,i ( s h )] for some state distribution ρ ∈ ∆( S ) . We denote the Q -function for the i -th player under policy π , at step h and state s h as Q π h,i ( s h , a h ) := E π [ ∑ H h ′ = h γ h ′ -h r h ′ ,i ( s h ′ , a h ′ ) ∣ ∣ s h , a h ] , which determines the expected cumulative reward for the i -th player at step h , when starting from the state-action pair ( s h , a h ) . For the infinite-horizon discounted setting, we also use V π i and Q π i to denote V π 1 ,i and Q π 1 ,i for short, respectively.

Approximate equilibrium. Define an ϵ -approximate Markov perfect Nash equilibrium as a product policy π , which satisfies max i ∈N max µ i ∈ (∆( A i )) |S|× H ( V µ i ,π -i h,i ( ρ ) -V π h,i ( ρ )) ≤ ϵ for all ρ ∈ ∆( S ) and h ∈ [ H ] , where π -i represents the marginalized policy of all players except player i . Define an ϵ -approximate Markov coarse correlated equilibrium as a joint policy π , which satisfies max i ∈N max µ i ∈ (∆( A i )) |S|× H ( V µ i ,π -i h,i ( ρ ) -V π h,i ( ρ )) ≤ ϵ for all ρ ∈ ∆( S ) and h ∈ [ H ] . In the infinite-horizon setting, they can be equivalently defined as satisfying max s ∈S max i ∈N max µ i ∈ (∆( A i )) |S| ( V µ i ,π -i i ( s ) -V π i ( s )) ≤ ϵ . If the above conditions only hold for certain ρ and h = 1 , we refer to them as Markov non-perfect NE and CCE, respectively. Unless otherwise noted, we hereafter focus on Markov perfect equilibria, and sometimes refer to them simply as Markov equilibria when it is clear from the context. In the infinite-horizon setting, if additionally, the policy is stationary , then they are referred to as a Markov stationary NE and CCE, respectively.

2.2 Multi-player zero-sum games with networked separable interactions

As a generalization of two-player zero-sum matrix games, (multi-player) zero-sum polymatrix games have been introduced in [31, 28, 23, 24]. A polymatrix game , also known as a separable network game is defined by a tuple ( G = ( N , E r ) , A = ∏ i ∈N A i , ( r i,j ) ( i,j ) ∈E r ) . Here, G is an undirected connected graph where N = [ n ] denotes the set of players and E r ⊆ N × N denotes the set of edges describing the rewards' networked structures, where the graph neighborhoods represent the interactions among players. For each edge, a two-player game is defined for players i and j , with action sets A i and A j , and reward functions r i,j : A i ×A j → R , and similarly for r j,i . The reward for player i for a given joint action a = ( a i ) i ∈N ∈ ∏ i ∈N A i is calculated as the sum of the rewards for all edges involving player i , that is, r i ( a ) = ∑ j :( i,j ) ∈E r r i,j ( a i , a j ) . To be consistent with our terminology later, hereafter, we also refer to such games as (multi-player) Games with Networked separable interactions (NGs) .

In a zero-sum polymatrix game (i.e., a (multi-player) zero-sum Game with Networked separable interactions (zero-sum NG)), the sum of rewards for all players at any joint action a = ( a i ) i ∈N ∈ ∏ i ∈N A i equals zero, i.e., ∑ i ∈N r i ( a ) = 0 . One can define the policy of agent i , i.e., π i ∈ ∆( A i ) , so that the agent takes actions by sampling a i ∼ π i ( · ) . Note that π i can be viewed as the reduced case of the policy defined in Section 2.1 when S = ∅ and H = 1 . The expected reward for player i

under π can then be computed as:

where r i denotes the matrix r i := ( r i, 1 , . . . , r i, ( i -1) , 0 , r i, ( i +1) , . . . , r i,n ) ∈ R |A i |× ∑ i ∈N |A i | and π := ( π ⊺ 1 , π ⊺ 2 , . . . , π ⊺ n ) ⊺ ∈ R ∑ i ∈N |A i | . We define r := ( r ⊺ 1 , r ⊺ 2 , . . . , r ⊺ n ) ⊺ ∈ R ∑ i ∈N |A i |× ∑ i ∈N |A i | . Then in this case we have ∑ i ∈N r i ( π ) = 0 for any policy π . See more prominent application examples of zero-sum polymatrix games in [23, 24].

3 Multi-Player (Zero-Sum) MGs with Networked Separable Interactions

We now introduce our model of multi-player zero-sum MGs with networked separable interactions.

3.1 Definitions

Definition 1. An infinite-horizon γ -discounted MG is called a (multi-player) MG with Networked separable interactions (NMG) characterized by a tuple ( G = ( N , E Q ) , S , A , P , ( r i ) i ∈N , γ ) if for any function V : S → R , defining Q V i ( s, a ) := r i ( s, a ) + γ ∑ s ′ ∈S P ( s ′ | s, a ) V ( s ′ ) , there exist a set of functions ( Q V i,j ) ( i,j ) ∈E Q and an undirected connected graph G = ( N , E Q ) such that Q V i ( s, a ) = ∑ j ∈E Q,i Q V i,j ( s, a i , a j ) holds for every i ∈ N , s ∈ S , a ∈ A , where E Q,i denotes the neighbors of player i induced by the edge set E Q (without including i ). When it is clear from the context, we represent the NMG tuple simply as G = ( N , E Q ) .

A finite-horizon MG is called a (multi-player) MG with Networked separable interactions if for any set of functions V := { V h } h ∈ [ H +1] where V h : S → R , defining Q V h,i ( s, a ) := r h,i ( s, a ) + γ ∑ s ′ ∈S P h ( s ′ | s, a ) V h +1 ( s ′ ) , there exist a set of functions ( Q V h,i,j ) ( i,j ) ∈E Q ,h ∈ [ H ] such that Q V h,i ( s, a ) = ∑ j ∈E Q,i Q V h,i,j ( s, a i , a j ) holds for every i ∈ N , s ∈ S , h ∈ [ H ] , a ∈ A .

A (multi-player) NMG is called a (multi-player) zero-sum MG with Networked separable interactions (zero-sum NMG) if additionally ( G = ( N , E Q ) , A = ∏ i ∈N A i , ( r i,j ( s ) := Q 0 i,j ( s )) ( i,j ) ∈E Q ) forms a zero-sum NG for all s ∈ S in the infinite-horizon γ -discounted case, or ( G = ( N , E Q ) , A = ∏ i ∈N A i , ( r h,i,j ( s ) := Q 0 h,i,j ( s )) ( i,j ) ∈E Q ) forms a zero-sum NG for all s ∈ S and h ∈ [ H ] in the finite-horizon case.

Regarding the assumption that the above conditions hold under any (set of) functions V , one may understand this as a structural requirement to inherit the polymatrix structure in the Markov game case. It is natural since { Q V i } i ∈N would play the role of the payoff matrix in the normal-form case, when value-(iteration) based algorithms are used to solve the MG. As our hope is to exploit the networked structure in the payoff matrices to develop efficient algorithms for solving such MGs, if we do not know a priori which value function estimate V will be encountered in the algorithm update, the networked structure may easily break if we do not assume them to hold for all possible V . Moreover, such a definition easily encompasses the normal-form case, by preserving the polymatrix structure of the reward functions (when substituting V to be a zero function). Some alternative definition (see Remark 2) may not necessarily preserve the polymatrix structure of even the reward functions in a consistent way (see Section B.4 for a concrete example). We thus focus on Definition 1, which at least covers the polymatrix structure of the reduced case regarding only reward functions.

Indeed, such a networked structure in Markov games may be fragile. We now propose both sufficient and necessary conditions for the reward function's structure and the transition dynamics of the MG, to be an NMG. Here we focus on the infinite-horizon discounted setting for a simpler exposition. For finite-horizon cases, a similar statement holds, which is deferred to Appendix B. We also defer the full statement and proof of the following result to Appendix B. We first introduce the definition of decomposability and the set N C , which will be used in establishing the conditions. For a graph G = ( N , E ) , we define N C := { i | ( i, j ) ∈ E for all j ∈ N} , which may be an empty set if no such node i exists.

Definition 2 (Decomposability) . A non-negative function f : X |D| → R + ∪ { 0 } is decomposable with respect to a set D ̸ = ∅ if there exists a set of non-negative functions ( f i ) i ∈D with f i : X → R + ∪ { 0 } , such that f ( x ) = ∑ i ∈D f i ( x i ) holds for any x ∈ X |D| . A non-negative function f : X |D| → R + ∪ { 0 } is decomposable with respect to a set D = ∅ , if there exists a non-negative constant f o such that f ( x ) = f o holds for any x ∈ X |D| .

̸

Proposition 1. For a given graph G = ( N , E Q ) , an MG ( N , S , A , P , ( r i ) i ∈N , γ ) with more than two players is an NMG with respect to G if and only if: (1) r i ( s, a i , · ) is decomposable with respect to E Q,i for each i ∈ N , s ∈ S , a i ∈ A i , i.e., r i ( s, a ) = ∑ j ∈E Q,i r i,j ( s, a i , a j ) for a set of functions { r i,j ( s, a i , · ) } j ∈E Q,i , and (2) the transition dynamics P ( s ′ | s, · ) is decomposable with respect to the set N C of this G , i.e., P ( s ′ | s, a ) = ∑ i ∈N C F i ( s ′ | s, a i ) for a set of functions { F i ( s ′ | s, · ) } i ∈N C if N C = ∅ , or P ( s ′ | s, a ) = F o ( s ′ | s ) for some constant function (of a ) F o ( s ′ | s ) if N C = ∅ . Moreover, an MG qualifies as a zero-sum NMG if and only if it satisfies an additional condition: the NG, characterized by ( G , A , ( r i,j ( s )) ( i,j ) ∈E Q ) , is a zero-sum NG for all s ∈ S . In the case of two players, every (zero-sum) Markov game becomes a (zero-sum) NMG.

Remark 1 (Stronger sufficient condition) . We note that for an MG ( N , S , A , P , ( r i ) i ∈N , γ ) , if for every agent i , r i ( s, a i , · ) is decomposable with respect to some E r ⊆ E Q , and P ( s ′ | s, · ) is decomposable with respect to some N P ⊆ N C , then one can still prove the if part, i.e., there exists some G = ( N , E Q ) such that the game is an NMG with respect to this G . See Figure 1 for the illustration. This is because by our definition, being decomposable with respect to a subset implies being decomposable with respect to a larger set, as one can choose the functions f i for the i in the complement of the subset to be simply zero. We chose to state as in Proposition 1 just for the purpose of presenting both the if and only if conditions in a concise and unified way.

Remark 2 (An alternative NMG definition) . Another reasonable definition of NMG may be as follows: if for any policy π , there exist a set of functions ( Q π i,j ) ( i,j ) ∈E Q and an undirected connected graph G = ( N , E Q ) such that Q π i ( s, a ) = ∑ j ∈E Q,i Q π i,j ( s, a i , a j ) holds for every i ∈ N , s ∈ S , a ∈ A . Note that such a definition can be useful in developing policy -based algorithms (while Definition 1 is more amenable to developing value -based algorithms), e.g., policy iteration, policy gradient, actor-critic methods, where the Q -value under certain policy π will appear in the updates and may need to preserve certain decomposability structure, for any policy π encountered in the algorithm updates. However, in this case, we cannot always guarantee the decomposability of P ( s ′ | s, · ) or r i ( s, a i , · ) . For example, if we assume that r i ( s, a ) = 0 for every i ∈ N , s ∈ S , a ∈ A , then Q π i ( s, a ) = ∑ j ∈E Q,i 0 and thus Q π i ( s, a i , · ) is always decomposable regardless of P ( s ′ | s, a ) . However, interestingly, we can show that the decomposability of the transition dynamics and the reward function as in Proposition 1 can still be guaranteed, as long as some degenerate cases as above do not occur. In particular, if there exist no i ∈ N and s ∈ S such that Q π i ( s, a ) is a constant function of a for any π , then the results in Proposition 1 and hence after still hold. We defer a detailed discussion on this alternative definition to Appendix B.

̸

Remark 3 (Implication of decomposable transition dynamics) . For an MG to be an NMG, by Proposition 1 the transition dynamics should be decomposable, i.e., P ( · | s, a ) = ∑ j ∈N C F j ( · | s, a j ) or P ( · | s, a ) = F o ( s ′ | s ) . We first focus on the discussion of the former case. Define w j ( s, a j ) := ∑ s ′ ∈S F j ( s ′ | s, a j ) . If we fix the value of s and a -j , then w j ( s, a j ) has to be the same for different values of a j due to the fact ∑ j ∈N C w j ( s, a j ) = 1 . Also note that by definition, w j ( s, a j ) does not depend on the choice of this fixed a -j . Therefore, such a w j ( s, a j ) can be written as w j ( s ) , where w j ( s ) = ∑ s ′ ∈S F j ( s ′ | s, a j ) for all a j ∈ A j . We can thus rewrite ( F j ) j ∈N C using some actual probability distributions ( P j ) j ∈N C , such that if w j ( s ) = 0 , then we rewrite F j as F j ( s ′ | s, a j ) = w j ( s ) F j ( s ′ | s,a j ) w j ( s ) = w j ( s ) P j ( s ′ | s, a j ) , and if w j ( s ) = 0 , we rewrite F j as F j ( s ′ | s, a j ) = w j ( s ) P j ( s ′ | s, a j ) for an arbitrary probability distribution P j ( · | s, a j ) . Notice that ∑ s ′ ∈S P j ( s ′ | s, a j ) = 1 for any j ∈ N C . Then, the decomposable transition dynamics can be represented as P ( · | s, a ) = ∑ j ∈N C w j ( s ) P j ( · | s, a j ) , i.e., an ensemble of the transition dynamics that is only controlled by single controllers. The model's transition dynamics thus act according to the following two steps: (1) sampling the controller according to the distribution Multinomial ( ( w i ( s )) i ∈N C ) , and (2) transitioning the state following the sampled controller's dynamics. Such a model has also been investigated under the name of transition dynamics with additive structures in [33]. Note that our model is more general and thus covers the single-controller MG setting [34], where there is only one agent controlling the transition dynamics at all states. It also covers the setting of turn-based MGs [34], where in each round, depending on the current state s , the transition dynamics is by turns affected by only one of the agents. This can be captured by the proper choice of ( w i ( s )) i ∈N C ) that takes value 1 only for one agent at each state s (while takes value 0 for all other non-controller agents at each state s ). Additionally, the second case where P ( · | s, a ) = F o ( s ′ | s ) corresponds to the one with no ensemble of controller agents.

̸

Proposition 2 (Decomposition of ( Q V i ) i ∈N ) . For an infinite-horizon γ -discounted NMG with G = ( N , E Q ) such that N C = ∅ , if we know that P ( s ′ | s, a ) = ∑ i ∈N C F i ( s ′ | s, a i ) , and

1

2

3

5

4

1

2

3

5

4

1

2

3

5

4

E

r

E

N

P

,

connected

E

Q

∪

̸

r i ( s, a ) = ∑ j ∈E Q,i r i,j ( s, a i , a j ) for some { F i } i ∈N C and { r i,j } ( i,j ) ∈E Q , then the Q V i,j given in Definition 1 can be represented as

Q V i,j ( s, a i , a j ) = r i,j ( s, a i , a j ) + ∑ s ′ ∈S γ ( 1( j ∈ N C ) F j ( s ′ | s, a j ) + 1( i ∈ N C ) λ i,j ( s ) F i ( s ′ | s, a i ) ) V ( s ′ )

for any non-negative ( λ i,j ( s )) ( i,j ) ∈E Q such that ∑ j ∈E Q,i λ i,j ( s ) = 1 , for all i ∈ N and s ∈ S . We call it the canonical decomposition of { Q V i } i ∈N when Q V i,j can be represented as above with λ i,j ( s ) = 1 / |E Q,i | for j ∈ E Q,i . The case with N C = ∅ is deferred to Appendix B.

We introduce this canonical decomposition since the representation of Q V i,j is in general not unique, and we may use this canonical form to simplify the algorithm design later.

3.2 Examples of multi-player (zero-sum) NMGs

We now provide several examples of (multi-player) MGs with networked separable interactions here and in Section B.5.

Example 1 (Markov fashion games). Fashion games are an intriguing class of games [25, 26, 27] that plays a vital role not only in Economics theory but also in practice. A fashion game is a networked extension of the Matching Pennies game, in which each player has the action space A i = {-1 , +1 } , which means light and dark color fashions, respectively, for example. There are two types of players: conformists ( c ), who prefer to conform to their neighbors' fashion (action), and rebels ( r ), who prefer to oppose their neighbors' fashion (action). Such interactions with the neighbors are exactly captured by polymatrix games. We denote the interaction network between players as G = ( N , E ) .

̸

̸

Such a game naturally involves the following state transition dynamics: we introduce the state s ∈ S = Z by setting s 0 = 0 and s t +1 ∼ s t + Unif (( a t,c ) c ∈C ) , which indicates the fashion trend where C ⊆ N is the set of influencers. The fashion trend favors either light or dark colors if s ≥ 0 or s < 0 , respectively. We can think of dynamics as the impact of the influencers on the fashion trend at time t . For each ( s, a ) , the reward function for player i , depending on whether she is a conformist or a rebel, are defined as r c ,i ( s, a ) = ∑ j ∈E i r c ,i,j ( s, a i , a j ) = ∑ j ∈E i ( 1 |E i | 1( sgn ( s ) = a i ) +1( a i = a j )) and r r ,i ( s, a ) = ∑ j ∈E i r r ,i,j ( s, a i , a j ) = ∑ j ∈E i ( 1 |E i | 1( sgn ( s ) = a i ) + 1( a i = a j )) , respectively. This is an NMG as defined in Definition 1. Moreover, if the conformists and rebels constitute a bipartite graph, i.e., the neighbors of a conformist are all rebels and vice versa, it becomes a multiplayer constant-sum MG with networked separable interactions, and we can subtract the constant offset to make it a zero-sum NMG.

3.3 Relationship between CCE and NE in zero-sum NMGs

A well-known property for zero-sum NGs is that marginalizing a CCE leads to a NE, which makes it computationally tractable to find the NE [23, 24]. We now provide below a counterpart in the Markov game setting, and provide a more detailed statement of the result in Appendix B.

Proposition 3. Given an ϵ -approximate Markov CCE of an infinite-horizon γ -discounted zero-sum NMG, marginalizing it at each state results in an ( n +1) (1 -γ ) ϵ -approximate Markov NE of the zero-sum NMG. The same argument also holds for the finite-horizon episodic setting with (1 -γ ) -1 being replaced by H .

This result holds for both stationary and non-stationary ϵ -approximate Markov CCEs. We defer the proof of Proposition 3 to Appendix B. This proposition suggests that if we can have some algorithms to find an approximate Markov CCE for a zero-sum NMG, we can obtain an approximate Markov NE by marginalizing the approximate CCE at each state. We also emphasize that the Markovian property of the equilibrium policies is important for the result to hold. As a result, the learning algorithm in [30], which learns an approximate Markov non-stationary CCE with polynomial time and samples, may thus be used to find an approximate Markov non-stationary NE in zero-sum NMGs. However, as the focus of [30] was the more challenging setting of model-free learning , the complexity therein has a high dependence on the problem parameters, and the algorithm can only find non-perfect equilibria. When it comes to (perfect) equilibrium computation, one may exploit the multi-player zero-sum structure of zero-sum NMGs, and develop more natural and faster algorithms to find a Markov non-stationary NE. Moreover, when it comes to stationary equilibrium computation, even Markov CCE is not tractable in general-sum cases [30, 29]. Hereafter, we will focus on approaching zero-sum NMGs from these perspectives.

4 Hardness for Stationary CCE Computation

Given the results in Section 3.3, it seems tempting and sufficient to compute the Markov CCE of the zero-sum NMG. Indeed, computing CCE (and thus NE) in zero-sum polymatrix games is known to be tractable [23, 24]. It is thus natural to ask: Is finding Markov CCE computationally tractable? Next, we answer the question with different answers for finding stationary CCE (in infinite-horizon γ -discounted setting) and non-stationary CCE (in finite-horizon episodic setting), respectively.

For two-player infinite-horizon γ -discounted zero-sum MGs, significant progress in computing/learning the (Markov) stationary NE has been made recently [35, 36, 37, 38, 39, 40, 41, 42]. On the other hand, for multi-player general-sum MGs, recent results in [30, 29] showed that computing (Markov) stationary CCE can be PPAD -hard and thus believed to be computationally intractable. We next show that this hardness persists in most non-degenerate cases even if one enforces the zero-sum and networked interaction structures in the multi-player case. We state the formal result as follows, whose detailed proof is available in Section C.

Theorem 1. There is a constant ϵ > 0 for which computing an ϵ -approximate Markov perfect stationary CCE in infinite-horizon 1 2 -discounted zero-sum NMGs, whose underlying network structure contains either a triangle or a 3-path subgraph, is PPAD -hard. Moreover, given the PCP for PPAD conjecture [43], there is a constant ϵ > 0 such that computing even an ϵ -approximate Markov non-perfect stationary CCE in such zero-sum NMGs is PPAD -hard.

Proof Sketch of Theorem 1. Due to space constraints, we focus on the case with three players, and the underlying network structure has a triangle subgraph. Proof for the 3 -path case is similar and can be found in Section C. We will show that for any general-sum two-player turn-based MG (A) , the problem of computing its Markov stationary CCE, which is inherently a PPAD -hard problem [30], can be reduced to computing the Markov stationary CCE of a three-player zero-sum MG with a triangle structure networked separable interactions (B) . Consider an MG (A) with two players, players 1 and 2, and reward functions r 1 ( s, a 1 , a 2 ) and r 2 ( s, a 2 , a 1 ) , where a i is the action of the i -th player and r i is the reward function of the i -th player. The transition dynamics is given by P ( s ′ | s, a 1 , a 2 ) . In even rounds, player 2's action space is limited to Noop2 , and in odd rounds, player 1's action space is limited to Noop1 , where Noop is an abbreviation of 'no-operation', i.e., the player does not affect the transition dynamics or the reward in that round. We denote player 1's action space in even rounds as A 1 , even and player 2's action space in odd rounds as A 2 , odd, respectively.

̸

Now, we construct a three-player zero-sum NMG. with a triangle network structure. We set the reward function as ˜ r i ( s, a ) = ∑ j = i ˜ r i,j ( s, a i , a j ) and ˜ r i,j ( s, a i , a j ) = -˜ r j,i ( s, a j , a i ) . The reward functions are designed so that ˜ r i,j = -˜ r j,i for all i, j , ˜ r 1 , 2 + ˜ r 1 , 3 = r 1 , and ˜ r 2 , 1 + ˜ r 2 , 3 = r 2 , where r 1 , r 2 are the reward functions in game (A) , by introducing a dummy player, player 3. In even rounds, player 2's action space is limited to Noop2 , and in odd rounds, player 1's action space is limited to Noop1 . Player 3's action space is always limited to Noop3 in all rounds. The transition dynamics is defined as ˜ P ( s ′ | s, a 1 , a 2 , a 3 ) = P ( s ′ | s, a 1 , a 2 ) , since a 3 is always chosen from Noop3 . In other words, player 3's action does not affect the rewards of the other two players, nor the transition dynamics, and players 1 and 2 will receive the reward as in the two-player turn-based MG. Also, note that due to the turn-based structure of the game (A) , the transition dynamics satisfy the decomposable condition in our Proposition 1, and it is thus a zero-sum NMG. In fact, turn-based dynamics can be represented as an ensemble of single controller dynamics, as we have discussed in Section 3.

1

2

3

1

2

3

4

r

1

,

2

r

2

,

1

˜

r

1

,

3

˜

r

3

,

1

˜

r

3

,

2

˜

r

2

,

3

r

1

,

2

r

2

,

1

˜

r

1

,

3

˜

r

3

,

1

˜

r

2

,

4

˜

r

4

,

2

1

center-player

2

3

5

4

˜

˜

˜

˜

Note that the new game (B) is still a turn-based game, and thus the Markov stationary CCE is the same as the Markov stationary NE. Also, note that by construction, the equilibrium policies of players 1 and 2 at the Markov stationary CCE of the game (B) constitute a Markov stationary CCE of the game (A) . If the underlying network is more general than a triangle, but contains a triangle subgraph, we can specify the reward and transition dynamics of these three players as above, and specify all other players to be dummy players, whose reward functions are all zero, and do not affect the reward functions of these three players, nor the transition dynamics. This completes the proof.

Figure 2 briefly explains how we may reduce the equilibrium computation problem of (A) to that of (B) . In fact, a connected graph that does not contain a subgraph of a triangle or a 3-path has to be a star-shaped network (Proposition 7), which is proved in Section C. Hence, by Theorem 1, we know that in the infinite-horizon discounted setting, finding Markov stationary NE/CE/CCE is a computationally hard problem unless the underlying network is star-shaped. This may also imply that learning Markov stationary NE in zero-sum NMGs, e.g., using natural dynamics like fictitious play to reach the NE, can be challenging, unless in the star-shaped case. In turn, one may hope fictitious-play dynamics to converge for star-shaped zero-sum NMGs. We instantiate this idea next in Section 5. Furthermore, in light of Theorem 1, we will shift gear to computing Markov non-stationary NE by utilizing the structure of networked separable interactions, as to be detailed in Section 6.

5 Fictitious-Play Property

In this section, we study the fictitious-play property of multi-player zero-sum games with networked separable interactions, for both the matrix and Markov game settings. Following the convention in [8], we refer to the games in which fictitious-play dynamics converge to the NE as the games that have the fictitious-play property . We defer the matrix game case results to Section D, where we have also established convergence of the well-known variant of FP, smooth FP [7], in zero-sum NGs.

Echoing the computational intractability of computing CCE of zero-sum NMG unless the underlying network structure is star-shaped in the infinite-horizon discounted setting (c.f. Theorem 1), we now consider the FP property in such games. Note that by Proposition 1, E Q is a star-shape if and only if the reward structure is a star shape and N C = { 1 } , where player 1 is the center of the star (Figure 2), or there are only two players in zero-sum NMG. There is already existing literature for the latter case [15, 16], so we focus on the former case, which is a single-controller case where player 1 controls the transition dynamics, i.e., P ( s ′ | s, a ) = P 1 ( s ′ | s, a 1 ) for some P 1 . We now introduce the fictitious-play dynamics for such zero-sum NMGs.

Each player i first initializes her beliefs of other players' policies as uniform distributions, and also initializes her belief of the Q -value estimates with arbitrary values. Then, at iteration k , player i takes the best-response action based on her belief of other players' policies ( ̂ π ( k ) -i ( s ( k ) )) , and their Q beliefs ̂ Q ( k ) i ( s ( k ) , a ) :

Then, player i implements the action a ( k ) i , observes other players' actions a ( k ) -i , and updates her beliefs as follows: for each player i ∈ N , she updates her belief of the opponents' policies as

for all s ∈ S , with stepsize α N ( s ) ≥ 0 where N ( s ) is the visitation count for the state s ; then if i = 1 , this player 1 updates the belief of Q 1 ,j for all j ∈ N / { 1 } and her own ̂ Q 1 ( s, a ) for all s ∈ S as

̸

which is based on the canonical decomposition given in Proposition 2, where ̂ V ( k ) 1 ( s ) = max a 1 ∈A 1 ̂ Q ( k ) 1 ( s, e a 1 , ̂ π ( k ) -1 ( s )) , and β N ( s ) ≥ 0 is the stepsize. The agent then updates ̂ Q ( k +1) 1 ( s, a ) = ∑ j ∈N / { 1 } ̂ Q ( k +1) 1 ,j ( s, a 1 , a j ) , for all s ∈ S , a ∈ A . Otherwise, if i = 1 , then player i updates the belief of her ̂ Q i, 1 ( s, a ) for all s ∈ S , a ∈ A as

̸

where ̂ V ( k ) i ( s ) = max a i ∈A i ̂ Q ( k ) i ( s, e a i , ̂ π ( k ) -i ( s )) , and we let ̂ Q ( k +1) i ( s, a ) = ̂ Q ( k +1) i, 1 ( s, a i , a 1 ) for these i = 1 . The overall dynamics are summarized in Algorithm 4, which resembles the FP dynamics for two-player zero-sum [16] and identical-interest [19] MGs. Now we are ready to present the convergence guarantees.

Assumption 1. The sequences of step sizes { α k ∈ (0 , 1] } k ≥ 0 and { β k ∈ (0 , 1] } k ≥ 0 satisfy the following conditions: (1) ∑ ∞ k =0 α k = ∞ , ∑ ∞ k =0 β k = ∞ , and lim k →∞ α k = lim k →∞ β k = 0 ; (2) lim k →∞ β k α k = 0 , indicating that the rate at which the beliefs about Q -functions are updated is slower than the rate at which the beliefs about policies are updated.

Theorem 2. Suppose Assumption 1 holds and Algorithm 4 visits every state infinitely often with probability 1 . Then, for a star-shaped multi-player zero-sum NMG, the belief ( ̂ π ( k ) ) k ≥ 0 converges to a Markov stationary NE and the belief ( ̂ Q ( k ) ) k ≥ 0 converges to the corresponding NE value of the zero-sum NMG with probability 1, as k →∞ .

We defer the proof to Section D.2 due to space constraints. Note that to illustrate the idea, we only present the result for the model-based case, i.e., when the transition dynamics P is known. With this result, it is direct to extend to the model-free and learning case, where P is not known [16, 19, 17], still using the tool of stochastic approximation [44]. See Section D for more details.

Remark 4 (Challenges for analyzing general cases) . One might ask why we had to focus on a star-shaped structure. First, for general networked structures, even in the matrix-game case, it is known that the NE values of a zero-sum NG may not be unique [24]. Hence, suppose one performs Nash-value iteration , i.e., solving for the NE of the stage game and conducting backward induction, this value iteration process does not converge in general as the number of backward steps increases, since the solution at each stage is not even unique, and there may not exist a unique fixed point. This is in stark contrast to the max and maxmin operators in the value iteration updates for single-player and two-player zero-sum cases, respectively. By exploiting a star-shaped structure, we managed to reformulate a minimax optimization problem when solving each stage game, which makes the corresponding value iteration operator contracting , and thus iterating it infinitely converges to the unique fixed point. Second, suppose there exists some other network structure (other than star-shaped ones) that also leads to a contracting value iteration operator, then for a fixed constant γ , the fixed point (which corresponds to the Markov stationary CCE/NE of the zero-sum NMG) becomes unique and can be computed efficiently, which contradicts our hardness result in Theorem 1. Indeed, it was the exclusion of a star-shaped structure in Theorem 1 that inspired us to consider this structure in proving the convergence of FP dynamics. That being said, we note that having a contracting value iteration operator is only a sufficient condition for the FP dynamics to converge. It would be interesting to explore other structures that enjoy the FP property for reasons beyond this contraction property. We leave this as an immediate future work.

Next, we present another positive result in light of the hardness in Theorem 1, regarding the computation of non-stationary equilibria in multi-player zero-sum NMGs.

6 Non-Stationary NE Computation

We now focus on computing an (approximate) Markov non-stationary equilibrium in zero-sum NMGs. In particular, we show that when relaxing the stationarity requirement, not only CCE, but NE, can be computed efficiently. Before introducing our algorithm, we first recall the folklore result that approximating Markov non-stationary NE in infinite-horizon discounted settings can be achieved by finding approximate Markov NE in finite-horizon settings, with a large enough horizon length (c.f. Proposition 10). Hence, we will focus on the finite-horizon setting from now on.

Before delving into the details of our algorithm, we introduce the notation Q h,i ( s ) and Q h ( s ) for h ∈ [ H ] , i ∈ N , s ∈ S as follows:

̸

Here, Q h,i,j represents an estimate of the equilibrium value function with canonical decomposition (Proposition 2). Hereafter, we similarly define the notation of Q π h,i and Q π h . Our algorithm is based on value iteration, and iterates three main steps from h = H to 1 as follows: (1) Q -value computation: compute Q h,i,j , which estimates the equilibrium Q -value with a canonical decomposition form; in particular, when N C = ∅ , Q h,i,j is updated for all s ∈ S , ( i, j ) ∈ E Q , a i ∈ A i , and a j ∈ A j :

(2) Policy update: update π h ( s ) with an NE-ORACLE : finding (approximate)-NE of some zero-sum NG ( G , A , ( Q h,i,j ( s )) ( i,j ) ∈E Q ) for all s ∈ S , and (3) Value function update: compute V h,i , which estimates the equilibrium value function as follows for all s ∈ S , i ∈ N : V h,i ( s ) = π ⊺ h,i ( s ) Q h,i ( s ) π h ( s ) . The overall procedure is summarized in Algorithm 6.

NE-ORACLE and iteration complexity. The NE-ORACLE in Algorithm 6 can be instantiated by several different algorithms that can find an NE in a zero-sum NG. Depending on the algorithms, the convergence guarantees can be either in terms of average-iterate, best-iterate, or last-iterate. Note that for algorithms with average-iterate convergence, one may additionally need to marginalize the output joint policy, i.e., the approximate CCE, and combine them as a product policy that is an approximate NE (Proposition 6). For those with best-/last-iterate convergence, by contrast, the best-/last-iterate is already in product form, and one can directly output it as an approximate NE. Moreover, last-iterate convergence is known to be a more favorable metric than the average-iterate one in learning in games [45, 46, 47, 48, 49], which is able to characterize the day-to-day behavior of the iterates and implies the stability of the update rule. Hence, one may prefer to have last-iterate convergence for solving zero-sum N(M)Gs. To this end, two algorithmic ideas may be useful: adding regularization to the payoff matrix [39, 42, 50, 51, 52], and/or using the idea of optimism [47, 36, 53]. Recent results [54, 51] have instantiated the ideas of optimism-only and optimism + regularization , respectively, for best-/last-iterate convergence in zero-sum polymatrix games. We additionally established results for the idea of regularization-only in obtaining last-iterate convergence in these games. Specifically, we propose to study the vanilla Multiplicative Weight Update (MWU) algorithm [55] in the regularized zero-sum NG, as tabulated in Algorithm 9. We have also introduced a variant with diminishing regularization, and summarize the update rule in Algorithm 10.

Given the results above, aggregating ϵ -approximate NE for the zero-sum NGs ( G , A , ( Q h,i,j ( s )) ( i,j ) ∈E Q ) for all h ∈ [ H ] , i ∈ N , s ∈ S provides an Hϵ -approximate NE for the corresponding zero-sum NMG. We have the following formal result.

Proposition 4. Suppose that for all h ∈ [ H ] , i ∈ N , s ∈ S , NE-ORACLE ( G , A , ( Q h,i,j ( s )) ( i,j ) ∈E Q ) provides an ϵ h,s -approximate NE for the zerosum NG ( G , A , ( Q h,i,j ( s )) ( i,j ) ∈E Q ) in Algorithm 6. Then, the output policy π in Algorithm 6 is an ( ∑ h ∈ [ H ] max s ∈S ϵ h,s ) -approximate NE for the corresponding zero-sum NMG ( G = ( N , E Q ) , S , A , H, ( P h ) h ∈ [ H ] , ( r h,i,j ( s )) ( i,j ) ∈E Q ,s ∈S ) .

The proof of Proposition 4 is deferred to Section F. In light of Proposition 4 and Table 1, we obtain Table 2, which summarizes the iteration complexities required to find an ϵ -NE for zero-sum NMGs, with different NE-ORACLE subroutines. Note that the iteration complexities are all polynomial in H,n, |S| , and inherit the order of dependencies on ϵ from Table 1 for the matrix-game case. In particular, Algorithm 6 with the OMWU in [51] yields the fast rate of ˜ O (1 /ϵ ) for the last iterate.

References

John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior. Princeton University Press , 1947.

John F Nash Jr. Equilibrium points in n-person games. Proceedings of the National Academy of sciences , 36(1):48-49, 1950.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge University press, 2006.

Drew Fudenberg and David K Levine. The theory of learning in games , volume 2. MIT Press, 1998.

George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation , 13(1):374, 1951.

Julia Robinson. An iterative method of solving a game. Annals of mathematics , pages 296-301, 1951.

Drew Fudenberg and David M Kreps. Learning mixed equilibria. Games and Economic Behavior , 5:320-367, 1993.

Dov Monderer and Lloyd S Shapley. Fictitious play property for games with identical interests. Journal of economic theory , 68(1):258-265, 1996.

Koichi Miyasawa. On the convergence of the learning process in a 2 x 2 non-zero-sum two-person game. Princeton University Press , 1961.

Ulrich Berger. Fictitious play in 2 x n games. Journal of Economic Theory , 120(2):139-154, 2005.

Aner Sela. Fictitious play in 'one-against-all' multi-player games. Economic Theory , 14(3):635-651, 1999.

Christian Ewerhart and Kremena Valkanova. Fictitious play in networks. Games and Economic Behavior , 123:182-206, 2020.

Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences , 39(10):1095-1100, 1953.

David S Leslie, Steven Perkins, and Zibo Xu. Best-response dynamics in zero-sum stochastic games. Journal of Economic Theory , 189:105095, 2020.

Muhammed Sayin, Kaiqing Zhang, David Leslie, Tamer Basar, and Asuman Ozdaglar. Decentralized Q-learning in zero-sum Markov games. NeurIPS , 2021.

Muhammed O Sayin, Francesca Parise, and Asuman Ozdaglar. Fictitious play in zero-sum stochastic games. SIAM Journal on Control and Optimization , 60(4):2095-2114, 2022.

Lucas Baudin and Rida Laraki. Fictitious play and best-response dynamics in identical interest and zero-sum stochastic games. ICML , 2022.

Zaiwei Chen, Kaiqing Zhang, Eric Mazumdar, Asuman Ozdaglar, and Adam Wierman. A finite-sample analysis of payoff-based independent learning in zero-sum stochastic games. NeurIPS , 2023.

Muhammed O Sayin, Kaiqing Zhang, and Asuman Ozdaglar. Fictitious play in markov games with single controller. EC , 2022.

Constantinos Daskalakis, Paul W Goldberg, and Christos H Papadimitriou. The complexity of computing a nash equilibrium. SIAM Journal on Computing , 39(1):195-259, 2009.

Xi Chen, Xiaotie Deng, and Shang-Hua Teng. Settling the complexity of computing two-player nash equilibria. Journal of the ACM (JACM) , 56(3):1-57, 2009.

Christos H Papadimitriou and Tim Roughgarden. Computing correlated equilibria in multiplayer games. Journal of the ACM (JACM) , 55(3):1-29, 2008.

Yang Cai and Constantinos Daskalakis. On minmax theorems for multiplayer games. SODA , 2011.

Yang Cai, Ozan Candogan, Constantinos Daskalakis, and Christos Papadimitriou. Zerosum polymatrix games: A generalization of minmax. Mathematics of Operations Research , 41(2):648-655, 2016.

Zhigang Cao, Haoyu Gao, Xinglong Qu, Mingmin Yang, and Xiaoguang Yang. Fashion, cooperation, and social interactions. PLoS One , 8(1):e49441, 2013.

Zhigang Cao and Xiaoguang Yang. The fashion game: Network extension of matching pennies. Theoretical Computer Science , 540:169-181, 2014.

Boyu Zhang, Zhigang Cao, Cheng-Zhong Qin, and Xiaoguang Yang. Fashion and homophily. Operations Research , 66(6):1486-1497, 2018.

L M Bergman and I N Fokin. On separable non-cooperative zero-sum games. Optimization , 44(1):69-84, 1998.

Yujia Jin, Vidya Muthukumar, and Aaron Sidford. The complexity of infinite-horizon generalsum stochastic games. ITCS , 2022.

Constantinos Daskalakis, Noah Golowich, and Kaiqing Zhang. The complexity of markov equilibrium in stochastic games. COLT , 2023.

L M Bergman and I N Fokin. Methods of determining equilibrium situations in zero-sum polymatrix games. Optimizatsia , 40(57):70-82, 1987.

Christos H Papadimitriou. On the complexity of the parity argument and other inefficient proofs of existence. Journal of Computer and System Sciences , 48(3):498-532, 1994.

János Flesch, Frank Thuijsman, and Okko Jan Vrieze. Stochastic games with additive transitions. European Journal of Operational Research , 179(2):483-497, 2007.

Jerzy Filar and Koos Vrieze. Competitive Markov decision processes . Springer Science & Business Media, 2012.

Constantinos Daskalakis, Dylan J Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. NeurIPS , 2020.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive Markov games. COLT , 2021.

Yulai Zhao, Yuandong Tian, Jason D Lee, and Simon S Du. Provably efficient policy gradient methods for two-player zero-sum markov games. AISTATS , 2022.

Ziyi Chen, Shaocong Ma, and Yi Zhou. Sample efficient stochastic policy extragradient algorithm for zero-sum markov game. ICLR , 2021.

Shicong Cen, Yuting Wei, and Yuejie Chi. Fast policy extragradient methods for competitive games with entropy regularization. NeurIPS , 2021.

Ahmet Alacaoglu, Luca Viano, Niao He, and Volkan Cevher. A natural actor-critic framework for zero-sum markov games. ICML , 2022.

Sihan Zeng, Thinh T Doan, and Justin Romberg. Regularized gradient descent ascent for two-player zero-sum markov games. NeurIPS , 2022.

Shicong Cen, Yuejie Chi, Simon S Du, and Lin Xiao. Faster last-iterate convergence of policy optimization in zero-sum markov games. ICLR , 2023.

Yakov Babichenko, Christos Papadimitriou, and Aviad Rubinstein. Can almost everybody be almost happy? pcp for ppad and the inapproximability of nash. arXiv preprint arXiv:1504.02411 , 2015.

Michel Benaïm, Josef Hofbauer, and Sylvain Sorin. Stochastic approximations and differential inclusions. SIAM Journal on Control and Optimization , 44(1):328-348, 2005.

Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. SODA , 2018.

Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training GANs with optimism. ICLR , 2018.

Constantinos Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. ITCS , 2019.

James P Bailey and Georgios Piliouras. Multiplicative weights update in zero-sum games. EC , 2018.

Panayotis Mertikopoulos, Bruno Lecouat, Houssam Zenati, Chuan-Sheng Foo, Vijay Chandrasekhar, and Georgios Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. ICLR , 2019.

Shicong Cen, Fan Chen, and Yuejie Chi. Independent natural policy gradient methods for potential games: Finite-time global convergence with entropy regularization. CDC , 2022.

Ruicheng Ao, Shicong Cen, and Yuejie Chi. Asynchronous gradient play in zero-sum multiagent games. ICLR , 2023.

Sarath Pattathil, Kaiqing Zhang, and Asuman Ozdaglar. Symmetric (optimistic) natural policy gradient for multi-agent learning with parameter convergence. AISTATS , 2023.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. ICLR , 2022.

Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm. On lastiterate convergence beyond zero-sum games. ICML , 2022.

Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: A meta-algorithm and applications. Theory of Computing , 8(1):121-164, 2012.

Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) , 38(2):156-172, 2008.

Kaiqing Zhang, Zhuoran Yang, and Tamer Ba¸ sar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control , pages 321-384, 2021.

Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. Machine learning proceedings 1994 , pages 157-163, 1994.

Michael L Littman et al. Friend-or-foe q-learning in general-sum games. ICML , 2001.

Michael L Littman. Value-function reinforcement learning in markov games. Cognitive systems research , 2(1):55-66, 2001.

Junling Hu and Michael P Wellman. Nash q-learning for general-sum stochastic games. JMLR , 2003.

Yu Bai and Chi Jin. Provable self-play algorithms for competitive reinforcement learning. ICML , 2020.

Aaron Sidford, Mengdi Wang, Lin Yang, and Yinyu Ye. Solving discounted stochastic two-player games with near-optimal time and sample complexity. AISTATS , 2020.

Qiaomin Xie, Yudong Chen, Zhaoran Wang, and Zhuoran Yang. Learning zero-sum simultaneous-move markov games using function approximation and correlated equilibrium. COLT , 2020.

Yu Bai, Chi Jin, and Tiancheng Yu. Near-optimal reinforcement learning with self-play. NeurIPS , 2020.

Qinghua Liu, Tiancheng Yu, Yu Bai, and Chi Jin. A sharp analysis of model-based reinforcement learning with self-play. ICML , 2021.

Kaiqing Zhang, Sham Kakade, Tamer Basar, and Lin Yang. Model-based multi-agent rl in zero-sum markov games with near-optimal sample complexity. NeurIPS , 2020.

Gen Li, Yuejie Chi, Yuting Wei, and Yuxin Chen. Minimax-optimal multi-agent rl in Markov games with a generative model. NeurIPS , 2022.

Jayakumar Subramanian, Amit Sinha, and Aditya Mahajan. Robustness and sample complexity of model-based MARL for general-sum Markov games. Dynamic Games and Applications , pages 1-33, 2023.

Ziang Song, Song Mei, and Yu Bai. When can we learn general-sum markov games with a large number of players sample-efficiently? ICLR , 2021.

Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning-a simple, efficient, decentralized algorithm for multiagent RL. ICLR 2022 workshop 'Gamification and Multiagent Solutions' , 2022.

Weichao Mao, Lin Yang, Kaiqing Zhang, and Tamer Basar. On improving model-free algorithms for decentralized multi-agent reinforcement learning. ICML , 2022.

Weichao Mao and Tamer Ba¸ sar. Provably efficient reinforcement learning in decentralized general-sum Markov games. Dynamic Games and Applications , pages 1-22, 2022.

Qiwen Cui, Kaiqing Zhang, and Simon S Du. Breaking the curse of multiagents in a large state space: RL in Markov games with independent linear function approximation. COLT , 2023.

Yuanhao Wang, Qinghua Liu, Yu Bai, and Chi Jin. Breaking the curse of multiagency: Provably efficient decentralized multi-agent RL with function approximation. COLT , 2023.

Liad Erez, Tal Lancewicki, Uri Sherman, Tomer Koren, and Yishay Mansour. Regret minimization and convergence to equilibria in general-sum markov games. ICML , 2023.

Aviad Rubinstein. Settling the complexity of computing approximate two-player Nash equilibria. ACM SIGecom Exchanges , 15(2):45-49, 2017.

Constantinos Daskalakis. Non-concave games: A challenge for game theory's next 100 years. 2022.

Matthew O Jackson and Yves Zenou. Games on networks. In Handbook of game theory with economic applications , volume 4, pages 95-163. Elsevier, 2015.

Michael Kearns, Michael L Littman, and Satinder Singh. Graphical models for game theory. UAI , 2001.

Sham Kakade, Michael Kearns, John Langford, and Luis Ortiz. Correlated equilibria in graphical games. EC , 2003.

Constantinos Daskalakis and Christos H Papadimitriou. On a network generalization of the minmax theorem. In International Colloquium on Automata, Languages, and Programming , pages 423-434. Springer, 2009.

Stefanos Leonardos, Georgios Piliouras, and Kelly Spendlove. Exploration-exploitation in multi-agent competition: convergence with bounded rationality. NeurIPS , 2021.

Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Ba¸ sar. Fully decentralized multi-agent reinforcement learning with networked agents. ICML , 2018.

Kaiqing Zhang, Zhuoran Yang, and Tamer Basar. Networked multi-agent reinforcement learning in continuous spaces. CDC , 2018.

Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning for multiagent networked systems. Operations Research , 2022.

Xin Liu, Honghao Wei, and Lei Ying. Scalable and sample efficient distributed policy gradient algorithms in multi-agent networked systems. arXiv preprint arXiv:2212.06357 , 2022.

Yizhou Zhang, Guannan Qu, Pan Xu, Yiheng Lin, Zaiwei Chen, and Adam Wierman. Global convergence of localized policy iteration in networked multi-agent reinforcement learning. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 7(1):1-51, 2023.

Zhaoyi Zhou, Zaiwei Chen, Yiheng Lin, and Adam Wierman. Convergence rates for localized actor-critic in networked markov potential games. UAI , 2023.

Georgios Piliouras, Lillian Ratliff, Ryann Sim, and Stratis Skoulakis. Fast convergence of optimistic gradient ascent in network zero-sum extensive form games. SAGT , 2022.

Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science , 3(3):241-268, 1991.

Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. 2010.

Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularized Markov decision processes. NeurIPS , 2017.

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. ICML , 2017.

Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. ICML , 2020.

Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 70(4):25632578, 2022.

Mingyang Liu, Asuman E. Ozdaglar, Tiancheng Yu, and Kaiqing Zhang. The power of regularization in solving extensive-form games. ICLR , 2023.

Samuel Sokota, Ryan D'Orazio, J Zico Kolter, Nicolas Loizou, Marc Lanctot, Ioannis Mitliagkas, Noam Brown, and Christian Kroer. A unified approach to reinforcement learning, quantal response equilibria, and two-player zero-sum games. ICLR , 2023.

Lloyd Shapley. Some topics in two-person games. Advances in game theory , 52:1-29, 1964.

Muhammed O Sayin. On the global convergence of stochastic fictitious play in stochastic games with turn-based controllers. CDC , 2022.

Asuman Ozdaglar, Muhammed O Sayin, and Kaiqing Zhang. Independent learning in stochastic games. International Congress of Mathematicians , 2022.

Fivos Kalogiannis and Ioannis Panageas. Zero-sum polymatrix Markov games: Equilibrium collapse and efficient computation of Nash equilibria. NeurIPS , 2023.

Jens Grossklags, Nicolas Christin, and John Chuang. Secure or insure? a game-theoretic analysis of information security games. WWW , 2008.

Robert Wilson. Game-theoretic analysis of trading processes. Technical report, Stanford University, 1985.

Robert Wilson. Game theoretic analysis of trading. In Advances in Economic Theory: Fifth World Congress , number 12, page 33. CUP Archive, 1989.

Dayong Zhang, Lei Lei, Qiang Ji, and Ali M Kutan. Economic policy uncertainty in the us and china and their impact on the global markets. Economic Modelling , 79:47-56, 2019.

Michael Beckley. The power of nations: Measuring what matters. International Security , 43(2):7-44, 2018.

Charles Freedman, Michael Kumhof, Douglas Laxton, Dirk Muir, and Susanna Mursula. Global effects of fiscal stimulus during the crisis. Journal of monetary economics , 57(5):506526, 2010.

Klaus Armingeon. The politics of fiscal responses to the crisis of 2008-2009. Governance , 25(4):543-565, 2012.

Fábio Augusto Reis Gomes, Sergio Naruhiko Sakurai, and Gian Paulo Soave. Government spending multipliers in good times and bad times: The case of emerging markets. Macroeconomic Dynamics , 26(3):726-768, 2022.

Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games. Games and economic behavior , 10(1):6-38, 1995.

Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for extensive form games. Experimental economics , 1:9-41, 1998.

Panayotis Mertikopoulos and William H Sandholm. Learning in games via reinforcement and regularization. Mathematics of Operations Research , 41(4):1297-1324, 2016.

Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. NeurIPS , 2021.

Ioannis Anagnostides, Gabriele Farina, Christian Kroer, Chung-Wei Lee, Haipeng Luo, and Tuomas Sandholm. Uncoupled learning dynamics with o ( \ log t ) swap regret in multiplayer games. NeurIPS , 2022.

Gabriele Farina, Ioannis Anagnostides, Haipeng Luo, Chung-Wei Lee, Christian Kroer, and Tuomas Sandholm. Near-optimal no-regret learning dynamics for general convex games. NeurIPS , 2022.

David Reeb and Michael M Wolf. Tight bound on relative entropy by entropy difference. IEEE Transactions on Information Theory , 61(3):1458-1473, 2015.

Haipeng Luo. Introduction to online optimization/learning (fall 2022), lecture note 1.

Yang Cai, Haipeng Luo, Chen-Yu Wei, and Weiqiang Zheng. Uncoupled and convergent learning in two-player zero-sum markov games. arXiv preprint arXiv:2303.02738 , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a subclass of Markov Games (MG) that have local interactions and a zero-sum structure for their Q value functions. This subclass generalizes zero-sum polymatrix games. It's shown that the marginal distribution of a Markov CCE for these games is a Markov  NE. It's then shown that computing a stationary Markov CCE/NE is PPAD-hard. It's then shown that a fictitious play type algorithm converges to a stationary Markov CCE for the special case where the local interactions are given by a star graph where the player at the center of the star controls the state transitions. It's then shown that another algorithm based on optimistic multiplicative weight updates converges to a non-stationary Markov\n\nSTRENGTHS:\nMarkov game is a challenging class of games with limited results. The paper makes some progress for this class of games. The analysis is rigorous and non-trivial. The technical aspects of the results are well-discussed.\n\nWEAKNESSES:\nThe motivation for this class of games is unclear and weak. The motivation for the results of this class of games is also vague. The paper is dry and technical with little insight. The presentation is notationally heavy and some of the statements are a bit sloppy. \n\nI'm not sure what meaning the proposed class of games has beyond its technical definition and the paper does very little to expose such meaning. The definition is done on the Q values, which is awkward and unintuitive, and then proposition 1 basically states that a better definition exists. In a polymatrix zero-sum game, the decomposition of the reward makes simple sense. To generalize them to MG, the paper here adds that the transition probabilities need to be decomposable - which doesn't seem nice or intuitive anymore beyond trivial special cases (one player controls the transitions, please see Q3 below). The paper presents one example (even though the title is \"Examples of M(Z)NMGs\") of such a game which is highly contrived, and even for this example, the strong bipartite graph structure condition is needed. This is very unconvincing regarding how useful this class of games is. \n\nThe results for stationary equilibrium are a bit disappointing. The hardness result would be more interesting if it was already agreed that this class of games is interesting - but otherwise, it just looks like a reason not to consider this class of games proposed here. A star structure with the central player controlling the transitions basically means that the N-1 different games are almost separate (up to the mutual state). No motivation is provided for wanting to compute a stationary equilibrium, so these disappointing results are uncalled for. If nothing meaningful can be said there, and there is no reason to believe this case is of special importance, I don't see why the stationary case is any interesting.  \n\nThe statements of the definition and proposition are mostly messy. They are written as one block of text even if they are split into several cases, and if some equations are more important than others. Therefore they are unnecessarily hard to digest. \n\nMore Comments: \n\nThe acronyms are very confusing because they never follow the first letters of the words. \n\nIn Definition 1, Multi-player MG is italic but \"with networked local interactions\" isn't, making it look like only the italic part is the issue here. \n\n\"every stage game with continuation payoffs being the payoff matrices qualifies as an\" - continuation payoffs? \n\nProposition 1 (b) - by always, you mean for every discount factor? this is a bit confusing as stated. \n\nThe definition of N_c can use some words to explain the meaning. This is also true for other definitions in the paper.\n\nThe caption of Figure 1 doesn't explain what's going on (not even that the red and blue are the nodes in the set N_c). Also, shouldn't the arrow by an equality? \n\n\"based on the PCP for PPAD conjecture\" - I guess you mean, if the conjecture holds, then... otherwise nothing can be based on a conjecture \n\nLine 165: if we do not know a priori about  - no \"about\" needed \n\nWhat's the point of the title of Subsection 5.1? that's the only case and only subsection \n\nLine 367 - double period \n\nThe statement of Theorem 2 is lacking and misleading. The main assumption is the star structure (and a single player driving the state transitions), and it's not stated here.\n\nQUESTIONS:\nQ1: Most of the paper analyzes the computation of stationary Markov CCE/NE.  Why is this interesting that stationary equilibrium is hard to compute if a non-stationary one isn't? Why do we want to compute the equilibrium in the first place? The research question proposed in Lines 36-37 is already answered by the non-stationary part of the paper. \n\nQ2: Why does \\mathcal{E}_r appear in a  MZNMG? (e.g., Figure 1 t comes from a polymatrix zero-sum game, but a MZNMG is not defined using a polymatrix zero-sum game. Why is another graph even needed? shouldn't the reward dependencies be dictated by \\mathcal{E}_Q? why would these two graphs be different?  \n\nQ3: Are there interesting cases beyond a single (state-dependent) player driving the state transitions? Given that it was done in [50], I think that a detailed comparison with [50] is needed.\n\nLIMITATIONS:\nThe technical limitations in this paper are discussed. There is no potential negative societal impact since the paper is computational.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of learning equilibria in multi-player zero-sum Markov games with networked local interactions (abbreviated as MZNMGs for short). In such games, the set of Markov CCEs collapses into that of Markov NEs. The paper shows that finding approximate Markov stationary CCEs is PPAD-hard in MZNMGs with infinite horizon, unless the network structure has a specific star shape. Then, the paper proves that in games with such a structure fictitious-play-based dynamics are guaranteed to converge to a Markov stationary NE. Moreover, the paper also shows that the negative PPAD-hardness result can be circumvented by considering Markov non-stationary CCEs, providing finite-iteration convergence guarantees for an algorithm based on optimistic multiplication weight updates.\n\nSTRENGTHS:\nORIGINALITY\n\nThe paper proposes a new class of Markov games and it shows that such a class of games enjoys some appealing properties in terms of convergence of learning dynamics.\n\nQUALITY\n\nThe claims seem sound (but I did not check the proofs in details).\n\nCLARITY\n\nThe paper is well written.\n\nSIGNIFICENCE\n\nThe problem of learning equilibria in Markov games has receiving considerable attention over the last years, and the results presented in the paper could be of interest to both researchers in algorithmic game theory and those in multi-agent RL.\n\nWEAKNESSES:\nORIGINALITY\n\nThe ideas behind the presented results and the techniques used to prove them seem adaptations and combinations of already known results and techniques. The authors should more specifically address the novelty of their results/techniques.\n\nCLARITY\n\nThe notation used in the paper is quite cumbersome, and in many spots it is easy to get lost in symbols. I think that these issues are inherent in the model studied in the paper, since it encompasses many elements, but I encourage the authors to try to lighten the notation as much as possible.\n\nMINOR COMMENTS\n\n- The experimental results seem an unneeded extra in this paper, I would consider removing them (actually, they are only discussed in the abstract and in the last section of the Appendix).\n- The discussion on related works in neglecting the recent works addressing the problem of learning correlated equilibria in sequential games. These are related to Markov games and should be adequately discussed.\n- It is not clear what you mena by single controller.\n- Line 107: It should be $\\Delta(\\mathcal{A})$ rather than $\\Delta(\\mathcal{A}_i)$.\n- Line 122 and onward, the notation $\\mu_i, \\pi_{-I}$ has not been introduced.\n- Line 162, it is not clear why $I \\neq j$.\n- Please give more textual intuition to Propositions 1 and 2.\n- Lines 260-261, discuss more why Markovianity is important, it is not clear otherwise.\n\nQUESTIONS:\n1) What is the relation between the results in this paper and the following recent work?\n\nFoster, Dylan J., Noah Golowich, and Sham M. Kakade. \"Hardness of Independent Learning and Sparse Equilibrium Computation in Markov Games.\" arXiv preprint arXiv:2303.12287 (2023).\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the problem of computing both in a centralized and in a decentralized way equilibria for a special class of stochastic games. These are games that may be found in multiple states. At each state players have a set of available actions, that once played will define both the reward that each player achieves and the next state in which the game will be found. It is known that with only two players or multi-player potential games equilibria of these games can be efficiently computed by fictitious play. The paper consider multi-player non-potential games such that the utility that a player achieves by playing an action can be decomposed as the sum of the utilities that this action will give in two-players games against a few of other players. Indeed, it is known that these games in their normal form (non stochastic non repeated) enjoy a lot of common properties with the two-player zero sum games, such as the possibility to efficiently compute an equilibirum.\n\nUnfortunately, the answer of the paper is essentially negative: computing an equilibrium for this class of stochastic games is hard even for coarse correlated equilibrium, unless the utility relationships is such that there is a center player c such that the utility of all remaining players depends on c only. In this case, fictitious play is known to converge to the equilibrium under opportune assumptions on the rate at which the relevant learning value are updated. Finally, it is showed that is possible to compute through multiplicative weights updates a dynamics converging to an approximation of a non-stationary equilibrium, i.e., an equilibrium that is not a fixed point independent of history.\n\nSTRENGTHS:\nThe problem attracted the interest of the community in recent years, and hence it is relevant. Moreover it is a well-motivated class of problems, both for their applications and for the theoretical foundations (as stated above in the summary).\n\nThe contribution is clearly stated, and the paper is quite easy to read.\n\nWEAKNESSES:\nThe results are essentially negative and positive cases have quite limited applicztion.\nE.g., Prop 3 (stating correlated equilibria are the same as Nash equilibria) only holds for decomposable  transition matrix, that in turn mean that there must be at least a single player that depends on all other player, that is often not the case (usually local interactions are very local).\nThm 1 allows equilibrium computation only for a further restriction of this class. And convergence of fictitious play holds under further assumptions.\n\nQUESTIONS:\nDoes decomposable tranisition matrix mean that N_c >= 1? If so, I believe that would make the definition more easy to grasp to write down this fact.\n\nRemove double point at the end of page 8.\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The authors define a notion of Markov Games with local interactions with the extra assumption that are zero-sum (inspired from the polymatrix/graphical games that appeared in papers in Algorithmic Game Theory). They manage to show that Markovian Coarse Correlated Equilibria correspond to Markovian Nash Equilibria when marginalized (called the phenomenon collapsing) in their setting. They also show that approximating stationary and Markovian NE  is PPAD-hard and they finally provide an algorithm that gives Markovian non-stationary NE for their class of Markov Games.

STRENGTHS:
The authors manage to define another class of Markov games in which finding NE (Markovian but not stationary) is tractable. The collapsing phenomenon is quite interesting and the idea of using QRE to get Markovian non-stationary NE is natural and elegant.

WEAKNESSES:
The model assumes some decomposability of the probability transition matrix that makes the model a bit complicated to parse. It seems though that the model captures interesting settings. The hardness result is not surprising and seems straightforward from Daskalakis et al.

QUESTIONS:
Can you elaborate more on the uniqueness of the QRE? If this is the case for infinite horizon settings, why your algorithm does not work for infinite horizon and you need to truncate after log(1/eps) episodes?

LIMITATIONS:
Not applicable
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies multi-player zero-sum Markov games with networked interactions, generalizing the well-studied class of polymatrix normal-form games. They first show that Markov coarse correlated equilibria (CCE) collapse to Markov Nash equilibria (NE), allowing to focus on computing Markov CCE for which they provide polynomial-time algorithms based on optimistic multiplicative weights update. They also rely on recent advances to show that the stronger notion of stationary CCE is PPAD-hard, unless the network is star-shaped. For the latter class of Markov games, they establish that fictitious play converges asymptotically to the set of stationary NE.\n\nSTRENGTHS:\nThe paper studies a natural subclass of Markov games inspired by a number of positive results on polymatrix zero-sum normal-form games. The authors motivate this class with a number of concrete examples and applications, and they make concrete contributions towards understanding the complexity of computing equilibria in that class of games. In particular, they show that certain positive results in normal-form games can be extended to their Markov games counterpart as well, eluding a number of hardness results in general Markov games. These are some of the few positive results for Markov NE under a non-trivial class of multi-player games, so I believe the results will be valued by the community. \n\nThe paper also appears sound; I did not find any notable issue in the proofs. The paper is also well-written and organized, and the key ideas are nicely exposed in the main body.\n\nWEAKNESSES:\nThere are a couple of issues regarding the exposition/novelty of the results. First, the authors claim that they provide the first guarantees even in zero-sum polymatrix normal-form games under fictitious-play dynamics, but this does not appear to be the case. In particular, see the paper \"Fictitious play in Networks.\" (Note that the latter paper also provides guarantees under discrete-time FP.) In fact, the results of the latter paper apply without the restrictive assumption of a star-shaped topology. So the authors should carefully clarify the connection with the aforementioned paper. I should note that this does not affect significantly the narrative of the paper since the results on the more general class of Markov games are novel and non-trivial even in light of that other paper. For completeness, it would be good to see if your techniques provide asymptotic guarantees for polymatrix zero-sum normal-form games without the star-shaped assumption. \n\nThe second issue relates to the novelty of the results in Section 6. In light of the collapse of Markov CCE with Markov NE, one can obtain immediately a polynomial-time algorithm using any algorithm for computing Markov CCE (e.g., the one by Daskalakis et al. 2022). In contrast, instead of using known algorithms, the authors design a new algorithm whose analysis is quite technically involved; it is not clear why the authors did not simply rely on known algorithms. At the very least, the authors should clarify that getting a Markov NE follows directly by the shown collapse; the current write-up can cause confusion. The designed algorithm based on OMWU has the nice property of getting a rate of $1/T$, which is certainly not the case with other algorithms for computing Markov CCE, but this is not highlighted at all. Regarding the last point, if one only cares about centralized algorithms, it should be possible to use the ellipsoid against hope algorithm of Papadimitriou and Roughgarden (along with backward induction) to get an exact Markov CCE (and hence Markov NE).\n\nQUESTIONS:\nSome minor issues:\n\n1. The exposition in Section 3 regarding Markov games with network interactions appears to be more complex than what is actually needed. I would recommend slightly revising and simplifying the exposition in Section 3.1 \n2. Is there a polynomial algorithm for computing stationary NE under a star-shaped topology? The results based on FP do not give any non-asymptotic guarantees\n3. A related paper worth inclusing: “Fast Convergence of Optimistic Gradient Ascent in Network Zero-Sum Extensive Form Games” \n4. Propositions and theorems are typically in italic, but I leave this up to the authors\n5. Line 373: approximates -> approximate\n6. Line 668: correlated equilibria (CCE)\n7. Line 718: do -> take\n8. Line 719: gives -> which gives\n9. Line 1001: It maybe good to point to Appendix E here for the definition of a differential inclusion as it is quite non standard\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}