Title: Zero-sum Polymatrix Markov Games: Equilibrium Collapse and Efficient Computation of Nash Equilibria
Venue: NeurIPS 2023
Accepted: yes
Keywords: network games

Nash equilibrium

equilibrium

game theory

learning
Abstract: The works of (Daskalakis et al., 2009, 2022; Jin et al., 2022; Deng et al., 2023) indicate that computing Nash equilibria in multi-player Markov games is a computationally hard task.  This fact raises the question of whether or not computational intractability can be circumvented if one focuses on specific classes of Markov games. One such example is two-player zero-sum Markov games, in which efficient ways to compute a Nash equilibrium are known. Inspired by zero-sum polymatrix normal-form games (Cai et al., 2016), we define a class of zero-sum multi-agent Markov games in which there are only pairwise interactions described by a graph that changes per state.
For this class of Markov games, we show that an $\epsilon$-approximate Nash equilibrium can be found efficiently. To do so, we generalize the techniques of (Cai et al., 2016), by showing that the set of coarse-correlated equilibria collapses to the set of Nash equilibria. Afterwards, it is possible to use any algorithm in the literature that computes approximate coarse-correlated equilibria Markovian policies to get an approximate Nash equilibrium.

Zero-sum Polymatrix Markov Games: Equilibrium Collapse and Efficient Computation of Nash Equilibria

Abstract

The works of (Daskalakis et al., 2009, 2022; Jin et al., 2022; Deng et al., 2023) indicate that computing Nash equilibria in multi-player Markov games is a computationally hard task. This fact raises the question of whether or not computational intractability can be circumvented if one focuses on specific classes of Markov games. One such example is two-player zero-sum Markov games, in which efficient ways to compute a Nash equilibrium are known. Inspired by zero-sum polymatrix normal-form games (Cai et al., 2016), we define a class of zero-sum multi-agent Markov games in which there are only pairwise interactions described by a graph that changes per state. For this class of Markov games, we show that an ϵ -approximate Nash equilibrium can be found efficiently. To do so, we generalize the techniques of (Cai et al., 2016), by showing that the set of coarse-correlated equilibria collapses to the set of Nash equilibria. Afterwards, it is possible to use any algorithm in the literature that computes approximate coarse-correlated equilibria Markovian policies to get an approximate Nash equilibrium.

1 Introduction

Multi-agent reinforcement learning (MARL) is the discipline that is concerned with strategic interactions between agents who find themselves in a dynamically changing environment. Early aspects of MARL can be traced back to, as early as, the initial text on two-player zero-sum stochastic/Markov games (Shapley, 1953). Today, Markov games have been established as the theoretical framework for MARL (Littman, 1994). The connection between game theory and MARL has lead to several recent cornerstone results in benchmark domains in AI (Bowling et al., 2015; Brown and Sandholm, 2019, 2018; Brown et al., 2020; Silver et al., 2017; Moravˇ cík et al., 2017; Perolat et al., 2022; Vinyals et al., 2019). The majority of the aforementioned breakthroughs relied on computing Nash equilibria (Nash, 1951) in a scalable and often decentralized manner. Although the theory of single agent reinforcement learning (RL) has witnessed an outstanding progress ( e.g. , see (Agarwal et al., 2020; Bertsekas, 2000; Jin et al., 2018; Li et al., 2021; Luo et al., 2019; Panait and Luke, 2005; Sidford et al., 2018; Sutton and Barto, 2018), and references therein), the landscape of multi-agent settings eludes a thorough understanding. In fact, guarantees for provably efficient computation of Nash equilibria remain limited to either environments in which agents strive to coordinate towards a shared goal (Chen et al., 2022; Claus and Boutilier, 1998; Ding et al., 2022; Fox et al., 2022; Leonardos et al., 2021; Maheshwari et al., 2022; Wang and Sandholm, 2002; Zhang et al., 2021) or fully competitive such as two-player zero-sum games (Cen et al., 2021; Condon, 1993; Daskalakis et al., 2020; Sayin et al., 2021, 2020; Wei et al., 2021) to name a few. Part of the lack of efficient algorithmic results in MARL is the fact that computing approximate Nash equilibria in (general-sum) games is computationally intractable (Daskalakis et al., 2009; Rubinstein, 2017; Chen et al., 2009; Etessami and Yannakakis, 2010) even when the games have a single state, i.e. , normal-form two-player games.

We aim at providing a theoretical framework that captures an array of real-world applications with multiple agents - which admittedly correspond to a big portion of all modern applications. A recent contribution that computes NE efficiently in a setting that combines both collaboration and competition, (Kalogiannis et al., 2022), concerns adversarial team Markov games, or competition between an adversary and a group of uncoordinated agents with common rewards. Efficient algorithms for computing Nash equilibria in settings that include both cooperation and competition are far fewer and tend to impose assumptions that are restrictive and difficult to meet in most applications (Bowling, 2000; Hu and Wellman, 2003). The focus of our work is centered around the following question:

Are there any other settings of Markov games that encompass both competition and coordination while mantaining the tractability of Nash equilibrium computation? ( ⋆ )

Inspired by contemporary works in algorithmic game theory and specifically zero-sum polymatrix normal-form games (Cai et al., 2016), we focus on the problem of computing Nash equilibria in zero-sum polymatrix Markov games. Informally, a polymatrix Markov game is a multi-agent Markov decision process with n agents, state-space S , action space A k for agent k , a transition probability model P and is characterized by a graph G s ( V , E s ) which is potentially different in every state s . For a fixed state s , the nodes of the graph V correspond to the agents, and the edges E s of the graph are two-player normal-form games (different per state). Every node/agent k has a fixed set of actions A k , and chooses a strategy from this set to play in all games corresponding to adjacent edges. Given an action profile of all the players, the node's reward is the sum of its rewards in all games on the edges adjacent to it. The game is globally zero-sum if, for all strategy profiles, the rewards of all players add up to zero. Afterwards, the process transitions to a state s ′ according to P . In a more high-level description, the agents interact over a network whose connections change at every state.

Our results. We consider a zero-sum polymatrix Markov game with the additional property that a single agent (not necessarily the same) controls the transition at each state, i.e. , the transition model is affected by a single agent's actions for each state s . These games are known as switching controller Markov games. We show that we can compute in time poly ( |S| , n, max i ∈ [ n ] |A i | , 1 /ϵ ) an ϵ -approximate Nash equilibrium. The proof relies on the fact that zero-sum polymatrix Markov games with a switching controller have the following important property: the marginals of a coarsecorrelated equilibrium constitute a Nash equilibrium (see Section 3.2). We refer to this phenomenon as equilibrium collapse . This property was already known for zero-sum polymatrix normal-form games by Cai et al. (2016) and our results generalize the aforementioned work for Markov games. As a corollary, we get that any algorithm in the literature that guarantees convergence to approximate coarse-correlated equilibria Markovian policiese.g. , (Daskalakis et al., 2022)-can be used to get approximate Nash equilibria. Our contribution also unifies previous results that where otherwise only applicable to the settings of single and switching-control two-player zero-sum games , or zero-sum polymatrix normal-form games . Finally, we show that the equilibrium collapsing phenomenon does not carry over if there are two or more controllers per state (see Section 3.3).

Technical overview. In order to prove our results, we rely on nonlinear programming and, in particular, nonlinear programs whose optima coincide with the Nash equilibria for a particular Markov game (Filar et al., 1991; Filar and Vrieze, 2012). Our approach is analogous to the one used by (Cai et al., 2016) which uses linear programming to prove the collapse of the set of CCE to the set of NE. Nevertheless, using the duality of linear programming in our case is not possible since a Markov game introduces nonlinear terms in the program. It is noteworthy that we do not need to invoke (Lagrangian) duality or an argument that relies on stationary points of a Lagrangian function. Rather, we use the structure of the zero-sum polymatrix Markov games with a switching controller to conclude the relation between a correlated policy and the individual policies formed by its marginals in terms of the individual utilities of the game.

1.1 Importance of zero-sum polymatrix Markov games

Strategic interactions of agents over a network is a topic of research in multiple disciplines that span computer science (Easley and Kleinberg, 2010), economics (Schweitzer et al., 2009), control theory (Tipsuwan and Chow, 2003), and biology (Szabó and Fath, 2007) to name a few.

In many environments where multiple agents interact with each other, they do so in a localized manner. That is, every agent is affected by the set of agents that belong to their immediate 'neighborhood'. Further, it is quite common that these agents will interact independently with each one of their

neighbors; meaning that the outcome of their total interactions is a sum of pairwise interactions rather than interactions that depend on joint actions. Finally, players might remain indifferent to actions of players are not their neighbors.

To illustrate this phenomenon we can think of multiplayer e-games ( e.g. , CS:GO, Fortnite, League of Legends, etc) where each player interacts through the same move only with players that are present on their premises and, in general, the neighbors cannot combine their actions into something that is not a mere sum of their individual actions ( i.e., they rarely can 'multiply' the effect of the individual actions). In other scenarios, such as strategic games played on social networks ( e.g. , opinion dynamics) agents clearly interact in a pairwise manner with agents that belong to their neighborhood and are somewhat oblivious to the actions of agents who they do not share a connection with.

With the proposed model we provide the theoretical framework needed to reason about such strategic interactions over dynamically changing networks.

1.2 Related work

From the literature of Markov games, we recognize the settings of single controller (Filar and Raghavan, 1984; Sayin et al., 2022; Guan et al., 2016; Qiu et al., 2021) and switching controller (Vrieze et al., 1983) Markov games to be one of the most related to ours. In these settings, all agents' actions affect individual rewards, but in every state one particular player ( single controller ), or respectively a potentially different one ( switching controller ), controls the transition of the environment to a new state. To the best of our knowledge, prior to our work, the only Markov games that have been examined under this assumption are either zero-sum or potential games.

Further, we manage to go beyond the dichotomy of absolute competition or absolute collaboration by generalizing zero-sum polymatrix games to their Markovian counterpart. In this sense, our work is related to previous works of Cai et al. (2016); Anagnostides et al. (2022); Ao et al. (2022) which show fast convergence to Nash equilibria in zero-sum polymatrix normal-form games for various no-regret learning algorithms including optimistic gradient descent.

2 Preliminaries

Notation. We define [ n ] := { 1 , · · · , n } . Scalars are denoted using lightface variables, while, we use boldface for vectors and matrices. For simplicity in the exposition, we use O ( · ) to suppress dependencies that are polynomial in the parameters of the game. Additionally, given a collection x of policies or strategies for players [ n ] , x -k denotes the policies of every player excluding k .

2.1 Markov games

In its most general form, a Markov game (MG) with a finite number of n players is defined as a tuple Γ( H, S , {A k } k ∈ [ n ] , P , { r k } k ∈ [ n ] , γ, ρ ) . Namely,

H ∈ N + denotes the time horizon , or the length of each episode,

S , with cardinality S := |S| , stands for the state space,

{A k } k ∈ [ n ] is the collection of every player's action space, while A := A 1 × · · · × A n denotes the joint action space ; further, an element of that set -a joint action- is generally noted as a = ( a 1 , . . . , a n ) ∈ A ,

P := { P h } h ∈ [ H ] is the set of all transition matrices , with P h : S × A → ∆( S ) ; further, P h ( ·| s, a ) marks the probability of transitioning to every state given that the joint action a is selected at time h and state s -in infinite-horizon games P does not depend on h and the index is dropped,

r k := { r k,h } is the reward function of player k at time h ; r k,h : S , A → [ -1 , 1] yields the reward of player k at a given state and joint action - in infinite-horizon games, r k,h is the same for every h and the index is dropped,

a discount factor γ > 0 , which is generally set to 1 when H < ∞ , and γ < 1 when H →∞ ,

an initial state distribution ρ ∈ ∆( S ) .

Policies and value functions. We will define stationary and nonstationary Markov policies. When the horizon H is finite, a stationary policy equilibrium need not necessarily exist even for a singleagent MG, i.e. , a Markov decision process; in this case, we seek nonstationary policies. For the case of infinite-horizon games, it is folklore that a stationary Markov policy Nash equilibrium always exists.

We note that a policy is Markovian when it depends on the present state only. A nonstationary Markov policy π k for player k is defined as π k := { π k,h : S → ∆( A k ) , ∀ h ∈ [ H ] } . It is a sequence of mappings of states s to a distribution over actions ∆( A k ) for every timestep h . By π k,h ( a | s ) we will denote the probability of player k taking action a in timestep h and state s . A Markov policy is said to be stationary in the case that it outputs an identical probability distribution over actions whenever a particular state is visited regardless of the corresponding timestep h .

Further, we define a nonstationary Markov joint policy σ := { π h , ∀ h ∈ [ H ] } to be a sequence of mappings from states to distributions over joint actions ∆( A ) ≡ ∆( A 1 ×···×A n ) for all times steps h in the time horizon. In this case, the players can be said to share a common source of randomness, or that the joint policy is correlated.

A joint policy π will be said to be a product policy if there exist policies π k : [ H ] × S → ∆( A k ) , ∀ k ∈ [ n ] such that π h = π 1 ,h ×··· × π n,h , ∀ h ∈ [ H ] . Moreover, given a joint policy π we let a joint policy π -k stand for the marginal joint policy excluding player k , i.e. ,

By fixing a joint policy π we can define the value function of any given state s and timestep h for every player k as the expected cumulative reward they get from that state and timestep h onward,

Depending on whether the game is of finite or infinite horizon we get the followin displays,

In finite-horizon games, γ = 1 , the value function reads,

In infinite-horizon games, the value function of each state is,

Where P h ( π h ) , P ( π ) and r h ( π h ) , r ( π ) denote the state-to-state transition probability matrix and expected per-state reward vector for a given policy π h or π accordingly. Additionally, e s 1 is an allzero vector apart of a value of 1 in its s 1 -th position. Also, we denote V π k,h ( ρ ) = ∑ s ∈S ρ ( s ) V π k,h ( s ) .

Best-response policies. Given an arbitrary joint policy σ , we define the best-response policy of a player k to be a policy π † k := { π † k,h , ∀ h ∈ [ H ] } , such that it is a maximizer of max π ′ k V π ′ k × σ -k k, 1 ( s 1 ) . Additionally, we will use the following notation V † , σ -k k,h ( s ) := max π ′ k V π ′ k × σ -k k,h ( s ) .

Equilibrium notions. Having defined what a best-response is, it is then quite direct to define different notions of equilibria for Markov games.

Definition 2.1 (CCE) . We will say that a joint (potentially correlated) policy σ ∈ ∆( A ) H × S is an ϵ -approximate coarse-correlated equilibrium if it holds that, for an ϵ > 0 ,

Further, we will define a Nash equilibrium policy,

Definition 2.2 (NE) . A joint, product policy π ∈ ∏ k ∈ [ n ] ∆( A k ) H × S is an ϵ -approximate Nash equilibrium if it holds that, for an ϵ > 0 ,

It is quite evident that an approximate Nash equilibrium is also an approximate coarse-correlated equilibrium while the converse is not generally true. For infinite-horizon games the definitions are analogous and are deferred to the appendix.

2.2 Our setting

We focus on the setting of zero-sum polymatrix switching-control Markov games. This setting encompasses two major assumptions related to the reward functions in every state { r k } k ∈ [ n ] and the transition kernel P . The first assumption imposes a zero-sum, polymatrix structure on { r k } k ∈ [ n ] for every state and directly generalizes zero-sum polymatrix games for games with multiple states.

Assumption 1 (Zero-sum polymatrix games) . The reward functions of every player in any state s are characterized by a zero-sum , polymatrix structure.

Polymatrix structure. For every state s there exists an undirected graph G s ( V , E s ) where,

the set of nodes V coincides with the set of agents [ n ] ; the k -th node is the k -th agent,

̸

the set of edges E s stands for the set of pair-wise interactions; each edge e = ( k, j ) , k, j ∈ [ n ] , k = j stands for a general-sum normal-form game played between players k, j and which we note as ( r kj ( s, · , · ) , r jk ( s, · , · ) ) with r kj , r jk : S × A k ×A j → [ -1 , 1] .

Moreover, we define adj( s, k ) := { j ∈ [ n ] | ( k, j ) ∈ E s } ⊆ [ n ] to be the set of all neighbors of an arbitrary agent k in state s . The reward of agent k at state s given a joint action a depends solely on interactions with their neighbors,

Further, the zero-sum assumption implies that,

In the infinite-horizon setting, the subscript h can be dropped.

A further assumption ( switching-control ) is necessary in order to ensure the desirable property of equilibrium collapse.

Assumption 2 (Switching-control) . In every state s ∈ S , there exists a single player (not necessarily the same), or controller , whose actions determine the probability of transitioning to a new state.

The function argctrl : S → [ n ] returns the index of the player who controls the transition probability at a given state s . On the other hand, the function ctrl : S × A → A argctrl( s ) gets an input of a joint action a , for a particular state s , and returns the action of the controller of that state, a argctrl( s ) .

Remark 1. It is direct to see that Markov games with a single controller and turn-based Markov games (Daskalakis et al., 2022), are special case of Markov games with switching controller.

3 Main results

In this section we provide the main results of this paper. We shall show the collapsing phenomenon of coarse-correlated equilibria to Nash equilibria in the case of zero-sum, single switching controller polymatrix Markov games. Before we proceed, we provide a formal definition of the notion of collapsing.

Definition 3.1 (CCE collapse to NE) . Let σ be any ϵ -CCE policy of a Markov game. Moreover, let the marginal policy π σ := ( π σ 1 , ..., π σ n ) be defined as:

If π σ is a O ( ϵ ) -NE equilibrium for every σ then we say the set of approximate CCE's collapses to that of approximate NE's.

We start with the warm-up result that the set of CCE's collapses to the set of NE's for two-player zero-sum Markov games.

3.1 Warm-up: equilibrium collapse in two-player zero-sum MG's

Since we focus on two-player zero-sum Markov games, we simplify the notation by using V · h =1 ( s ) := V · 2 , 1 ( s ) -i.e. , player 1 is the minimizing player and player 2 is the maximizer. We show the following theorem:

Theorem 3.1 (Collapse in two-player zero-sum MG's) . Let a two-player zero-sum Markov game Γ ′ and an ϵ -approximate CCE policy of that game σ . Then, the marginalized product policies π σ 1 , π σ 2 form a 2 ϵ -approximate NE.

Proof. Since σ is an ϵ -approximate CCE joint policy, by definition it holds that for any π 1 and any π 2 ,

Due to Claim A.1, the latter is equivalent to the following inequality,

Plugging in π σ 1 , π σ 2 alternatingly, we get the inequalities:

The latter leads us to conclude that for any π 1 and any π 2 ,

which is the definition of a NE in a zero-sum game.

3.2 Equilibrium collapse in finite-horizon polymatrix Markov games

In this section, we focus on the more challenging case of polymatrix Markov games which is the main focus of this paper. For any finite horizon Markov game, we define ( P NE ) to be the following nonlinear program with variables π , w :

Using the following theorem, we are able to use ( P NE ) to argue about equilibrium collapse.

Theorem 3.2 (NE and global optima of ( P NE )) . If ( π ⋆ , w ⋆ ) yields an ϵ -approximate global minimum of ( P NE ) , then π ⋆ is an nϵ -approximate NE of the zero-sum polymatrix switching controller MG, Γ . Conversely, if π ⋆ is an ϵ -approximate NE of the MG Γ with corresponding value function vector w ⋆ such that w ⋆ k,h ( s ) = V π ⋆ k,h ( s ) ∀ ( k, h, s ) ∈ [ n ] × [ H ] ×S , then ( π ⋆ , w ⋆ ) attains an ϵ -approximate global minimum of ( P NE ) .

Following, we are going to use ( P NE ) in proving the collapse of CCE's to NE's. We observe that the latter program is nonlinear and in general nonconvex. Hence, duality cannot be used in the way it was used in (Cai et al., 2016) to prove equilibrium collapse. Nevertheless, we can prove that given a CCE policy σ , the marginalized, product policy × k ∈ [ n ] π σ k along with an appropriate vector w σ achieves a global minimum in the nonlinear program ( P NE ). More precisely, our main result reads as the following statement.

Theorem 3.3 (CCE collapse to NE in polymatrix MG) . Let a zero-sum polymatrix switchingcontrol Markov game, i.e., a Markov game for which Assumptions 1 and 2 hold. Further, let an ϵ -approximate CCE of that game σ . Then, the marginal product policy π σ , with π σ k,h ( a | s ) = ∑ a -k ∈A -k σ h ( a, a -k ) , ∀ k ∈ [ n ] , ∀ h ∈ [ H ] is an nϵ -approximate NE.

Proof. Let an ϵ -approximate CCE policy, σ , of game Γ . Moreover, let the best-response value-vectors of each agent k to joint policy σ -k , w † k .

Now, we observe that due to Assumption 1,

Further, due to Assumption 2,

or,

Putting these pieces together, we reach the conclusion that ( π σ , w † ) is feasible for the nonlinear program ( P NE ).

What is left is to prove that it is also an ϵ -approximate global minimum. Indeed, if ∑ k w † k,h ( s 1 ) ≤ ϵ (by assumption of an ϵ -approximate CCE), then the objective function of ( P NE ) will attain an ϵ -approximate global minimum. In turn, due to Theorem 3.2 the latter implies that π σ is an nϵ -approximate NE.

We can now conclude that due to the algorithm introduced in (Daskalakis et al., 2022) for CCE computation in general-sum MG's, the next statement holds true.

Corollary 3.1 (Computing a NE-finite-horizon) . Given a finite-horizon switching control zerosum polymatrix Markov game, we can compute an ϵ -approximate Nash equilibrium policy that is Markovian with probability at least 1 -δ in time poly ( n, H, S, max k |A k | , 1 ϵ , log(1 /δ ) ) .

In the next section, we discuss the necessity of the assumption of switching control using a counterexample of non-collapsing equilibria.

3.3 No equilibrium collapse with more than one controllers per-state

Although Assumption 1 is sufficient for the collapse of any CCE to a NE in single-state ( i.e. , normalform) games, we will prove that Assumption 2 is indispensable in guaranteeing such a collapse in zero-sum polymatrix Markov games. That is, if more than one players affect the transition probability from one state to another, a CCE is not guaranteed to collapse to a NE.

Example 1. We consider the following 3 -player Markov game that takes place for a time horizon H = 3 . There exist three states, s 1 , s 2 , and s 3 and the game starts at state s 1 . Player 3 has a single action in every state, while players 1 and 2 have two available actions { a 1 , a 2 } and { b 1 , b 2 } respectively in every state.

Reward functions. If player 1 (respectively, player 2 ) takes action a 1 (resp., b 1 ), in either of the states s 1 or s 2 , they get a reward equal to 1 20 . In state s 3 , both players get a reward equal to -1 2 regardless of the action they select. Player 3 always gets a reward that is equal to the negative sum of the reward of the other two players. This way, the zero-sum polymatrix property of the game is ensured (Assumption 1).

Transition probabilities. If players 1 and 2 select the joint action ( a 1 , b 1 ) in state s 1 , the game will transition to state s 2 . In any other case, it will transition to state s 3 . The converse happens if in state s 2 they take joint action ( a 1 , b 1 ) ; the game will transition to state s 3 . For any other joint action, it will transition to state s 1 . From state s 3 , the game transitions to state s 1 or s 2 uniformly at random.

At this point, it is important to notice that two players control the transition probability from one state to another. In other words, Assumption 2 does not hold.

s

1

s

2

s

3

1

/

2

1

/

2

1

-

π

1

(

a

1

|

s

1

)

π

2

(

b

1

|

s

1

)

π

1

(

a

1

|

s

1

)

π

2

(

b

1

|

s

1

)

π

1

(

a

1

|

s

2

)

π

2

(

b

1

|

s

2

)

1

-

π

1

(

a

1

|

s

2

)

π

2

(

b

1

|

s

2

)

Next, we consider the joint policy σ ,

Claim 3.1. The joint policy σ that assigns probability 1 2 to the joint actions ( a 1 , b 2 ) and ( a 2 , b 1 ) in both states s 1 , s 2 is a CCE and V σ 1 , 1 ( s 1 ) = V σ 2 , 1 ( s 1 ) = 1 20 .

Yet, the marginalized product policy of σ which we note as π σ 1 × π σ 2 does not constitute a NE. The components of this policy are,

I.e., the product policy π σ 1 × π σ 2 selects any of the two actions of each player in states s 1 , s 2 independently and uniformally at random. With the following claim, it can be concluded that in general when more than one player control the transition the set of equilibria do not collapse.

Claim 3.2. The product policy π σ 1 × π σ 2 is not a NE.

In conclusion, Assumption 1 does not suffice to ensure equilibrium collapse.

Theorem 3.4. There exists a zero-sum polymatrix Markov game (Assumption 2 is not satisfied) that has a CCE which does not collapse to a NE.

3.4 Equilibrium collapse in infinite-horizon polymatrix Markov games

In proving equilibrium collapse for infinite-horizon polymatrix Markov games, we use similar arguments and the following nonlinear program with variables π , w ,

We note that Example 1 can be properly adjusted to show that the switching-control assumption is necessary for equilibrium collapse in infinite-horizon games as well. Compared to finite-horizon games, infinite-horizon games cannot be possibly solved using backward induction. They pose a genuine computational challenge and, in that sense, the importance of the property of equilibrium collapse gets highlighted.

Computational implications. Equilibrium collapse in infinite-horizon MG's allows us to use the CCE computation technique found in (Daskalakis et al., 2022) in order to compute an ϵ -approximate NE. Namely, given an accuracy threshold ϵ , we truncate the infinite-horizon game to its effective horizon H := log(1 /ϵ ) 1 -γ . Then, we define reward functions that depend on the time-step h , i.e. , r k,h = γ h -1 r k . Finally,

Corollary 3.2. (Computing a NE-infinite-horizon) Given an infinite-horizon switching control zero-sum polymatrix game Γ , it is possible to compute a Nash equilibrium policy that is Markovian and nonstationary with probability at least 1 -δ in time poly ( n, 1 1 -γ , S, max k |A k | , 1 ϵ , log(1 /δ ) ) .

4 Conclusion and open problems

In this paper, we unified switching-control Markov games and zero-sum polymatrix normal-form games. We highlighted how numerous applications can be modeled using this framework and we focused on the phenomenon of equilibrium collapse from the set of coarse-correlated equilibria to that of Nash equilibria. This property holds implications for computing approximate Nash equilibria in switching control zero-sum polymatrix Markov games; it ensures that it can be done efficiently.

Open problems. In light of the proposed problem and our results there are multiple interesting open questions:

Is it possible to use a policy optimization algorithm similar to those of (Erez et al., 2022; Zhang et al., 2022) in order to converge to an approximate Nash equilibrium? We note that the question can be settled in one of two ways; either extend the current result of equilibrium collapse to policies that are non-Markovian or guarantee convergence to Markovian policies. The notion of regret in (Erez et al., 2022) gives rise to the computation of a CCE that is a non-Markovian policy in the sense that the policy at every timestep depends on the policy sampled from the history of no-regret play and not only the given state.

We conjecture that a convergence rate of O ( 1 T ) to a NE is possible, i.e. , there exists an algorithm with running time O (1 /ϵ ) that computes an ϵ -approximate NE.

Are there more classes of Markov games in which computing Nash equilibria is computationally tractable?

References

Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference on Learning Theory, COLT 2020, 9-12 July 2020 , volume 125 of Proceedings of Machine Learning Research , pages 64-66. PMLR, 2020.

Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm. On last-iterate convergence beyond zero-sum games. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 536-581. PMLR, 2022. URL https://proceedings.mlr.press/ v162/anagnostides22a.html .

Ruicheng Ao, Shicong Cen, and Yuejie Chi. Asynchronous gradient play in zero-sum multi-agent games, 2022.

P. Bertsekas. Dynamic Programming and Optimal Control . Athena Scientific, 2nd edition, 2000. ISBN 1886529094.

Michael Bowling. Convergence problems of general-sum multiagent reinforcement learning. In In Proceedings of the Seventeenth International Conference on Machine Learning , pages 89-94. Morgan Kaufmann, 2000.

Michael Bowling, Neil Burch, Michael Johanson, and Oskari Tammelin. Heads-up limit hold'em poker is solved. Science , 347(6218):145-149, 2015. doi: 10.1126/science.1259433.

Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science , 359(6374):418-424, 2018. doi: 10.1126/science.aao1733.

Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science , 365(6456): 885-890, 2019. doi: 10.1126/science.aay2400.

Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining deep reinforcement learning and search for imperfect-information games. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020 , 2020.

Yang Cai, Ozan Candogan, Constantinos Daskalakis, and Christos Papadimitriou. Zero-sum polymatrix games: A generalization of minmax. Mathematics of Operations Research , 41(2):648-655, 2016.

Shicong Cen, Yuting Wei, and Yuejie Chi. Fast policy extragradient methods for competitive games with entropy regularization. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , pages 27952-27964, 2021.

Dingyang Chen, Qi Zhang, and Thinh T. Doan. Convergence and price of anarchy guarantees of the softmax policy gradient in markov potential games. In Decision Awareness in Reinforcement Learning Workshop at ICML 2022 , 2022.

Xi Chen, Xiaotie Deng, and Shang-Hua Teng. Settling the complexity of computing two-player nash equilibria. J. ACM , 56(3):14:1-14:57, 2009. doi: 10.1145/1516512.1516516.

Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. In Proceedings of the Fifteenth National Conference on Artificial Intelligence and Tenth Innovative Applications of Artificial Intelligence Conference, AAAI 98 , pages 746-752. AAAI Press / The MIT Press, 1998.

Anne Condon. On algorithms for simple stochastic games. In Advances in Computational Complexity Theory, volume 13 of DIMACS Series in Discrete Mathematics and Theoretical Computer Science , pages 51-73. American Mathematical Society, 1993.

Qiwen Cui, Kaiqing Zhang, and Simon Du. Breaking the curse of multiagents in a large state space: Rl in markov games with independent linear function approximation. In The Thirty Sixth Annual Conference on Learning Theory , pages 2651-2652. PMLR, 2023.

Constantinos Daskalakis, Paul W Goldberg, and Christos H Papadimitriou. The complexity of computing a nash equilibrium. SIAM Journal on Computing , 39(1):195-259, 2009.

Constantinos Daskalakis, Dylan J Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. Advances in neural information processing systems , 33: 5527-5540, 2020.

Constantinos Daskalakis, Noah Golowich, and Kaiqing Zhang. The complexity of markov equilibrium in stochastic games. arXiv preprint arXiv:2204.03991 , 2022.

Xiaotie Deng, Ningyuan Li, David Mguni, Jun Wang, and Yaodong Yang. On the complexity of computing markov perfect equilibrium in general-sum stochastic games. National Science Review , 10(1):nwac256, 2023.

Dongsheng Ding, Chen-Yu Wei, Kaiqing Zhang, and Mihailo R Jovanovi´ c. Independent policy gradient for large-scale markov potential games: Sharper rates, function approximation, and game-agnostic convergence. arXiv preprint arXiv:2202.04129 , 2022.

David Easley and Jon Kleinberg. Networks, crowds, and markets: Reasoning about a highly connected world . Cambridge university press, 2010.

Liad Erez, Tal Lancewicki, Uri Sherman, Tomer Koren, and Yishay Mansour. Regret minimization and convergence to equilibria in general-sum markov games, 2022.

Kousha Etessami and Mihalis Yannakakis. On the complexity of nash equilibria and other fixed points. SIAM J. Comput. , 39(6):2531-2597, 2010. doi: 10.1137/080720826.

Jerzy Filar and Koos Vrieze. Competitive Markov decision processes . Springer Science & Business Media, 2012.

Jerzy A Filar and TES Raghavan. A matrix game solution of the single-controller stochastic game. Mathematics of Operations Research , 9(3):356-362, 1984.

Jerzy A Filar, Todd A Schultz, Frank Thuijsman, and OJ Vrieze. Nonlinear programming and stationary equilibria in stochastic games. Mathematical Programming , 50(1-3):227-237, 1991.

Roy Fox, Stephen M. McAleer, Will Overman, and Ioannis Panageas. Independent natural policy gradient always converges in markov potential games. In International Conference on Artificial Intelligence and Statistics, AISTATS 2022 , volume 151 of Proceedings of Machine Learning Research , pages 4414-4425. PMLR, 2022.

Peng Guan, Maxim Raginsky, Rebecca Willett, and Daphney-Stavroula Zois. Regret minimization algorithms for single-controller zero-sum stochastic games. In 2016 IEEE 55th Conference on Decision and Control (CDC) , pages 7075-7080. IEEE, 2016.

Junling Hu and Michael P. Wellman. Nash q-learning for general-sum stochastic games. J. Mach. Learn. Res. , 4:1039-1069, 2003.

Chi Jin, Zeyuan Allen-Zhu, Sébastien Bubeck, and Michael I. Jordan. Is q-learning provably efficient? In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018 , pages 4868-4878, 2018.

Yujia Jin, Vidya Muthukumar, and Aaron Sidford. The complexity of infinite-horizon generalsum stochastic games. CoRR , abs/2204.04186, 2022. doi: 10.48550/arXiv.2204.04186. URL https://doi.org/10.48550/arXiv.2204.04186 .

Fivos Kalogiannis, Ioannis Anagnostides, Ioannis Panageas, Emmanouil-Vasileios VlatakisGkaragkounis, Vaggos Chatziafratis, and Stelios Stavroulakis. Efficiently computing nash equilibria in adversarial team markov games. arXiv preprint arXiv:2208.02204 , 2022.

Stefanos Leonardos, Will Overman, Ioannis Panageas, and Georgios Piliouras. Global convergence of multi-agent policy gradient in markov potential games. arXiv preprint arXiv:2106.01969 , 2021.

Yuanzhi Li, Ruosong Wang, and Lin F. Yang. Settling the horizon-dependence of sample complexity in reinforcement learning. In 62nd IEEE Annual Symposium on Foundations of Computer Science, FOCS 2021 , pages 965-976. IEEE, 2021. doi: 10.1109/FOCS52979.2021.00097.

Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994 , pages 157-163. Elsevier, 1994.

Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In 7th International Conference on Learning Representations, ICLR 2019 . OpenReview.net, 2019.

Chinmay Maheshwari, Manxi Wu, Druv Pai, and Shankar Sastry. Independent and decentralized learning in markov potential games, 2022.

Matej Moravˇ cík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science , 356(6337):508-513, 2017. doi: 10.1126/science. aam6960.

John Nash. Non-cooperative games. Annals of mathematics , pages 286-295, 1951.

Gergely Neu and Ciara Pike-Burke. A unifying view of optimism in episodic reinforcement learning. Advances in Neural Information Processing Systems , 33:1392-1403, 2020.

Panait and S. Luke. Cooperative Multi-Agent Learning: The State of the Art. Autonomous Agents and Multi-Agent Systems , 11(3):387-434, Nov 2005. doi: 10.1007/s10458-005-2631-2.

Julien Perolat, Bart de Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T. Connor, Neil Burch, Thomas Anthony, Stephen McAleer, Romuald Elie, Sarah H. Cen, Zhe Wang, Audrunas Gruslys, Aleksandra Malysheva, Mina Khan, Sherjil Ozair, Finbarr Timbers, Toby Pohlen, Tom Eccles, Mark Rowland, Marc Lanctot, Jean-Baptiste Lespiau, Bilal Piot, Shayegan Omidshafiei, Edward Lockhart, Laurent Sifre, Nathalie Beauguerlange, Remi Munos, David Silver, Satinder Singh, Demis Hassabis, and Karl Tuyls. Mastering the game of stratego with model-free multiagent reinforcement learning, 2022.

Shuang Qiu, Xiaohan Wei, Jieping Ye, Zhaoran Wang, and Zhuoran Yang. Provably efficient fictitious play policy optimization for zero-sum markov games with structured transitions. In International Conference on Machine Learning , pages 8715-8725. PMLR, 2021.

Aviad Rubinstein. Settling the complexity of computing approximate two-player nash equilibria. SIGecom Exch. , 15(2):45-49, 2017. doi: 10.1145/3055589.3055596.

Muhammed Sayin, Kaiqing Zhang, David Leslie, Tamer Basar, and Asuman Ozdaglar. Decentralized q-learning in zero-sum markov games. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 18320-18334. Curran Associates, Inc., 2021.

Muhammed O Sayin, Francesca Parise, and Asuman Ozdaglar. Fictitious play in zero-sum stochastic games. arXiv preprint arXiv:2010.04223 , 2020.

Muhammed O Sayin, Kaiqing Zhang, and Asuman Ozdaglar. Fictitious play in markov games with single controller. In Proceedings of the 23rd ACM Conference on Economics and Computation , pages 919-936, 2022.

Frank Schweitzer, Giorgio Fagiolo, Didier Sornette, Fernando Vega-Redondo, Alessandro Vespignani, and Douglas R White. Economic networks: The new challenges. science , 325(5939):422-425, 2009.

Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences , 39(10): 1095-1100, 1953.

Aaron Sidford, Mengdi Wang, Xian Wu, Lin Yang, and Yinyu Ye. Near-optimal time and sample complexities for solving markov decision processes with a generative model. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018 , pages 5192-5202, 2018.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nat. , 550(7676):354-359, 2017. doi: 10.1038/nature24270.

S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction . A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.

György Szabó and Gabor Fath. Evolutionary games on graphs. Physics reports , 446(4-6):97-216, 2007.

Yodyium Tipsuwan and Mo-Yuen Chow. Control methodologies in networked control systems. Control engineering practice , 11(10):1099-1111, 2003.

Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nat. , 575(7782): 350-354, 2019. doi: 10.1038/s41586-019-1724-z.

OJ Vrieze, SH Tijs, TES Raghavan, and JA Filar. A finite algorithm for the switching control stochastic game. Or Spektrum , 5(1):15-24, 1983.

Xiaofeng Wang and Tuomas Sandholm. Reinforcement learning to play an optimal nash equilibrium in team markov games. In Advances in Neural Information Processing Systems 15 [Neural Information Processing Systems, NIPS 2002, December 9-14, 2002 , pages 1571-1578. MIT Press, 2002.

Yuanhao Wang, Qinghua Liu, Yu Bai, and Chi Jin. Breaking the curse of multiagency: Provably efficient decentralized multi-agent rl with function approximation. arXiv preprint arXiv:2302.06606 , 2023.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive markov games. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 4259-4299. PMLR, 15-19 Aug 2021.

Runyu Zhang, Zhaolin Ren, and Na Li. Gradient play in stochastic games: stationary points, convergence, and sample complexity. arXiv preprint arXiv:2106.00198 , 2021.

Runyu Zhang, Qinghua Liu, Huan Wang, Caiming Xiong, Na Li, and Yu Bai. Policy optimization for markov games: Unified framework and faster convergence, 2022.

A Missing statements and proofs

A.1 Statements for Section 3.1

Claim A.1. Let a two-player Markov game where both players affect the transition. Further, consider a correlated policy σ and its corresponding marginalized product policy π σ = π σ 1 × π σ 2 . Then, for any π ′ 1 , π ′ 2 ,

Proof. We will effectively show that the problem of best-responding to a correlated policy σ is equivalent to best-responding to the marginal policy of σ for the opponent. The proof follows from the equivalence of the two MDPs.

As a reminder,

As we have seen in Section 2.1, in the case of unilateral deviation from joint policy σ , an agent faces a single agent MDP. More specifically, agent 2 , best-responds by optimizing a reward function ¯ r 2 ,h ( s, b ) under a transition kernel ¯ P 2 for which,

Similarly,

Analogously, for each of the transition kernels,

Hence, it follows that, V σ -2 × π ′ 2 2 , 1 ( s 1 ) = V π σ 1 × π ′ 2 2 , 1 ( s 1 ) , ∀ π ′ 2 and V π ′ 1 × σ -1 1 , 1 ( s 1 ) = V π ′ 1 × π σ 2 1 , 1 ( s 1 ) , ∀ π ′ 2 .

A.2 Proof of Theorem 3.2

The best-response program. First, we state the following lemma that will prove useful for several of our arguments,

Lemma A.1 (Best-response LP) . Let a (possibly correlated) joint policy ˆ σ . Consider the following linear program with variables w ∈ R n × H × S ,

( P BR ) min ∑ k ∈ [ n ] w k,s ( s 1 ) -e ⊤ s 1 H ∑ h =1 ( h ∏ τ =1 P τ (ˆ σ τ ) ) r k,h (ˆ σ h ) s . t . w k,h ( s ) ≥ r k,h ( s, a, ˆ σ -k,h ) + P h ( s, a, ˆ σ -k,h ) w k,h +1 , ∀ s ∈ S , ∀ h ∈ [ H ] , ∀ k ∈ [ n ] , ∀ a ∈ A k ; w k,H ( s ) = 0 , ∀ k ∈ [ n ] , ∀ s ∈ S .

The optimal solution w † of the program is unique and corresponds to the value function of each player k ∈ [ n ] when player k best-responds to ˆ σ .

as for agent 1

Proof. We observe that the program is separable to n independent linear programs, each with variables w k ∈ R n × H ,

Each of these linear programs describes the problem of a single agent MDP (Neu and Pike-Burke, 2020, Section 2) -that agent being k - which, as we have seen in Best-response policies, is equivalent to the problem of finding a best-response to ˆ σ -k . It follows that the optimal w † k for every program is unique (each program corresponds to a set of Bellman optimality equations).

Properties of the NE program. Second, we need to prove that the minimum value of the objective function of the program is nonnegative.

Lemma A.2 (Feasibility of ( P ′ NE ) and global optimum) . The nonlinear program ( P ′ NE ) is feasible, has a nonnegative objective value, and its global minimum is equal to 0 .

Proof. Analogously to the finite-horizon case, for the feasibility of the nonlinear program, we invoke the theorem of the existence of a Nash equilibrium. We let a NE product policy, π ⋆ , and a vector w ⋆ ∈ R n × S such that w ⋆ k ( s ) = V † , π ⋆ -k k ( s ) , ∀ k ∈ [ n ] ×S .

By Lemma A.1, we know that ( π ⋆ , w ⋆ ) satisfies all the constraints of ( P NE ). Additionaly, because π ⋆ is a NE, V π ⋆ k,h ( s 1 ) = V † , π ⋆ -k k,h ( s 1 ) for all k ∈ [ n ] . Observing that,

concludes the argument that a NE attains an objective value equal to 0 .

Continuing, we observe that due to (1) the objective function can be equivalently rewritten as,

Next, we focus on the inequality constraint

which holds for all s ∈ S , all players k ∈ [ n ] , all a ∈ A k , and all timesteps h ∈ [ H -1] .

By summing over a ∈ A k while multiplying each term with a corresponding coefficient π k,h ( a | s ) , the display written in an equivalent element-wise vector inequality reads:

Finally, after consecutively substituting w k,h +1 with the element-wise lesser term r k,h +1 ( π h +1 ) + P h +1 ( π h +1 ) w k,h +2 , we end up with the inequality:

Summing over k , it holds for the s 1 -th entry of the inequality,

Where the equality holds due to the zero-sum property, (1).

An approximate NE is an approximate global minimum. We show that an ϵ -approximate NE, π ⋆ , achieves an nϵ -approximate global minimum of the program. Utilizing Lemma A.1, setting w ⋆ k ( s 1 ) = V † , π ⋆ -k k, 1 ( s 1 ) , and the definition of an ϵ -approximate NE we see that,

Indeed, this means that π ⋆ , w ⋆ is an nϵ -approximate global minimizer of ( P NE ).

An approximate global minimum is an approximate NE. For the opposite direction, we let a feasible ϵ -approximate global minimizer of the program ( P NE ), ( π ⋆ , w ⋆ ) . Because a global minimum of the program is equal to 0 , an ϵ -approximate global optimum must be at most ϵ > 0 . We observe that for every k ∈ [ n ] ,

which follows from induction on the inequality constraint over all h similar to (5).

Consequently, the assumption that

and Equation (6), yields the fact that

where the second inequality holds from the fact that w ⋆ is feasible for ( P BR ). The latter concludes the proof, as the display coincides with the definition of an ϵ -approximate NE.

A.3 Proof of Claim 3.1

Proof. The value function of s 1 for h = 1 of players 1 and 2 read:

and,

We are indifferent to the corresponding value function of player 3 as they only have one available action per state and hence, cannot affect their rewards. For the joint policy σ , the corresponding value functions of both players 1 and 2 are V σ 1 , 1 ( s 1 ) = V σ 2 , 1 ( s 1 ) = 1 20 .

Deviations. We will now prove that no deviation of player 1 manages to accumulate a reward greater than 1 20 . The same follows for player 2 due to symmetry.

When a player deviates unilaterally from a joint policy, they experience a single agent Markov decision process (MDP). It is well-known that MDPs always have a deterministic optimal policy. As such, it suffices to check whether V π 1 , σ -1 1 , 1 ( s 1 ) is greater than 1 20 for any of the four possible deterministic policies:

Finally, the value function of any deviation π ′ 1 writes,

We can now check that for all deterministic policies V π ′ 1 × σ -1 1 , 1 ( s 1 ) ≤ 1 20 . By symmetry, it follows that V π ′ 2 × σ -2 2 , 1 ( s 1 ) ≤ 1 20 and as such σ is indeed a CCE.

A.4 Proof of Claim 3.2

Proof. In general, the value functions of each player 1 and 2 are:

and

Plugging in π σ 1 , π σ 2 yields V π σ 1 × π σ 2 1 , 1 ( s 1 ) = V π σ 1 × π σ 2 2 , 1 ( s 1 ) = -13 160 . But, if player 1 deviates to say π ′ 1 ( s 1 ) = π ′ 1 ( s 2 ) = (0 1) , they get a value equal to 0 which is clearly greater than -13 160 . Hence, π σ 1 × π σ 2 is not a NE.

A.5 Proof of Theorem 3.4

Proof. The proof follows from the game of Example 1, and Claims 3.1 and 3.2.

B Proofs for infinite-horizon Zero-Sum Polymatrix Markov Games

In this section we will explicitly state definitions, theorems and proofs relating to the infinite-horizon discounted zero-sum polymatrix Markov games.

B.1 Definitions of equilibria for the infinite-horizon

Let us restate the definition specifically for infinite-horizon Markov games. They are defined as a tuple Γ( H, S , {A k } k ∈ [ n ] , P , { r k } k ∈ [ n ] , γ, ρ ) .

H = ∞ denotes the time horizon

S , with cardinality S := |S| , stands for the state space,

{A k } k ∈ [ n ] is the collection of every player's action space, while A := A 1 × · · · × A n denotes the joint action space ; further, an element of that set -a joint action- is generally noted as a = ( a 1 , . . . , a n ) ∈ A ,

P : S × A → ∆( S ) is the transition probability function,

r k : S , A → [ -1 , 1] yields the reward of player k at a given state and joint action,

a discount factor 0 < γ < 1 ,

an initial state distribution ρ ∈ ∆( S ) .

Policies and value functions. In infinite-horizon Markov games policies can still be distinguished in two main ways, Markovian/non-Markovian and stationary/nonstationary . Moreover, a joint policy can be a correlated policy or a product policy.

Markovian policies attribute a probability over the simplex of actions solely depending on the running state s of the game. On the other hand, non-Markovian policies attribute a probability over the simplex of actions that depends on any subset of the history of the game. I.e. , they can depend on any sub-sequence of actions and states up until the running timestep of the horizon.

Stationary policies are those that will attribute the same probability distribution over the simplex of actions for every timestep of the horizon. Nonstationary policies, on the contrary can change depending on the timestep of the horizon.

A joint Markovian stationary policy σ is said to be correlated when for every state s ∈ S , attributes a probability distribution over the simplex of joint actions A for all players, i.e. , σ ( s ) ∈ ∆( A ) . A Markovian stationary policy π is said to be a product policy when for every s ∈ S , π ( s ) ∈ ∏ n k =1 ∆( A k ) . It is rather easy to define correlated/product policies for the case of non-Markovian and nonstationary policies.

Given a Markovian stationary policy π , the value function for an infinite-horizon discounted game is defined as,

It is possible to express the value function of each player k in the following way,

Where I is the identity matrix of appropriate dimensions. Also, when the initial state is drawn from the initial state distribution, we denote, the value function reads V π k ( ρ ) = ρ ⊤ ( I -γ P ( π )) -1 r ( π ) .

Best-response policies. Given an arbitrary joint policy σ (which can be either a correlated or product policy), a best-response policy of a player k is defined to be π † k ∈ ∆( A k ) S such that π † k ∈ arg max π ′ k V π ′ k × σ -k k ( s ) . Also, we will denote V † , σ -k k ( s ) = max π ′ k V π ′ k , σ -k k ( s ) . It is rather straightforward to see that the problem of computing a best-response to a given policy is equivalent to solving a single-agent MDP problem.

Notions of equilibria. Now that best-response policies have been defined, it is straightforward to define the different notions of equilibria. First, we define the notion of a coarse-correlated equilibrium. Definition B.1 (CCE-infinite-horizon) . A joint (potentially correlated) policy σ ∈ ∆( A ) S is an ϵ -approximate coarse-correlated equilibrium if it holds that for an ϵ ,

Second, we define the notion of a Nash equilibrium. The main difference of the definition of the coarse-correlated equilibrium, is the fact that a NE Markovian stationary policy is a product policy . Definition B.2 (NE-infinite-horizon) . A joint (potentially correlated) policy π ∈ ∏ k ∈ [ n ] ∆( A k ) S is an ϵ -approximate coarse-correlated equilibrium if it holds that for an ϵ ,

As it is folklore by now, infinite-horizon discounted Markov games have a stationary Markovian Nash equilibrium.

C Main results for infinite-horizon games

The workhorse of our arguments in the following results is still the following nonlinear program with variables π , w ,

As we will prove, approximate NE's correspond to approximate global minima of ( P ′ NE ) and viceversa. Before that, we need some intermediate lemmas. The first lemma we prove is about the best-response program.

The best-response program. Even for the infinite-horizon, we can define a linear program for the best-responses of all players. That program is the following, with variables w ,

Lemma C.1 (Best-response LP-infinite-horizon) . Let a (possibly correlated) joint policy ˆ σ . Consider the linear program ( P ′ BR ). The optimal solution w † of the program is unique and corresponds to the value function of each player k ∈ [ n ] when player k best-responds to ˆ σ .

Proof. We observe that the program is separable to n independent linear programs, each with variables w k ∈ R n ,

Each of these linear programs describes the problem of a single agent MDP -that agent being k . It follows that the optimal w † k for every program is unique (each program corresponds to a set of Bellman optimality equations).

Properties of the NE program. Second, we need to prove that the minimum value of the objective function of the program is nonnegative.

Lemma C.2 (Feasibility of ( P ′ NE ) and global optimum) . The nonlinear program ( P ′ NE ) is feasible, has a nonnegative objective value, and its global minimum is equal to 0 .

Proof. For the feasibility of the nonlinear program, we invoke the theorem of the existence of a Nash equilibrium. i.e. , let a NE product policy, π ⋆ , and a vector w ⋆ ∈ R n × H × S such that w ⋆ ( s ) = V † , π ⋆ -k ( s ) , ∀ k ∈ [ n ] ×S .

By Lemma C.1, we know that ( π ⋆ , w ⋆ ) satisfies all the constraints of ( P ′ NE ). Additionally, because π ⋆ is a NE, V π ⋆ k ( ρ ) = V † , π ⋆ -k k ( ρ ) for all k ∈ [ n ] . Observing that,

concludes the argument that a NE attains an objective value equal to 0 .

Continuing, we observe that due to (1) the objective function can be equivalently rewritten as,

Next, we focus on the inequality constraint

which holds for all s ∈ S , all players k ∈ [ n ] , and all a ∈ A k .

By summing over a ∈ A k while multiplying each term with a corresponding coefficient π k ( a | s ) , the display written in an equivalent element-wise vector inequality reads:

Finally, after consecutively substituting w k with the element-wise lesser term r k ( π ) + γ P ( π ) w k , we end up with the inequality:

We note that I + γ P ( π ) + γ 2 P 2 ( π ) + · · · = ( I -γ P ( π )) -1 .

Summing over k , it holds for the s 1 -th entry of the inequality,

Where the equality holds due to the zero-sum property, (1).

Theorem C.1 (NE and global optima of ( P ′ NE )-infinite-horizon) . If ( π ⋆ , w ⋆ ) yields an ϵ -approximate global minimum of ( P ′ NE ) , then π ⋆ is an nϵ -approximate NE of the infinite-horizon zero-sum polymatrix switching controller MG, Γ . Conversely, if π ⋆ is an ϵ -approximate NE of the MG Γ with corresponding value function vector w ⋆ such that w ⋆ k ( s ) = V π ⋆ k ( s ) ∀ ( k, s ) ∈ [ n ] ×S , then ( π ⋆ , w ⋆ ) attains an ϵ -approximate global minimum of ( P ′ NE ) .

Proof.

An approximate NE is an approximate global minimum. We show that an ϵ -approximate NE, π ⋆ , achieves an nϵ -approximate global minimum of the program. Utilizing Lemma C.1 by setting w ⋆ k = V † , π ⋆ -k ( ρ ) , feasibility , and the definition of an ϵ -approximate NE we see that,

or,

Indeed, this means that π ⋆ , w ⋆ is an nϵ -approximate global minimizer of ( P ′ NE ).

An approximate global minimum is an approximate NE. For this direction, we let a feasible ϵ -approximate global minimizer of the program ( P ′ NE ), ( π ⋆ , w ⋆ ) . Because a global minimum of the program is equal to 0 , an ϵ -approximate global optimum must be at most ϵ > 0 . We observe that for every k ∈ [ n ] ,

which follows from induction on the inequality constraint (9).

Consequently, the assumption that

and Equation (10), yields the fact that

where the second inequality holds from the fact that w ⋆ is also feasible for ( P ′ BR ). The latter concludes the proof, as the display coincides with the definition of an ϵ -approximate NE.

Theorem C.2 (CCE collapse to NE in polymatrix MG-infinite-horizon) . Let a zero-sum polymatrix switching-control Markov game, i.e., a Markov game for which Assumptions 1 and 2 hold. Further, let an ϵ -approximate CCE of that game σ . Then, the marginal product policy π σ , with π σ k ( a | s ) = ∑ a -k ∈A -k σ ( a, a -k ) , ∀ k ∈ [ n ] is an nϵ -approximate NE.

Proof. Let an ϵ -approximate CCE policy, σ , of game Γ . Moreover, let the best-response value-vectors of each agent k to joint policy σ -k , w † k .

Now, we observe that due to Assumption 1,

Further, due to Assumption 2,

Putting these pieces together, we reach the conclusion that ( π σ , w † ) is feasible for the nonlinear program ( P ′ NE ).

What is left is to prove that it is also an ϵ -approximate global minimum. Indeed, if ∑ k ρ ⊤ w † k ≤ ϵ (by assumption of an ϵ -approximate CCE), then the objective function of ( P ′ NE ) will attain an ϵ -approximate global minimum. In turn, due to Theorem C.1 the latter implies that π σ is an nϵ -approximate NE.

C.1 No equilibrium collapse with more than one controllers per-state

Example 2. We consider the following 3 -player Markov game that takes place for a time horizon H = 3 . There exist three states, s 1 , s 2 , and s 3 and the game starts at state s 1 . Player 3 has a single action in every state, while players 1 and 2 have two available actions { a 1 , a 2 } and { b 1 , b 2 } respectively in every state. The initial state distribution ρ is the uniform probability distribution over S .

Reward functions. If player 1 (respectively, player 2 ) takes action a 1 (resp., b 1 ), in either of the states s 1 or s 2 , they get a reward equal to 1 20 . In state s 3 , both players get a reward equal to -1 2 regardless of the action they select. Player 3 always gets a reward that is equal to the negative sum of the reward of the other two players. This way, the zero-sum polymatrix property of the game is ensured (Assumption 1).

Transition probabilities. If players 1 and 2 select the joint action ( a 1 , b 1 ) in state s 1 , the game will transition to state s 2 . In any other case, it will transition to state s 3 . The converse happens if in state s 2 they take joint action ( a 1 , b 1 ) ; the game will transition to state s 3 . For any other joint action, it will transition to state s 1 . From state s 3 , the game transition to state s 1 or s 2 uniformally at random.

At this point, it is important to notice that two players control the transition probability from one state to another. In other words, Assumption 2 does not hold.

s

1

s

2

s

3

1

/

2

1

/

2

1

-

π

1

(

a

1

|

s

1

)

π

2

(

b

1

|

s

1

)

π

1

(

a

1

|

s

1

)

π

2

(

b

1

|

s

1

)

π

1

(

a

1

|

s

2

)

π

2

(

b

1

|

s

2

)

1

-

π

1

(

a

1

|

s

2

)

π

2

(

b

1

|

s

2

)

Next, we consider the joint policy σ ,

Claim C.1. The joint policy σ that assigns probability 1 2 to the joint actions ( a 1 , b 2 ) and ( a 2 , b 1 ) in both states s 1 , s 2 is a CCE and V σ 1 ( ρ ) = V σ 2 ( ρ ) = -1 10 .

Proof.

We check every deviation,

π 1 ( s 1 ) = π 1 ( s 2 ) = (1 0) , V π 1 × σ -1 ( ρ ) = -2 5 ,

For every such deviation the value of player 1 is smaller than -1 10 . For player 2 , the same follows by symmetry. Hence, σ is indeed a CCE.

Yet, the marginalized product policy of σ which we note as π σ 1 × π σ 2 does not constitute a NE. The components of this policy are,

I.e., the product policy π σ 1 × π σ 2 selects any of the two actions of each player in states s 1 , s 2 independently and uniformally at random. With the following claim, it can be concluded that in general when more than one player control the transition the set of equilibria do not collapse.

Claim C.2. The product policy π σ 1 × π σ 2 is not a NE.

Proof. For π σ = π σ 1 × π σ 2 we get,

But, for the deviation π 1 ( a 1 | s 1 ) = π 1 ( a 1 | s 2 ) = 0 , the value funciton of player 1 , is equal to -1 6 . Hence, π σ is not a NE.

In conclusion, Assumption 1 does not suffice to ensure equilibrium collapse.

Theorem C.3 (No collapse-infinite-horizon) . There exists a zero-sum polymatrix Markov game (Assumption 2 is not satisfied) that has a CCE which does not collapse to a NE.

Proof. The proof follows from the game of Example 2, and Claims C.1 and C.2.

D An algorithm for approximating Markovian CCE

In this section we describe the algorithm in (Daskalakis et al., 2022) used for the computation a of Markovian CCE. We note that M,N visit , p are parameters that affect the accuracy of the approximation and we kindly ask the reader to refer to (Daskalakis et al., 2022) for further details. This algorithm computes an ϵ -approximate CCE in time ˜ O ( ϵ -3 ) . Newer works have improved the dependence on ϵ to ˜ O ( ϵ -2 ) , see (Wang et al., 2023; Cui et al., 2023).

As soon as an ϵ -approximate CCE is computed, what is left to do is to compute the marginal policy of every player, π σ k,h ( a | s ) = ∑ a -k ∈A -k σ ( a, a -k | s ) .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCai (2016) proved that in normal-form polymatrix games the set of CCEs corresponds with the set of NEs. This paper provides a similar result for polymatrix, switching controller, Markov games.\n\nAfter introducing the formalism used throughout the paper, the main result is presented for both fixed horizon and infinite horizon. A counter example is also offered regarding the necessity of the switching controller condition.\n\nThe crucial aspect of the paper from a theoretical perspective lies in the fact that the polymatrix + switching controller assumptions allow to switch from a correlated distribution to a product distribution in the formulation of a CCE, thus proving the equivalence to a NE.\n\nSTRENGTHS:\n- Clear step by step structure of the paper\n- Solid and intuitive proofs\n\nWEAKNESSES:\nThe clarity of the paper can be improved in multiple points:\n1. Sometimes the notation is confusing and feels heavy\n\t- the $\\cdot^\\dagger$ for best responses is unintuitive and does not feel natural\n\t- appendix A.1 is missing a cartesian product $\\times$ in the Best responses symbols (after line 525)\n\t- in many points of the paper, some symbols are silently redefined to avoid explicit expectation terms, like $r_{k,h}(s,a,b)$ suddenly accepting a probability distribution as in $r_{k,h}(s, \\pi, b)$. At a first glance, this formalism bugged me. Having those shortcuts defined in line 167-169 would help in avoiding surprises in the formalism.\n 2. The definition of the linear program $P_{NE}$ would be easier to comprehend if the program was accompanied by a short textual description of the variables' meaning and of the constraints. This will greatly improve the smoothness of the read and ease the comprehension of the proofs.\n 3. The relation between CCEs and Best response policies defined as product distribution should be explicitly explained to allow the connection between the definition of the coarse correlated and its practical meaning (i.e. CCEs are stable to policy deviations which renounce to see the correlation signal). It is to be noted that at the moment there is no proper intuition behind the idea of a CCE.\n\nQUESTIONS:\nAfter carefully reading the paper, I have some questions/comments regarding specific passages of the paper:\n1. I suggest to move lines 167-169 *before* the use of the defined symbols\n2. Why is the warm-up in section 3.1 included in the body of the paper? I struggle to see its usefulness to make the whole paper clearer\n3. Is $w_k^\\dagger$ missing a $\\cdot_h$ subscript in line 255?\n4. is the sole purpose if including the subtraction of the expected value in the objective function of $P_{NE}$ to have a more comfortable global minimum in 0? (i.e. it simplifies the following statements)\n5. Is the first constraint of program $P'_{NE}$ missing a $\\gamma$ term?\n\nMoreover, I'd like the authors to properly address the weaknesses from the previous sections.\n\nOn a side note, I highlight some of the typos: \n- Shapely citation in line 18 seems to have the wrong format\n- Unfinished sentence on line 72-73\n\nRegarding the novelty of the technical approach and the relevance of the result, I think that those are good in the present paper, but I cannot evaluate them with high confidence as I work in a different yet related subfield. On the other hand, I checked the proofs in the main body and in the appendices A.1 and A.2 and I found them both clear and correct.\n\nLIMITATIONS:\nThe main limitation of the paper is that specific assumptions have to be made on the reward structure (polymatrix) and transition function (switching control). These are clearly addressed and explicated in the paper. However, I do not agree with the authors regarding the practical real-world importance of polymatrix games (Section 1.1)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper defines a class of mulit-agent Markov games called zero-sum polymatrix Markov games, which is a generalization of the zero-sum polymatrix normal-form games. Specifically, it defines a class of Markov games where each state is a zero-sum polymatrix game. The main results of the paper shows that: in both the finite-horizon and the infinite-horizon setting (1) when the switching control assumption holds, i.e., the transition is determined by one player on each state, then the marginal policy of any (approximate) coerce correlated equilibrium (CCE) is an (approximate) Nash equilibrium (NE). This equilibrium collapse results ensures efficient computation of NE by reduction to computation of CCE. (2) when the transition is controlled by more than two players, equilibrium collapse does not hold.\n\nSTRENGTHS:\n1. This paper introduces an interesting class of Markov games where efficient computation of NE is possible, which covers switching control zero-sum Markov games and zero-sum polymatrix games as special cases. The results contribute to a better understanding of equilibrium computation in Markov games. \n2. This paper is fairly well-written and easy to follow. The results are complete in the sense that the authors showed equilibrium collapse with the swtiching control assumption and also provided counterexample when the assumption does not hold.\n\nWEAKNESSES:\n1. The swtiching control assumption is kind of strong and limits the significance of the results. This paper would be much stronger with a more general sufficient condition of efficient computation of NE in zero-sum polymatrix Markov game. Currently, computation among  zero-sum polymatrix Markov game is rather unclear. \n\nSome minor comments:\n\n1. Check the notation of $V_{k,h}^{\\pi}$ in line 161, 164 and some other places. As the text suggests, it means the cumulative reward of player $k$ after timestep $h$ but what is written refers to $V^{\\pi}_{k,1}$? \n2. Adding a formula might be helpful for readers to understand Assumption 2 \n3. Line 324: \"can (be) modelled\"\n\nQUESTIONS:\n1. Is it possible to design *decentralized* algorithm with convergence to NE in switch-control zero-sum polymatrix Markov games?\n2. Zero-sum polymatrix normal-form games belong to the more general class of monotone games. Do the authors have any insights on the possiblity of generalizing monotone games to monotone Markov games and also ensure efficient computation of NE? \n3. Does the set of correlated equiliria (CE) collapses to NE in zero-sum polymatrix Markov games without the swtiching control assumption?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of computing approximate Nash equilibria in Markov (aka stochastic) games. It is known that such equilibria can be computed efficiently in the non-stochastic setting when the game is zero-sum polymatrix. The authors show that approximate equilibria can also be computed efficiently in zero-sum polymatrix Markov games, when the game has the switching controller property (i.e., a single, but possibly different, agent influences the transition at each step).\n\nThe result is proved by showing that the coarse-correlated equilibria (CCE) collapse to Nash equilibria in this setting. As a result, it suffices to compute a CCE, which can be done efficiently using prior work. The same approach of equilibrium collapse was also used by Cai et al. (2016) in the non-stochastic setting. The authors also show that equilibrium collapse fails to hold if the switching controller assumption is removed.\n\nSTRENGTHS:\n- the paper studies a very natural computational problem\n- the results are obtained by a non-trivial generalization of the techniques used by Cai et al. (2016)\n\nWEAKNESSES:\n- the writing is quite sloppy in some parts\n\nQUESTIONS:\nQuestions:\n\n1. In Corollary 3.2 you say that you can only compute a nonstationary equilibrium. Can you comment on this? Why is that the case? Is the problem for stationary equilibria open? Are there some obstacles?\n\n2. Your results yield a polynomial time algorithm for the case where epsilon is inverse polynomial  in the other parameters of the problem. What about the case where epsilon is inverse exponential? Is there a hardness result known for this case, or is this an open question? In other words, is it known whether an algorithm with running time poly(log(1/epsilon)) should be possible?\n\n\nFurther Comments:\n\n- it would be nice to explain in the introduction why the zero-sum polymatrix setting captures both competition and coordination\n\n- lines 539-541: it looks like something is missing here\n\n- lines 558-559, Lemma A.2: Do you mean P_NE here? (the finite horizon case)\n\n- lines 726-736, Proof of Theorem C.2: It looks like gamma is missing in the equations in this proof\n\n- line 125: here you probably also want to allow H = \\infty\n\n- line 161: there seem to be some typos in the equation below this line. \"h\" is used both as a parameter of the quantity that is defined here, and also on the right-hand side as an index for the sum. The same applies below as well.\n\n- line 205: the notation \"A_{argctrl(s)}\" is informal. The co-domain of the function cannot depend on an input of the function.\n\n- line 216: I think the O(epsilon) notation should not be used in the definition here. Please expand and use quantifiers.\n\n- line 245: It would be nice to add some detail about which parts of the program are linear and which are not.\n\n- Figure 1: There are some typos in this figure. Please check that the edge labels are correct.\n\n- line 308: There are many typos in P'_NE. The version in the appendix is correct.\n\n- line 321: what is H? It seems like there should be no H in the infinite horizon case. Also shouldn't gamma appear in the bound somewhere? I assume that H is the expression on line 317, but this is not a parameter of the problem, so you replace the dependency on H by a dependency on 1/(1-gamma) in the bound.\n\n- line 342-343: what is the exact dependence on epsilon in your algorithm?\n\n\nTypos:\n- line 59: remove \"that\"\n- lines 72-73: sentence not finished\n- line 175: remove \"a an\" -> \"an\"\n- line 215: defined -> be defined\n- line 289+299: uniformally -> uniformly\n- line 318: sentence not finished\n- line 321: proabiblity\n- line 324: modelled -> be modelled\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores a specific class of Markov games called zero-sum polymatrix Markov games, a stochastic generalization of one-shot polymatrix Markov games, and shows how an ϵ-approximate Nash equilibrium can be efficiently computed. The authors show equality between the set of coarse-correlated equilibria and Nash equilibria for a subset of  these games with switching control. This implies that Nash equilibria in switching control zero-sum polymatrix Markov games can be computed efficiently. The paper also discusses open questions related to using a policy optimization algorithm to converge to an approximate Nash equilibrium. Overall, the paper provides a theoretical framework for reasoning about strategic interactions over dynamically changing networks.\n\nSTRENGTHS:\nThe paper tackles a well-defined question, which is of interest to the community and which I would also find interesting. Overall the authors seem precise in their analysis. Writing is also often clear and and results seem reproducible.\n\nWEAKNESSES:\nThe paper is not entirely self-contained (for instance when it comes to the references to Daslakakis et al’s algorithm), and lacks important intuition for the programs provided and theorem proofs. Given that the authors seems use traditional tools to obtain their results, and the paper is entirely theoretical I would expect a much more thorough explanation of theoretical result and would have liked a discussion of the algorithm introduced by Daslakakis et al. since the paper is recent and the reader might not be familiar with it.\n\nFor a purely theoretical paper, I would expect more intuition to be provided and more explanations of results. As it is, the reader has to spend its entire time decoding the results without any help from the authors. To be entirely clear, I think that the results, although they seem to rely on standard theoretical tools, are interesting and valuable to the community but the paper would be in much better shape with either experimental evaluation or with additional explanations. I remain open to changing my score and would appreciate it if the authors provide intuition and explanation of their results to this end.\n\nQUESTIONS:\nIf polymatrix games are solvable in polytime, even in the one-shot setting, why aren’t general-sum polymatrix games solvable? Can’t I just add for any polymatrix game an additional dummy player to make the game zero-sum without changing the equilibria?\n\nCan you build a regular game from a switching controller game by adding for each state a number of states equal to the number of players and where players get no rewards, and make the game go through these additional states for each individual state in the regular game so as to simulate more complicated transitions?\n\nIs the set of CCE, and consequently the set of NE, in a zero-sum polymatrix Markov game convex?\n\nThere is no explanation of why the set of CCE and NE coincide, why does this happen? The proof does not give me much intuition, can you describe in a few sentences why the “collapse” happens?\n\nI am not sure I understand Claim A1, how can we known that the correlated policy can be marginalized such that pi = pi_1 x pi_2. In general, it should be pi = pi_1|pi_2 x pi_2. What am I missing?\n\nCan you give an inuitive explanation of the program P_{NE}. Is the program convex? I assume not, if so, is it incave/satisfy a gradient dominance condition? What is the Algorithm given by Daslakakis et al. and why does it work here? \n\nNotes: Definition of value function (line 161): variable h overloaded on lhs and rhs\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}