SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chanwoo Park
cpark97@mit.edu
Kaiqing Zhang
kaiqing@umd.edu
Asuman Ozdaglar
asuman@mit.edu
We study a new class of Markov games, (multi-player) zero-sum Markov Games with Networked separable interactions (zero-sum NMGs), to model the local interaction structure in non-cooperative multi-agent sequential decision-making. We define a zero-sum NMG as a model where the payoffs of the auxiliary games associated with each state are zero-sum and have some separable (i.e., polymatrix) structure across the neighbors over some interaction network. We first identify the necessary and sufficient conditions under which an MG can be presented as a zero-sum NMG, and show that the set of Markov coarse correlated equilibrium (CCE) collapses to the set of Markov Nash equilibrium (NE) in these games, in that the product of per-state marginalization of the former for all players yields the latter. Furthermore, we show that finding approximate Markov stationary CCE in infinite-horizon discounted zero-sum NMGs is PPAD-hard, unless the underlying network has a "star topology". Then, we propose fictitious-play-type dynamics, the classical learning dynamics in normal-form games, for zero-sum NMGs, and establish convergence guarantees to Markov stationary NE under a star-shaped network structure. Finally, in light of the hardness result, we focus on computing a Markov non-stationary NE and provide finite-iteration guarantees for a series of value-iteration-based algorithms. We also provide numerical experiments to corroborate our theoretical results.
Nash Equilibrium (NE) has been broadly used as a solution concept in game theory, since the seminal works of [1, 2]. Perhaps equally important, NE is also deeply rooted in the prediction and analysis of learning dynamics in multi-agent strategic environments: it may appear as a natural outcome of many non-equilibrating learning processes of multiple agents interacting with each other [3, 4]. A prominent example of such learning processes is fictitious play (FP) [5, 6], in which myopic agents estimate the opponents' play using history, and then choose a best-response action (based on their payoff matrix) against this estimate, as if the opponents use it as their stationary strategy. The focus of these studies has initially been on the convergence to NE in zero-sum games (see [5, 6], and also [7, 4]), and in games with aligned objective (identical-interest and potential games, see [8]). Since then, FP has been shown to converge to NE in more important classes of games, including  $2 \times n$  games [9, 10], "one-against-all" games [11], and zero-sum polymatrix games [12], justifying the prediction power of NE in learning in normal-form/matrix games.
Some of these results have recently been extended to the stochastic game (also known as the Markov game (MG)) setting, a model for multi-agent sequential decision-making with state transition dynamics, first introduced in [13]. In particular, [14, 15, 16, 17, 18] have studied best-response-type learning dynamics in two-player zero-sum MGs, and [19, 17] have studied that in multi-player
$\mathfrak{h}$  Alphabetical order.
identical-interest games. Following the same path as studying matrix games, one natural question arises: Are there other classes of MGs beyond two-player zero-sum and identical-interest cases that allow natural learning dynamics, e.g., fictitious play, to justify NE as the long-run emerging outcome?
On the other end of the spectrum, it is well-known that for general-sum normal-form games, the special case of MGs without the state transition dynamics, computing an NE is intractable [20, 21]. Relaxed solution concepts as (coarse) correlated equilibrium ((C)CE) have thus been favored when it comes to equilibrium computation for general-sum, multi-player games [22, 3]. Encouragingly, when the interactions among players have some networked separable structure, also known as being polymatrix, computing NE may be made tractable even in multi-player settings. This has been instantiated in the seminal works [23, 24] in the normal-form game setting, which showed that any CCE collapses to the NE in such games when the payoffs are zero-sum. Thus, any algorithms that can efficiently compute the CCE in such games will lead to the efficient computation of the NE.
In fact, besides being of theoretical interest, multi-player zero-sum games with networked separable interactions also find a range of applications, including security games [24], fashion games [25, 26, 27], and resource allocation problems [28]. These examples, oftentimes, naturally involve some state transition that captures the dynamics of the evolution of the environment in practice. For example, in the security game, the protection level or immunity of a target increases as a function of the number of past attacks, leading to a smaller probability of an attack on the target being successful. Hence, it is imperative to study such multi-player zero-sum games with state transitions. As eluded in the recent results [29, 30], such a transition from stateless to stateful cases may not always yield straightforward and expected results, e.g., computing stationary CCE can be computationally intractable in stochastic games, in stark contrast to the normal-form case where CCE can be efficiently computed. This naturally prompts another question: Are there other types of (multi-player) MGs that may circumvent the computational hardness of computing NE/CCE?
In an effort to address these two questions, we introduce a new class of Markov games - (multi-player) zero-sum Markov games with networked separable interactions (zero-sum NMGs). We summarize our contributions as follows, and defer a more detailed literature review to Appendix A.
Contributions. First, we introduce a new class of non-cooperative Markov games: (multi-player) zero-sum MGs with Networked separable interactions (zero-sum NMGs), wherein the payoffs of the auxiliary-games associated with each state, i.e., the sum of instantaneous reward and expectation of any estimated state-value functions, possess the multi-player zero-sum and networked separable (i.e., polymatrix) structure as in [31, 28, 23, 24] for normal-form games, a strict generalization of the latter. We also provide structural results on the reward and transition dynamics of the game, as well as examples of this class of games. Specifically, for a Markov game to qualify as a zero-sum NMG, if and only if its reward function has the zero-sum polymatrix structure, and its transition dynamics is an ensemble of multiple single-controller transition dynamics that are sampled randomly at each state (see Remark 3 for more details). This transition dynamics covers the common ones in the MG literature, including the single-controller and turn-based dynamics. Second, we show that Markov CCE and Markov NE collapse in that the product of per-state marginal distributions of the former yields the latter, making it sufficient to focus on the former in equilibrium computation. We then show the PPAD-hardness [32] of computing the Markov stationary equilibrium, a natural solution concept in infinite-horizon discounted MGs, unless the underlying network has a star-topology. This is in contrast to the normal-form case where CCE is always computationally tractable. Third, we study the fictitious-play property [8] of zero-sum NMGs, showing that the fictitious-play dynamics [16, 19] converges to the Markov stationary NE, for zero-sum NMGs with a star-shaped network structure. Finally, in light of the hardness of computing stationary equilibria, we develop a series of value-iteration-based algorithms for computing a Markov non-stationary NE of zero-sum NMGs, with finite-iteration guarantees. We also provide numerical experiments to corroborate our theoretical results in Section G. We hope our results serve as a starting point for studying this networked separable interaction structure in non-cooperative Markov games.
Notation. For a real number  $c$ , we use  $(c)_+$  to denote  $\max \{c, 0\}$ . For an event  $\mathcal{E}$ , we use  $\mathbf{1}(\mathcal{E})$  to denote the indicator function such that  $\mathbf{1}(\mathcal{E}) = 1$  if  $\mathcal{E}$  is true, and  $\mathbf{1}(\mathcal{E}) = 0$  otherwise. We define multinomial distribution with probability  $(w_i)_{i \in \mathcal{N}}$  as Multinomial  $\left((w_i)_{i \in \mathcal{N}}\right)$ . We denote the uniform distribution over a set  $S$  as  $\mathrm{Unif}(S)$ . We denote the Bernoulli distribution with probability  $p$  as  $\mathrm{Bern}(p)$ . The sgn function is defined as  $\mathrm{sgn}(x) = 2 \times \mathbf{1}(x \geq 0) - 1$ . The KL-divergence between two probability distributions  $p, q$  is denoted as  $\mathrm{KL}(p, q) = \mathbb{E}_p[\log(p / q)]$ . For a graph  $\mathcal{G} = (\mathcal{N}, \mathcal{E})$ , we denote the set of neighboring nodes of node  $i \in \mathcal{N}$  as  $\mathcal{E}_i$  (without including  $i$ ). The maximum norm of a matrix  $X \in \mathbb{R}^{m \times n}$ , denoted as  $\|X\|_{\max}$ , is defined as  $\|X\|_{\max} := \max_{i \in [m], j \in [n]} |X_{i,j}|$ .
2.1 Markov games
We define a Markov game as a tuple  $(\mathcal{N},\mathcal{S},\mathcal{A},H,(\mathbb{P}_h)_{h\in [H]},(r_{h,i})_{h\in [H],i\in \mathcal{N}},\gamma)$ , where  $\mathcal{N} = [n]$  is the set of players,  $\mathcal{S}$  is the state space with  $|\mathcal{S}| = S$ ,  $\mathcal{A}_i$  is the action space for player  $i$  with  $|\mathcal{A}_i| = A_i$  and  $\mathcal{A} = \prod_{i\in \mathcal{N}}\mathcal{A}_i$ ,  $H\leq \infty$  is the length of the horizon,  $\mathbb{P}_h:\mathcal{S}\times \mathcal{A}\to \Delta (\mathcal{S})$  captures the state transition dynamics at timestep  $h$ ,  $r_{h,i}\in [0,R]$  is the reward function for player  $i$  at timestep  $h$ , bounded by some  $R > 0$ , and  $\gamma \in (0,1]$  is a discount factor. An MG with a finite horizon  $(H < \infty)$  is also referred to as an episodic MG, while an MG with an infinite horizon  $(H = \infty)$  and  $\gamma < 1$  is referred to as an infinite-horizon  $\gamma$ -discounted MG. When  $H = \infty$ , we will consider the transition dynamics and reward functions, denoted by  $\mathbb{P}$  and  $(r_i)_{i\in \mathcal{N}}$ , respectively, to be independent of  $h$ . Hereafter, we may use agent and player interchangeably.
Policy. Consider the stochastic Markov policy for player  $i$ , denoted by  $\pi_{i}$ , as  $\pi_{i} := \{\pi_{h,i} : S \to \Delta(\mathcal{A}_{i})\}_{h \in [H]}$ . A joint Markov policy is a policy  $\pi := \{\pi_{h} : S \to \Delta(\mathcal{A})\}_{h \in [H]}$ , where  $\pi_{h} : S \to \Delta(\mathcal{A})$  decides the joint action of all players that can be potentially correlated. A joint Markov policy is a product Markov policy if  $\pi_{h} : S \to \prod_{i \in \mathcal{N}} \Delta(\mathcal{A}_{i})$  for all  $h \in [H]$ , and is denoted as  $\pi = \pi_{1} \times \pi_{2} \times \dots \times \pi_{n}$ . When the policy is independent of  $h$ , the policy is called a stationary policy. We let  $\pmb{a}_{h}$  denote the joint action of all agents at timestep  $h$ . Unless otherwise noted, we will work with Markov policies throughout. We denote  $\pi(s) \in \Delta(\mathcal{A})$  as the joint policy at state  $s \in S$ .
Value function. For player  $i$ , the value function under joint policy  $\pi$ , at timestep  $h$  and state  $s_h$  is defined as  $V_{h,i}^{\pi}(s_h) := \mathbb{E}_{\pi}\left[\sum_{h' = h}^{H}\gamma^{h' - h}r_{h',i}(s_{h'},\pmb{a}_{h'})\mid s_h\right]$ , which denotes the expected cumulative reward for player  $i$  at step  $h$  if all players adhere to policy  $\pi$ . We also define  $V_{h,i}^{\pi}(\rho) := \mathbb{E}_{s_h\sim \rho}[V_{h,i}^{\pi}(s_h)]$  for some state distribution  $\rho \in \Delta (\mathcal{S})$ . We denote the  $Q$ -function for the  $i$ -th player under policy  $\pi$ , at step  $h$  and state  $s_h$  as  $Q_{h,i}^{\pi}(s_h,\pmb{a}_h) := \mathbb{E}_{\pi}\left[\sum_{h' = h}^{H}\gamma^{h' - h}r_{h',i}(s_{h'},\pmb{a}_{h'})\mid s_h,\pmb{a}_h\right]$ , which determines the expected cumulative reward for the  $i$ -th player at step  $h$ , when starting from the state-action pair  $(s_h,\pmb{a}_h)$ . For the infinite-horizon discounted setting, we also use  $V_{i}^{\pi}$  and  $Q_{i}^{\pi}$  to denote  $V_{1,i}^{\pi}$  and  $Q_{1,i}^{\pi}$  for short, respectively.
Approximate equilibrium. Define an  $\epsilon$ -approximate Markov perfect Nash equilibrium as a product policy  $\pi$ , which satisfies  $\max_{i\in \mathcal{N}}\max_{\mu_i\in (\Delta (\mathcal{A}_i))|S|\times H}(V_{h,i}^{\mu_i,\pi_{-i}}(\rho) - V_{h,i}^{\pi}(\rho))\leq \epsilon$  for all  $\rho \in \Delta (\mathcal{S})$  and  $h\in [H]$ , where  $\pi_{-i}$  represents the marginalized policy of all players except player  $i$ . Define an  $\epsilon$ -approximate Markov coarse correlated equilibrium as a joint policy  $\pi$ , which satisfies  $\max_{i\in \mathcal{N}}\max_{\mu_i\in (\Delta (\mathcal{A}_i))|S|\times H}(V_{h,i}^{\mu_i,\pi_{-i}}(\rho) - V_{h,i}^{\pi}(\rho))\leq \epsilon$  for all  $\rho \in \Delta (\mathcal{S})$  and  $\bar{h}\in [H]$ . In the infinite-horizon setting, they can be equivalently defined as satisfying  $\max_{s\in S}\max_{i\in \mathcal{N}}\max_{\mu_i\in (\Delta (\mathcal{A}_i))|S|}(V_i^{\mu_i,\pi_{-i}}(s) - V_i^{\pi}(s))\leq \epsilon$ . If the above conditions only hold for certain  $\rho$  and  $h = 1$ , we refer to them as Markov non-perfect NE and CCE, respectively. Unless otherwise noted, we hereafter focus on Markov perfect equilibria, and sometimes refer to them simply as Markov equilibria when it is clear from the context. In the infinite-horizon setting, if additionally, the policy is stationary, then they are referred to as a Markov stationary NE and CCE, respectively.
2.2 Multi-player zero-sum games with networked separable interactions
As a generalization of two-player zero-sum matrix games, (multi-player) zero-sum polymatrix games have been introduced in [31, 28, 23, 24]. A polymatrix game, also known as a separable network game is defined by a tuple  $(\mathcal{G} = (\mathcal{N},\mathcal{E}_r),\mathcal{A} = \prod_{i\in \mathcal{N}}\mathcal{A}_i,(r_{i,j})_{(i,j)\in \mathcal{E}_r})$ . Here,  $\mathcal{G}$  is an undirected connected graph where  $\mathcal{N} = [n]$  denotes the set of players and  $\mathcal{E}_r\subseteq \mathcal{N}\times \mathcal{N}$  denotes the set of edges describing the rewards' networked structures, where the graph neighborhoods represent the interactions among players. For each edge, a two-player game is defined for players  $i$  and  $j$ , with action sets  $\mathcal{A}_i$  and  $\mathcal{A}_j$ , and reward functions  $r_{i,j}:\mathcal{A}_i\times \mathcal{A}_j\to \mathbb{R}$ , and similarly for  $r_{j,i}$ . The reward for player  $i$  for a given joint action  $\pmb {a} = (a_i)_{i\in \mathcal{N}}\in \prod_{i\in \mathcal{N}}\mathcal{A}_i$  is calculated as the sum of the rewards for all edges involving player  $i$ , that is,  $r_i(\pmb {a}) = \sum_{j:(i,j)\in \mathcal{E}_r}r_{i,j}(a_i,a_j)$ . To be consistent with our terminology later, hereafter, we also refer to such games as (multi-player) Games with Networked separable interactions (NGs).
In a zero-sum polymatrix game (i.e., a (multi-player) zero-sum Game with Networked separable interactions (zero-sum NG)), the sum of rewards for all players at any joint action  $\pmb{a} = (a_{i})_{i\in \mathcal{N}}\in$ $\prod_{i\in \mathcal{N}}\mathcal{A}_i$  equals zero, i.e.,  $\sum_{i\in \mathcal{N}}r_i(\pmb {a}) = 0$ . One can define the policy of agent  $i$ , i.e.,  $\pi_i\in \Delta (\mathcal{A}_i)$  so that the agent takes actions by sampling  $a_{i}\sim \pi_{i}(\cdot)$ . Note that  $\pi_{i}$  can be viewed as the reduced case of the policy defined in Section 2.1 when  $S = \emptyset$  and  $H = 1$ . The expected reward for player  $i$
under  $\pi$  can then be computed as:
$$
r _ {i} (\pi) := \sum_ {j: (i, j) \in \mathcal {E} _ {r}} \sum_ {a _ {i} \in \mathcal {A} _ {i}, a _ {j} \in \mathcal {A} _ {j}} r _ {i, j} \left(a _ {i}, a _ {j}\right) \pi_ {i} \left(a _ {i}\right) \pi_ {j} \left(a _ {j}\right) = \pi_ {i} ^ {\intercal} \boldsymbol {r} _ {i} \boldsymbol {\pi}, \tag {1}
$$
where  $\pmb{r}_i$  denotes the matrix  $\pmb{r}_i\coloneqq (r_{i,1},\dots ,r_{i,(i - 1)},0,r_{i,(i + 1)},\dots ,r_{i,n})\in \mathbb{R}^{|\mathcal{A}_i|\times \sum_{i\in \mathcal{N}}|\mathcal{A}_i|}$  and  $\pmb{\pi}\coloneqq (\pi_1^{\intercal},\pi_2^{\intercal},\ldots ,\pi_n^{\intercal})^{\intercal}\in \mathbb{R}^{\sum_{i\in \mathcal{N}}|\mathcal{A}_i|}$ . We define  $\pmb {r}\coloneqq (\pmb {r}_1^{\intercal},\pmb {r}_2^{\intercal},\dots ,\pmb {r}_n^{\intercal})^{\intercal}\in \mathbb{R}^{\sum_{i\in \mathcal{N}}|\mathcal{A}_i|\times \sum_{i\in \mathcal{N}}|\mathcal{A}_i|}$ . Then in this case we have  $\sum_{i\in \mathcal{N}}r_i(\pi) = 0$  for any policy  $\pi$ . See more prominent application examples of zero-sum polymatrix games in [23, 24].
3 Multi-Player (Zero-Sum) MGs with Networked Separable Interactions
We now introduce our model of multi-player zero-sum MGs with networked separable interactions.
Definition 1. An infinite-horizon  $\gamma$ -discounted MG is called a (multi-player) MG with Networked separable interactions (NMG) characterized by a tuple  $(\mathcal{G} = (\mathcal{N}, \mathcal{E}_Q), \mathcal{S}, \mathcal{A}, \mathbb{P}, (r_i)_{i \in \mathcal{N}}, \gamma)$  if for any function  $V: \mathcal{S} \to \mathbb{R}$ , defining  $Q_i^V(s, \boldsymbol{a}) := r_i(s, \boldsymbol{a}) + \gamma \sum_{s' \in \mathcal{S}} \mathbb{P}(s' | s, \boldsymbol{a}) V(s')$ , there exist a set of functions  $(Q_{i,j}^V)_{(i,j) \in \mathcal{E}_Q}$  and an undirected connected graph  $\mathcal{G} = (\mathcal{N}, \mathcal{E}_Q)$  such that  $Q_i^V(s, \boldsymbol{a}) = \sum_{j \in \mathcal{E}_{Q,i}} Q_{i,j}^V(s, a_i, a_j)$  holds for every  $i \in \mathcal{N}$ ,  $s \in \mathcal{S}$ ,  $\boldsymbol{a} \in \mathcal{A}$ , where  $\mathcal{E}_{Q,i}$  denotes the neighbors of player  $i$  induced by the edge set  $\mathcal{E}_Q$  (without including  $i$ ). When it is clear from the context, we represent the NMG tuple simply as  $\mathcal{G} = (\mathcal{N}, \mathcal{E}_Q)$ .
A finite-horizon MG is called a (multi-player) MG with Networked separable interactions if for any set of functions  $V \coloneqq \{V_h\}_{h \in [H + 1]}$  where  $V_h : \mathcal{S} \to \mathbb{R}$ , defining  $Q_{h,i}^V(s, \boldsymbol{a}) \coloneqq r_{h,i}(s, \boldsymbol{a}) + \gamma \sum_{s' \in \mathcal{S}} \mathbb{P}_h(s' \mid s, \boldsymbol{a}) V_{h + 1}(s')$ , there exist a set of functions  $(Q_{h,i,j}^V)_{(i,j) \in \mathcal{E}_Q, h \in [H]}$  such that  $Q_{h,i}^V(s, \boldsymbol{a}) = \sum_{j \in \mathcal{E}_{Q,i}} Q_{h,i,j}^V(s, a_i, a_j)$  holds for every  $i \in \mathcal{N}$ ,  $s \in \mathcal{S}$ ,  $h \in [H]$ ,  $\boldsymbol{a} \in \mathcal{A}$ .
A (multi-player) NMG is called a (multi-player) zero-sum MG with Networked separable interactions (zero-sum NMG) if additionally  $(\mathcal{G} = (\mathcal{N},\mathcal{E}_Q),\mathcal{A} = \prod_{i\in \mathcal{N}}\mathcal{A}_i,(r_{i,j}(s):= Q_{i,j}^{\mathbf{0}}(s))_{(i,j)\in \mathcal{E}_Q})$  forms a zero-sum NG for all  $s\in S$  in the infinite-horizon  $\gamma$ -discounted case, or  $(\mathcal{G} = (\mathcal{N},\mathcal{E}_Q),\mathcal{A} = \prod_{i\in \mathcal{N}}\mathcal{A}_i,(r_{h,i,j}(s):= Q_{h,i,j}^{\mathbf{0}}(s))_{(i,j)\in \mathcal{E}_Q})$  forms a zero-sum NG for all  $s\in S$  and  $h\in [H]$  in the finite-horizon case.
Regarding the assumption that the above conditions hold under any (set of) functions  $V$ , one may understand this as a structural requirement to inherit the polymatrix structure in the Markov game case. It is natural since  $\{Q_i^V\}_{i \in \mathcal{N}}$  would play the role of the payoff matrix in the normal-form case, when value-(iteration) based algorithms are used to solve the MG. As our hope is to exploit the networked structure in the payoff matrices to develop efficient algorithms for solving such MGs, if we do not know a priori which value function estimate  $V$  will be encountered in the algorithm update, the networked structure may easily break if we do not assume them to hold for all possible  $V$ . Moreover, such a definition easily encompasses the normal-form case, by preserving the polymatrix structure of the reward functions (when substituting  $V$  to be a zero function). Some alternative definition (see Remark 2) may not necessarily preserve the polymatrix structure of even the reward functions in a consistent way (see Section B.4 for a concrete example). We thus focus on Definition 1, which at least covers the polymatrix structure of the reduced case regarding only reward functions.
Indeed, such a networked structure in Markov games may be fragile. We now propose both sufficient and necessary conditions for the reward function's structure and the transition dynamics of the MG, to be an NMG. Here we focus on the infinite-horizon discounted setting for a simpler exposition. For finite-horizon cases, a similar statement holds, which is deferred to Appendix B. We also defer the full statement and proof of the following result to Appendix B. We first introduce the definition of decomposability and the set  $\mathcal{N}_C$ , which will be used in establishing the conditions. For a graph  $\mathcal{G} = (\mathcal{N},\mathcal{E})$ , we define  $\mathcal{N}_C\coloneqq \{i\mid (i,j)\in \mathcal{E}$  for all  $j\in \mathcal{N}\}$ , which may be an empty set if no such node  $i$  exists.
Definition 2 (Decomposability). A non-negative function  $f: X^{|\mathcal{D}|} \to \mathbb{R}^+ \cup \{0\}$  is decomposable with respect to a set  $\mathcal{D} \neq \emptyset$  if there exists a set of non-negative functions  $(f_i)_{i \in \mathcal{D}}$  with  $f_i: X \to \mathbb{R}^+ \cup \{0\}$ , such that  $f(x) = \sum_{i \in \mathcal{D}} f_i(x_i)$  holds for any  $x \in X^{|\mathcal{D}|}$ . A non-negative function  $f: X^{|\mathcal{D}|} \to \mathbb{R}^+ \cup \{0\}$  is decomposable with respect to a set  $\mathcal{D} = \emptyset$ , if there exists a non-negative constant  $f_o$  such that  $f(x) = f_o$  holds for any  $x \in X^{|\mathcal{D}|}$ .
Proposition 1. For a given graph  $\mathcal{G} = (\mathcal{N},\mathcal{E}_Q)$ , an MG  $(\mathcal{N},\mathcal{S},\mathcal{A},\mathbb{P},(r_i)_{i\in \mathcal{N}},\gamma)$  with more than two players is an NMG with respect to  $\mathcal{G}$  if and only if: (1)  $r_i(s,a_i,\cdot)$  is decomposable with respect to  $\mathcal{E}_{Q,i}$  for each  $i\in \mathcal{N}$ ,  $s\in \mathcal{S}$ ,  $a_i\in \mathcal{A}_i$ , i.e.,  $r_i(s,\pmb {a}) = \sum_{j\in \mathcal{E}_{Q,i}}r_{i,j}(s,a_i,a_j)$  for a set of functions  $\{r_{i,j}(s,a_i,\cdot)\}_{j\in \mathcal{E}_{Q,i}}$ , and (2) the transition dynamics  $\mathbb{P}(s^{\prime}|s,\cdot)$  is decomposable with respect to the set  $\mathcal{N}_C$  of this  $\mathcal{G}$ , i.e.,  $\mathbb{P}(s^{\prime}|s,\pmb {a}) = \sum_{i\in \mathcal{N}_C}\mathbb{F}_i(s^{\prime}|s,a_i)$  for a set of functions  $\{\mathbb{F}_i(s^{\prime}|s,\cdot)\}_{i\in \mathcal{N}_C}$  if  $\mathcal{N}_C\neq \emptyset$ , or  $\mathbb{P}(s^{\prime}|s,\pmb {a}) = \mathbb{F}_o(s^{\prime}|s)$  for some constant function (of  $\pmb{a}$ )  $\mathbb{F}_o(s'|s)$  if  $\mathcal{N}_C = \emptyset$ . Moreover, an MG qualifies as a zero-sum NMG if and only if it satisfies an additional condition: the NG, characterized by  $(\mathcal{G},\mathcal{A},(r_{i,j}(s))_{(i,j)\in \mathcal{E}_Q})$ , is a zero-sum NG for all  $s\in S$ . In the case of two players, every (zero-sum) Markov game becomes a (zero-sum) NMG.