SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
V _ {h = 1} ^ {\pi_ {1} ^ {\sigma} \times \pi_ {2}} (s _ {1}) - 2 \epsilon \leq V _ {h = 1} ^ {\pi_ {1} ^ {\sigma} \times \pi_ {2} ^ {\sigma}} (s _ {1}) \leq V _ {h = 1} ^ {\pi_ {1} \times \pi_ {2} ^ {\sigma}} (s _ {1}) + 2 \epsilon ,
$$
which is the definition of a NE in a zero-sum game.
3.2 Equilibrium collapse in finite-horizon polymatrix Markov games
In this section, we focus on the more challenging case of polymatrix Markov games which is the main focus of this paper. For any finite horizon Markov game, we define  $(\mathrm{P_{NE}})$  to be the following nonlinear program with variables  $\pi, w$ :
$$
\min  \sum_ {k \in [ n ]} \left(w _ {k, 1} \left(s _ {1}\right) - e _ {s _ {1}} ^ {\top} \sum_ {h = 1} ^ {H} \left(\prod_ {\tau = 1} ^ {h} \mathbb {P} _ {\tau} \left(\boldsymbol {\pi} _ {\tau}\right)\right) \boldsymbol {r} _ {k, h} \left(\boldsymbol {\pi} _ {h}\right)\right)
$$
$$
\begin{array}{l} \text {s . t .} w _ {k, h} (s) \geq r _ {k, h} (s, a, \boldsymbol {\pi} _ {- k, h}) + \mathbb {P} _ {h} (s, a, \boldsymbol {\pi} _ {- k, h}) \boldsymbol {w} _ {k, h + 1}, \\ \forall s \in \mathcal {S}, \forall h \in [ H ], \forall k \in [ n ], \forall a \in \mathcal {A} _ {k}; \\ \end{array}
$$
$\left(\mathrm{P}_{\mathrm{NE}}\right)$
$$
w _ {k, H} (s) = 0, \quad \forall k \in [ n ], \forall s \in S;
$$
$$
\pi_ {k, h} (s) \in \Delta (\mathcal {A} _ {k}),
$$
$$
\forall s \in \mathcal {S}, \forall h \in [ H ], \forall k \in [ n ], \forall a \in \mathcal {A} _ {k}.
$$
Using the following theorem, we are able to use  $(\mathrm{P_{NE}})$  to argue about equilibrium collapse.
Theorem 3.2 (NE and global optima of  $(\mathrm{P_{NE}})$ ). If  $(\pi^{\star},\pmb{w}^{\star})$  yields an  $\epsilon$ -approximate global minimum of  $(\mathrm{P_{NE}})$ , then  $\pi^{\star}$  is an  $n\epsilon$ -approximate NE of the zero-sum polymatrix switching controller  $MG$ ,  $\Gamma$ . Conversely, if  $\pi^{\star}$  is an  $\epsilon$ -approximate NE of the  $MG\Gamma$  with corresponding value function vector  $\pmb{w}^{\star}$  such that  $w_{k,h}^{\star}(s) = V_{k,h}^{\pi^{\star}}(s)\forall (k,h,s)\in [n]\times [H]\times S$ , then  $(\pi^{\star},\pmb{w}^{\star})$  attains an  $\epsilon$ -approximate global minimum of  $(\mathrm{P_{NE}})$ .
Following, we are going to use  $(\mathrm{P_{NE}})$  in proving the collapse of CCE's to NE's. We observe that the latter program is nonlinear and in general nonconvex. Hence, duality cannot be used in the way it was used in (Cai et al., 2016) to prove equilibrium collapse. Nevertheless, we can prove that given a CCE policy  $\sigma$ , the marginalized, product policy  $\times_{k\in [n]}\pi_k^\sigma$  along with an appropriate vector  $w^{\sigma}$  achieves a global minimum in the nonlinear program  $(\mathrm{P_{NE}})$ . More precisely, our main result reads as the following statement.
Theorem 3.3 (CCE collapse to NE in polymatrix MG). Let a zero-sum polymatrix switching-control Markov game, i.e., a Markov game for which Assumptions 1 and 2 hold. Further, let an  $\epsilon$ -approximate CCE of that game  $\sigma$ . Then, the marginal product policy  $\pi^{\sigma}$ , with  $\pi_{k,h}^{\sigma}(a|s) = \sum_{a_{-k} \in \mathcal{A}_{-k}} \sigma_h(a, a_{-k})$ ,  $\forall k \in [n], \forall h \in [H]$  is an  $n\epsilon$ -approximate NE.
Proof. Let an  $\epsilon$ -approximate CCE policy,  $\sigma$ , of game  $\Gamma$ . Moreover, let the best-response value-vectors of each agent  $k$  to joint policy  $\sigma_{-k}$ ,  $\boldsymbol{w}_k^\dagger$ .
Now, we observe that due to Assumption 1,
$$
\begin{array}{l} w _ {k, h} ^ {\dagger} (s) \geq r _ {k, h} (s, a, \boldsymbol {\sigma} _ {- k, h}) + \mathbb {P} _ {h} (s, a, \boldsymbol {\sigma} _ {- k, h}) \boldsymbol {w} _ {k, h + 1} ^ {\dagger} \\ = \sum_ {j \in \operatorname {a d j} (k)} r _ {(k, j), h} (s, a, \boldsymbol {\pi} _ {j} ^ {\sigma}) + \mathbb {P} _ {h} (s, a, \boldsymbol {\sigma} _ {- k, h}) \boldsymbol {w} _ {k, h + 1} ^ {\dagger}. \\ \end{array}
$$
Further, due to Assumption 2,
$$
\mathbb {P} _ {h} (s, a, \boldsymbol {\sigma} _ {- k, h}) \boldsymbol {w} _ {k, h + 1} ^ {\dagger} = \mathbb {P} _ {h} (s, a, \boldsymbol {\pi} _ {\operatorname {a r g c t r l} (s), h} ^ {\boldsymbol {\sigma}}) \boldsymbol {w} _ {k, h + 1} ^ {\dagger},
$$
$$
\mathbb {P} _ {h} (s, a, \boldsymbol {\sigma} _ {- k, h}) \boldsymbol {w} _ {k, h + 1} ^ {\dagger} = \mathbb {P} _ {h} (s, a, \boldsymbol {\pi} ^ {\boldsymbol {\sigma}}) \boldsymbol {w} _ {k, h + 1} ^ {\dagger}.
$$
Putting these pieces together, we reach the conclusion that  $(\pi^{\sigma}, \boldsymbol{w}^{\dagger})$  is feasible for the nonlinear program  $(\mathrm{P}_{\mathrm{NE}})$ .
What is left is to prove that it is also an  $\epsilon$ -approximate global minimum. Indeed, if  $\sum_{k} w_{k,h}^{\dagger}(s_1) \leq \epsilon$  (by assumption of an  $\epsilon$ -approximate CCE), then the objective function of  $(\mathrm{P}_{\mathrm{NE}})$  will attain an  $\epsilon$ -approximate global minimum. In turn, due to Theorem 3.2 the latter implies that  $\pi^{\sigma}$  is an  $n\epsilon$ -approximate NE.
We can now conclude that due to the algorithm introduced in (Daskalakis et al., 2022) for CCE computation in general-sum MG's, the next statement holds true.
Corollary 3.1 (Computing a NE—finite-horizon). Given a finite-horizon switching control zero-sum polymatrix Markov game, we can compute an  $\epsilon$ -approximate Nash equilibrium policy that is Markovian with probability at least  $1 - \delta$  in time poly  $(n, H, S, \max_k |\mathcal{A}_k|, \frac{1}{\epsilon}, \log(1 / \delta))$ .
In the next section, we discuss the necessity of the assumption of switching control using a counterexample of non-collapsing equilibria.
3.3 No equilibrium collapse with more than one controllers per-state
Although Assumption 1 is sufficient for the collapse of any CCE to a NE in single-state (i.e., normalform) games, we will prove that Assumption 2 is indispensable in guaranteeing such a collapse in zero-sum polymatrix Markov games. That is, if more than one players affect the transition probability from one state to another, a CCE is not guaranteed to collapse to a NE.
Example 1. We consider the following 3-player Markov game that takes place for a time horizon  $H = 3$ . There exist three states,  $s_1, s_2$ , and  $s_3$  and the game starts at state  $s_1$ . Player 3 has a single action in every state, while players 1 and 2 have two available actions  $\{a_1, a_2\}$  and  $\{b_1, b_2\}$  respectively in every state.
Reward functions. If player 1 (respectively, player 2) takes action  $a_1$  (resp.,  $b_1$ ), in either of the states  $s_1$  or  $s_2$ , they get a reward equal to  $\frac{1}{20}$ . In state  $s_3$ , both players get a reward equal to  $-\frac{1}{2}$  regardless of the action they select. Player 3 always gets a reward that is equal to the negative sum of the reward of the other two players. This way, the zero-sum polymatrix property of the game is ensured (Assumption 1).
Transition probabilities. If players 1 and 2 select the joint action  $(a_{1}, b_{1})$  in state  $s_{1}$ , the game will transition to state  $s_{2}$ . In any other case, it will transition to state  $s_{3}$ . The converse happens if in state  $s_{2}$  they take joint action  $(a_{1}, b_{1})$ ; the game will transition to state  $s_{3}$ . For any other joint action, it will transition to state  $s_{1}$ . From state  $s_{3}$ , the game transitions to state  $s_{1}$  or  $s_{2}$  uniformly at random.
At this point, it is important to notice that two players control the transition probability from one state to another. In other words, Assumption 2 does not hold.
Figure 1: A graph of the state space with transition probabilities parametrized with respect to the policy of each player.
Next, we consider the joint policy  $\pmb{\sigma}$
$$
\boldsymbol {\sigma} (s _ {1}) = \boldsymbol {\sigma} (s _ {2}) = \begin{array}{c c c} & a _ {1} & b _ {1} \\ & a _ {2} & 0 \\ & & 1 / 2 \end{array} \left( \begin{array}{c c} 0 & 1 / 2 \\ 1 / 2 & 0 \end{array} \right).
$$
Claim 3.1. The joint policy  $\sigma$  that assigns probability  $\frac{1}{2}$  to the joint actions  $(a_{1}, b_{2})$  and  $(a_{2}, b_{1})$  in both states  $s_{1}, s_{2}$  is a CCE and  $V_{1,1}^{\sigma}(s_{1}) = V_{2,1}^{\sigma}(s_{1}) = \frac{1}{20}$ .
Yet, the marginalized product policy of  $\sigma$  which we note as  $\pi_1^\sigma \times \pi_2^\sigma$  does not constitute a NE. The components of this policy are,
$$
\left\{ \begin{array}{l} \pi_ {1} ^ {\sigma} (s _ {1}) = \pi_ {1} ^ {\sigma} (s _ {2}) = \left( \begin{array}{c c} a _ {1} & a _ {2} \\ 1 / 2 & 1 / 2 \end{array} \right), \\ \pi_ {2} ^ {\sigma} (s _ {1}) = \pi_ {2} ^ {\sigma} (s _ {2}) = \left( \begin{array}{c c} b _ {1} & b _ {2} \\ 1 / 2 & 1 / 2 \end{array} \right). \end{array} \right.
$$
I.e., the product policy  $\pi_1^\sigma \times \pi_2^\sigma$  selects any of the two actions of each player in states  $s_1, s_2$  independently and uniformly at random. With the following claim, it can be concluded that in general when more than one player controls the transition the set of equilibria do not collapse.
Claim 3.2. The product policy  $\pi_1^\sigma \times \pi_2^\sigma$  is not a NE.
In conclusion, Assumption 1 does not suffice to ensure equilibrium collapse.
Theorem 3.4. There exists a zero-sum polymatrix Markov game (Assumption 2 is not satisfied) that has a CCE which does not collapse to a NE.
3.4 Equilibrium collapse in infinite-horizon polymatrix Markov games
In proving equilibrium collapse for infinite-horizon polymatrix Markov games, we use similar arguments and the following nonlinear program with variables  $\pi, w$ ,
$$
\begin{array}{l} \min  \sum_ {k \in [ n ]} \boldsymbol {\rho} ^ {\top} \left(\boldsymbol {w} _ {k} - (\mathbf {I} - \gamma \mathbb {P} (\boldsymbol {\pi})) ^ {- 1} \boldsymbol {r} _ {k} (\boldsymbol {\pi})\right) \\ \text {s . t .} w _ {k} (s) \geq r _ {k} (s, a, \boldsymbol {\pi} _ {- k}) + \gamma \mathbb {P} (s, a, \boldsymbol {\pi} _ {- k}) \boldsymbol {w} _ {k}, \\ \begin{array}{c} (\mathrm {P} _ {\mathrm {N E}} ^ {\prime}) \hskip 1 4. 2 2 6 3 7 8 p t \forall s \in \mathcal {S}, \forall k \in [ n ], \forall a \in \mathcal {A} _ {k}; \\ \pi_ {k} (s) \in \Delta (\mathcal {A} _ {k}), \end{array} \\ \forall s \in \mathcal {S}, \forall k \in [ n ], \forall a \in \mathcal {A} _ {k}. \\ \end{array}
$$
We note that Example 1 can be properly adjusted to show that the switching-control assumption is necessary for equilibrium collapse in infinite-horizon games as well. Compared to finite-horizon games, infinite-horizon games cannot be possibly solved using backward induction. They pose a genuine computational challenge and, in that sense, the importance of the property of equilibrium collapse gets highlighted.
Computational implications. Equilibrium collapse in infinite-horizon MG's allows us to use the CCE computation technique found in (Daskalakis et al., 2022) in order to compute an  $\epsilon$ -approximate NE. Namely, given an accuracy threshold  $\epsilon$ , we truncate the infinite-horizon game to its effective horizon  $H \coloneqq \frac{\log(1 / \epsilon)}{1 - \gamma}$ . Then, we define reward functions that depend on the time-step  $h$ , i.e.,  $r_{k,h} = \gamma^{h-1} r_k$ . Finally,
Corollary 3.2. (Computing a NE—infinite-horizon) Given an infinite-horizon switching control zero-sum polymatrix game  $\Gamma$ , it is possible to compute a Nash equilibrium policy that is Markovian and nonstationary with probability at least  $1 - \delta$  in time poly  $\left(n, \frac{1}{1 - \gamma}, S, \max_k |\mathcal{A}_k|, \frac{1}{\epsilon}, \log(1 / \delta)\right)$ .
4 Conclusion and open problems
In this paper, we unified switching-control Markov games and zero-sum polymatrix normal-form games. We highlighted how numerous applications can be modeled using this framework and we focused on the phenomenon of equilibrium collapse from the set of coarse-correlated equilibria to that of Nash equilibria. This property holds implications for computing approximate Nash equilibria in switching control zero-sum polymatrix Markov games; it ensures that it can be done efficiently.
Open problems. In light of the proposed problem and our results there are multiple interesting open questions:
- Is it possible to use a policy optimization algorithm similar to those of (Erez et al., 2022; Zhang et al., 2022) in order to converge to an approximate Nash equilibrium? We note that the question can be settled in one of two ways; either extend the current result of equilibrium collapse to policies that are non-Markovian or guarantee convergence to Markovian policies. The notion of regret in (Erez et al., 2022) gives rise to the computation of a CCE that is a non-Markovian policy in the sense that the policy at every timestep depends on the policy sampled from the history of no-regret play and not only the given state.
- We conjecture that a convergence rate of  $O\left(\frac{1}{T}\right)$  to a NE is possible, i.e., there exists an algorithm with running time  $O(1 / \epsilon)$  that computes an  $\epsilon$ -approximate NE.
- Are there more classes of Markov games in which computing Nash equilibria is computationally tractable?
Aaron Sidford, Mengdi Wang, Xian Wu, Lin Yang, and Yinyu Ye. Near-optimal time and sample complexities for solving markov decision processes with a generative model. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, pages 5192-5202, 2018.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nat., 550(7676):354-359, 2017. doi: 10.1038/nature24270.
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
György Szabó and Gabor Fath. Evolutionary games on graphs. Physics reports, 446(4-6):97-216, 2007.
Yodyium Tipsuwan and Mo-Yuen Chow. Control methodologies in networked control systems. Control engineering practice, 11(10):1099-1111, 2003.
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michael Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Remi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Caglar Gulçehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wunsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nat., 575(7782): 350-354, 2019. doi: 10.1038/s41586-019-1724-z.
OJ Vrieze, SH Tijs, TES Raghavan, and JA Filar. A finite algorithm for the switching control stochastic game. Or Spektrum, 5(1):15-24, 1983.
Xiaofeng Wang and Tuomas Sandholm. Reinforcement learning to play an optimal nash equilibrium in team markov games. In Advances in Neural Information Processing Systems 15 [Neural Information Processing Systems, NIPS 2002, December 9-14, 2002, pages 1571-1578. MIT Press, 2002.
Yuanhao Wang, Qinghua Liu, Yu Bai, and Chi Jin. Breaking the curse of multiagency: Provably efficient decentralized multi-agent rl with function approximation. arXiv preprint arXiv:2302.06606, 2023.
Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive markov games. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pages 4259-4299. PMLR, 15-19 Aug 2021.
Runyu Zhang, Zhaolin Ren, and Na Li. Gradient play in stochastic games: stationary points, convergence, and sample complexity. arXiv preprint arXiv:2106.00198, 2021.
Runyu Zhang, Qinghua Liu, Huan Wang, Caiming Xiong, Na Li, and Yu Bai. Policy optimization for markov games: Unified framework and faster convergence, 2022.
A Missing statements and proofs
A.1 Statements for Section 3.1
Claim A.1. Let a two-player Markov game where both players affect the transition. Further, consider a correlated policy  $\sigma$  and its corresponding marginalized product policy  $\pi^{\sigma} = \pi_1^{\sigma} \times \pi_2^{\sigma}$ . Then, for any  $\pi_1', \pi_2'$ ,
$$
V _ {k, 1} ^ {\boldsymbol {\pi} _ {1} ^ {\prime}, \boldsymbol {\sigma} _ {- 1}} (s _ {1}) = V _ {k, 1} ^ {\boldsymbol {\pi} _ {1} ^ {\prime}, \boldsymbol {\pi} _ {2} ^ {\sigma}} (s _ {1}),
$$
$$
V _ {k, 2} ^ {\sigma_ {- 2}, \pi_ {2} ^ {\prime}} (s _ {1}) = V _ {k, 2} ^ {\pi_ {1} ^ {\sigma}, \pi_ {2} ^ {\prime}} (s _ {1}).
$$
Proof. We will effectively show that the problem of best-responding to a correlated policy  $\sigma$  is equivalent to best-responding to the marginal policy of  $\sigma$  for the opponent. The proof follows from the equivalence of the two MDPs.
As a reminder,
$$
\pi_ {1, h} (a | s) = \sum_ {b \in \mathcal {A} _ {2}} \sigma_ {h} (a, b | s)
$$
$$
\pi_ {2, h} (b | s) = \sum_ {a \in \mathcal {A} _ {1}} \sigma_ {h} (a, b | s)
$$
As we have seen in Section 2.1, in the case of unilateral deviation from joint policy  $\sigma$ , an agent faces a single agent MDP. More specifically, agent 2, best-responds by optimizing a reward function  $\bar{r}_{2,h}(s,b)$  under a transition kernel  $\mathbb{P}_2$  for which,
$$
\bar {r} _ {2, h} (s, b) = \mathbb {E} _ {b \sim \pmb {\sigma}} [ r _ {2, h} (s, a, b) ] = \mathbb {E} _ {b \sim \pmb {\pi} _ {1} ^ {\sigma}} [ r _ {2, h} (s, a, b) ] = r _ {2, h} (s, \pmb {\pi} _ {1} ^ {\sigma}, b).
$$
$$
\bar {r} _ {1, h} (s, b) = r _ {1, h} (s, a, \pi_ {2} ^ {\sigma}).
$$
Analogously, for each of the transition kernels,
$$
\bar {\mathbb {P}} _ {2, h} (s ^ {\prime} | s, b) = \mathbb {E} _ {a \sim \sigma} [ \mathbb {P} _ {2, h} (s ^ {\prime} | s, a, b) ] = \mathbb {E} _ {a \sim \pi_ {2} ^ {\sigma}} [ \mathbb {P} _ {2, h} (s ^ {\prime} | s, a, b) ] = \mathbb {P} _ {2, h} (s ^ {\prime} | s, \boldsymbol {\pi} _ {1} ^ {\sigma}, b),
$$
as for agent 1,
$$
\bar {\mathbb {P}} _ {1, h} \left(s ^ {\prime} | s, a\right) = \mathbb {P} _ {1, h} \left(s ^ {\prime} | s, a, \boldsymbol {\pi} _ {2} ^ {\sigma}\right).
$$
Hence, it follows that,  $V_{2,1}^{\sigma - 2 \times \pi_2'}(s_1) = V_{2,1}^{\pi_1^{\sigma} \times \pi_2'}(s_1), \forall \pi_2'$  and  $V_{1,1}^{\pi_1' \times \sigma - 1}(s_1) = V_{1,1}^{\pi_1' \times \pi_2'}(s_1), \forall \pi_2'$ .
The best-response program. First, we state the following lemma that will prove useful for several of our arguments,
Lemma A.1 (Best-response LP). Let a (possibly correlated) joint policy  $\hat{\sigma}$ . Consider the following linear program with variables  $\boldsymbol{w} \in \mathbb{R}^{n \times H \times S}$ ,
$$
\begin{array}{l} \min  \quad \sum_ {k \in [ n ]} w _ {k, s} (s _ {1}) - \boldsymbol {e} _ {s _ {1}} ^ {\top} \sum_ {h = 1} ^ {H} \left(\prod_ {\tau = 1} ^ {h} \mathbb {P} _ {\tau} (\hat {\boldsymbol {\sigma}} _ {\tau})\right) \boldsymbol {r} _ {k, h} (\hat {\boldsymbol {\sigma}} _ {h}) \\ \begin{array}{l} \text {s . t .} w _ {k, h} (s) \geq r _ {k, h} (s, a, \hat {\sigma} _ {- k, h}) + \mathbb {P} _ {h} (s, a, \hat {\sigma} _ {- k, h}) \boldsymbol {w} _ {k, h + 1}, \\ \forall s \in \mathcal {S}, \forall h \in [ H ], \forall k \in [ n ], \forall a \in \mathcal {A} _ {k}; \end{array} \\ w _ {k, H} (s) = 0, \forall k \in [ n ], \forall s \in \mathcal {S}. \\ \end{array}
$$
The optimal solution  $\pmb{w}^{\dagger}$  of the program is unique and corresponds to the value function of each player  $k\in [n]$  when player  $k$  best-responds to  $\hat{\sigma}$ .
Proof. We observe that the program is separable to  $n$  independent linear programs, each with variables  $\boldsymbol{w}_k \in \mathbb{R}^{n \times H}$ ,
$$
\begin{array}{l} \min  w _ {k, 1} \left(s _ {1}\right) \\ \mathrm {s . t .} w _ {k, h} (s) \geq r _ {k, h} (s, a, \hat {\sigma} _ {- k, h}) + \mathbb {P} _ {h} (s, a, \hat {\sigma} _ {- k, h}) w _ {k, h + 1}, \\ \forall s \in \mathcal {S}, \forall h \in [ H ], \forall a \in \mathcal {A} _ {k}; \\ w _ {k, H} (s) = 0, \forall k \in [ n ], \forall s \in \mathcal {S}. \\ \end{array}
$$
Each of these linear programs describes the problem of a single agent MDP (Neu and Pike-Burke, 2020, Section 2) that agent being  $k$  — which, as we have seen in Best-response policies, is equivalent to the problem of finding a best-response to  $\hat{\sigma}_{-k}$ . It follows that the optimal  $\boldsymbol{w}_k^\dagger$  for every program is unique (each program corresponds to a set of Bellman optimality equations).
Properties of the NE program. Second, we need to prove that the minimum value of the objective function of the program is nonnegative.
Lemma A.2 (Feasibility of  $\mathrm{(P_{NE}^{\prime})}$  and global optimum). The nonlinear program  $(\mathrm{P}_{\mathrm{NE}}^{\prime})$  is feasible, has a nonnegative objective value, and its global minimum is equal to 0.
Proof. Analogously to the finite-horizon case, for the feasibility of the nonlinear program, we invoke the theorem of the existence of a Nash equilibrium. We let a NE product policy,  $\pi^{\star}$ , and a vector  $\pmb{w}^{\star} \in \mathbb{R}^{n \times S}$  such that  $w_{k}^{\star}(s) = V_{k}^{\dagger, \pi_{-k}^{\star}}(s)$ ,  $\forall k \in [n] \times S$ .
By Lemma A.1, we know that  $(\pmb{\pi}^{\star}, \pmb{w}^{\star})$  satisfies all the constraints of  $(\mathrm{P}_{\mathrm{NE}})$ . Additionally, because  $\pmb{\pi}^{\star}$  is a NE,  $V_{k,h}^{\pmb{\pi}^{\star}}(s_1) = V_{k,h}^{\dagger,\pmb{\pi}^{\star}_{-k}}(s_1)$  for all  $k \in [n]$ . Observing that,
$$
w _ {k, 1} ^ {\star} (s _ {1}) - \boldsymbol {e} _ {s _ {1}} ^ {\top} \sum_ {h = 1} ^ {H} \left(\prod_ {\tau = 1} ^ {h} \mathbb {P} _ {\tau} (\boldsymbol {\pi} _ {\tau} ^ {\star})\right) \boldsymbol {r} _ {k, h} (\boldsymbol {\pi} _ {h} ^ {\star}) = V _ {k, h} ^ {\dagger , \boldsymbol {\pi} _ {- k} ^ {\star}} (s _ {1}) - V _ {k, h} ^ {\boldsymbol {\pi} ^ {\star}} (s _ {1}) = 0,
$$
concludes the argument that a NE attains an objective value equal to 0.
Continuing, we observe that due to (1) the objective function can be equivalently rewritten as,
$$
\begin{array}{l} \sum_ {k \in [ n ]} \left(w _ {k, 1} (s _ {1}) - e _ {s _ {1}} ^ {\top} \sum_ {h = 1} ^ {H} \left(\prod_ {\tau = 1} ^ {h} \mathbb {P} _ {\tau} (\boldsymbol {\pi} _ {\tau})\right) \boldsymbol {r} _ {k, h} (\boldsymbol {\pi} _ {h})\right) \\ = \sum_ {k \in [ n ]} w _ {k, 1} (s _ {1}) - e _ {s _ {1}} ^ {\top} \sum_ {h = 1} ^ {H} \left(\prod_ {\tau = 1} ^ {h} \mathbb {P} _ {\tau} (\boldsymbol {\pi} _ {\tau})\right) \sum_ {k \in [ n ]} \boldsymbol {r} _ {k, h} (\boldsymbol {\pi} _ {h}) \\ = \sum_ {k \in [ n ]} w _ {k, 1} \left(s _ {1}\right). \\ \end{array}
$$
Next, we focus on the inequality constraint
$$
w _ {k, h} (s) \geq r _ {k, h} (s, a, \pi_ {- k, h}) + \mathbb {P} _ {h} (s, a, \pi_ {- k, h}) \boldsymbol {w} _ {k, h + 1}
$$
which holds for all  $s \in S$ , all players  $k \in [n]$ , all  $a \in \mathcal{A}_k$ , and all timesteps  $h \in [H - 1]$ .
By summing over  $a \in \mathcal{A}_k$  while multiplying each term with a corresponding coefficient  $\pi_{k,h}(a|s)$ , the display written in an equivalent element-wise vector inequality reads:
$$
\boldsymbol {w} _ {k, h} \geq \boldsymbol {r} _ {k, h} (\boldsymbol {\pi} _ {h}) + \mathbb {P} _ {h} (\boldsymbol {\pi} _ {h}) \boldsymbol {w} _ {k, h + 1}.
$$
Finally, after consecutively substituting  $\pmb{w}_{k,h + 1}$  with the element-wise lesser term  $\pmb{r}_{k,h + 1}(\pmb{\pi}_{h + 1}) + \mathbb{P}_{h + 1}(\pmb{\pi}_{h + 1})\pmb{w}_{k,h + 2}$ , we end up with the inequality:
$$
\boldsymbol {w} _ {k, 1} \geq \sum_ {h = 1} ^ {H} \left(\prod_ {\tau = 1} ^ {h} \mathbb {P} _ {\tau} \left(\boldsymbol {\pi} _ {\tau}\right)\right) \boldsymbol {r} _ {k, h} \left(\boldsymbol {\pi} _ {h}\right). \tag {5}
$$
Summing over  $k$ , it holds for the  $s_1$ -th entry of the inequality,