SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Example 2. We consider the following 3-player Markov game that takes place for a time horizon  $H = 3$ . There exist three states,  $s_1, s_2$ , and  $s_3$  and the game starts at state  $s_1$ . Player 3 has a single action in every state, while players 1 and 2 have two available actions  $\{a_1, a_2\}$  and  $\{b_1, b_2\}$  respectively in every state. The initial state distribution  $\pmb{\rho}$  is the uniform probability distribution over  $S$ .
Transition probabilities. If players 1 and 2 select the joint action  $(a_{1}, b_{1})$  in state  $s_{1}$ , the game will transition to state  $s_{2}$ . In any other case, it will transition to state  $s_{3}$ . The converse happens if in state  $s_{2}$  they take joint action  $(a_{1}, b_{1})$ ; the game will transition to state  $s_{3}$ . For any other joint action, it will transition to state  $s_{1}$ . From state  $s_{3}$ , the game transition to state  $s_{1}$  or  $s_{2}$  uniformly at random.
Figure 2: A graph of the state space with transition probabilities parametrized with respect to the policy of each player.
Claim C.1. The joint policy  $\sigma$  that assigns probability  $\frac{1}{2}$  to the joint actions  $(a_{1},b_{2})$  and  $(a_{2},b_{1})$  in both states  $s_1,s_2$  is a CCE and  $V_{1}^{\sigma}(\pmb {\rho}) = V_{2}^{\sigma}(\pmb {\rho}) = -\frac{1}{10}$ .
$$
\begin{array}{l} V _ {1} ^ {\sigma} (\boldsymbol {\rho}) = \boldsymbol {\rho} ^ {\top} \left(\mathbf {I} - \gamma \mathbb {P} (\boldsymbol {\sigma})\right) ^ {- 1} \boldsymbol {r} _ {1} (\boldsymbol {\sigma}) \\ = \left( \begin{array}{c c c} \frac {1}{3} & \frac {1}{3} & \frac {1}{3} \end{array} \right) \left( \begin{array}{c c c} \frac {9}{5} & \frac {6}{5} & 0 \\ \frac {6}{5} & \frac {9}{5} & 0 \\ 1 & 1 & 1 \end{array} \right) \left( \begin{array}{c} \frac {1}{4 0} \\ \frac {1}{4 0} \\ - \frac {1}{2} \end{array} \right) \\ = - \frac {1}{1 0}. \\ \end{array}
$$
We check every deviation,
$$
\begin{array}{l} \bullet \pi_ {1} (s _ {1}) = \pi_ {1} (s _ {2}) = (1 0), V ^ {\pi_ {1} \times \sigma_ {- 1}} (\boldsymbol {\rho}) = - \frac {2}{5}, \\ \bullet \pi_ {1} (s _ {1}) = \pi_ {1} (s _ {2}) = (0 \quad 1), V ^ {\pi_ {1} \times \sigma_ {- 1}} (\boldsymbol {\rho}) = - \frac {1}{6}, \\ \bullet \pi_ {1} (s _ {1}) = (1 0), \pi_ {1} (s _ {2}) = (0 1), V ^ {\pi_ {1} \times \sigma_ {- 1}} (\rho) = - \frac {5}{1 6}, \\ \bullet \pi_ {1} (s _ {1}) = (0 1), \pi_ {1} (s _ {2}) = (1 0), V ^ {\pi_ {1} \times \sigma_ {- 1}} (\rho) = - \frac {5}{1 6}. \\ \end{array}
$$
For every such deviation the value of player 1 is smaller than  $-\frac{1}{10}$ . For player 2, the same follows by symmetry. Hence,  $\sigma$  is indeed a CCE.
Claim C.2. The product policy  $\pi_1^\sigma \times \pi_2^\sigma$  is not a NE.
Proof. For  $\pi^{\sigma} = \pi_1^{\sigma}\times \pi_2^{\sigma}$  we get,
$$
\begin{array}{l} V _ {1} ^ {\boldsymbol {\pi} ^ {\sigma}} = \rho^ {\top} (\mathbf {I} - \gamma \mathbb {P} (\boldsymbol {\pi} ^ {\sigma})) ^ {- 1} \boldsymbol {r} _ {1} (\boldsymbol {\pi} ^ {\sigma}) \\ = \left( \begin{array}{c c c} \frac {1}{3} & \frac {1}{3} & \frac {1}{3} \end{array} \right) \left( \begin{array}{c c c} \frac {3 4}{2 1} & \frac {2 0}{2 1} & \frac {3}{7} \\ \frac {2 0}{2 1} & \frac {3 4}{2 1} & \frac {3}{7} \\ \frac {6}{7} & \frac {6}{7} & \frac {9}{7} \end{array} \right) \left( \begin{array}{c} \frac {1}{4 0} \\ \frac {1}{4 0} \\ - \frac {1}{2} \end{array} \right) \\ = - \frac {3}{1 0}. \\ \end{array}
$$
But, for the deviation  $\pi_1(a_1|s_1) = \pi_1(a_1|s_2) = 0$ , the value functor of player 1, is equal to  $-\frac{1}{6}$ . Hence,  $\pi^{\sigma}$  is not a NE.
Theorem C.3 (No collapse—infinite-horizon). There exists a zero-sum polymatrix Markov game (Assumption 2 is not satisfied) that has a CCE which does not collapse to a NE.
Proof. The proof follows from the game of Example 2, and Claims C.1 and C.2.  $\square$
D An algorithm for approximating Markovian CCE
In this section we describe the algorithm in (Daskalakis et al., 2022) used for the computation of a Markovian CCE. We note that  $M, N_{\mathrm{visit}}, p$  are parameters that affect the accuracy of the approximation and we kindly ask the reader to refer to (Daskalakis et al., 2022) for further details. This algorithm computes an  $\epsilon$ -approximate CCE in time  $\tilde{O}(\epsilon^{-3})$ . Newer works have improved the dependence on  $\epsilon$  to  $\tilde{O}(\epsilon^{-2})$ , see (Wang et al., 2023; Cui et al., 2023).
As soon as an  $\epsilon$ -approximate CCE is computed, what is left to do is to compute the marginal policy of every player,  $\pi_{k,h}^{\sigma}(a|s) = \sum_{\mathbf{a}_{-k}\in \mathcal{A}_{-k}}\sigma (a,\mathbf{a}_{-k}|s)$ .
Algorithm 1 SPoCMAR (Daskalakis et al., 2022)
1: procedure SPOCMAR(n, S, A, H, M, Nvisit, p)  
2: Set V = ∅.  
3: For each h ∈ [H], s ∈ S, set σqcover =⊥.  
4: for q ≥ 1 and while τ = 0 do  
5: Set τ = 1 #Terminate flag.  
6: Set Πh^q := {σh,s^cover : s ∈ S} for each h ∈ [H].  
7: for h = H, H-1, ..., 1 do  
8: Set k = 0, and V^q_{i,H+1}(s) = 0 for all s ∈ S and i ∈ [m].  
9: Each player i initializes an adversarial bandit for all (s,h) ∈ S × [H].  
10: for each σ ∈ Πh^q ∪ σU do # σU chooses actions uniformly at random  
11: for a total of M times do  
12: k ← k + 1.  
13: Let σ be the policy which follows σ for the first h - 1 steps and plays according to the bandit algorithm for the state visited at step h (and acts arbitrarily for steps h' > h).  
14: Draw a joint trajectory (s1,m, a1,m, r1,m, ..., sH,m, aH,m, rH,m) from σ.  
15: if (h, sh,m) ∈ V then  
16: Each i updates its bandit alg. at (h, sh,m) with (ai,h,m, H-r_i,h,m-V^q_{i,h+1}(sh+1,m)/H).  
17: else  
18: Each i updates its bandit alg. at (h, sh,m) with (ai,h,m, H-(H+1-h)/H).  
19: end if  
20: end for  
21: end for  
22: For each s ∈ S, and j ≥ 1, let m_j,h,s ∈ [M+1] denote the jth smallest value of k so that sh,m = s, or M + 1 if such a jth smallest value does not exist.  
23: For each s ∈ S, let J_h,s denote the largest integer j so that k_j,h,s ≤ M.  
24: Define σ^q ∈ Δ(A)^S to be the 1-step policy: σ^q(h|s) = 1/∑j=1J_h,s[H] [a = ah,m_j,h,s].  
25: Set V^q_{i,h}(s) := { \begin{cases} \frac{1}{J_h,s} \sum_{j=1}^{J_h,s} \left( r_{i,h,k_j,h,s} + \bar{V}_{i,h+1}^q (sh+1,k_j,h,s) \right) & : (h,s) \in \mathcal{V} \\ (H+1-h) & : (h,s) \notin \mathcal{V}. \end{cases} }  
26: end for  
27: Define the joint policy σ^q, which follows σ^q_{h'} at each step h' ∈ [H].  
28: Call EstVisitation(σ^q, Nvisit) (Alg. 2) to obtain estimates d^q_{h'} ∈ Δ(S) for each h' ∈ [H].  
29: for each s ∈ S and h' ∈ [H] do  
30: if d^q_{h'}(s) ≥ p and (h', s)∉ V then  
31: Set σ^cover_{h',s} ← σ^q.  
32: Add (h', s) to V.  
33: Set τ ← 0.  
34: end if  
35: end for  
36: end for  
37: return the policy σ := σ^q.  
38: end procedure
Algorithm 2 EstVisitation
1: procedure ESTVISITATION  $(\sigma ,N)$    
2: for  $1\leq n\leq N$  do   
3: Draw a trajectory from  $\pmb{\sigma}$  , and let  $(s_1^n,\dots,s_H^n)$  denote the sequence of states observed.   
4: end for   
5: for  $h\in [H]$  do   
6: Let  $\hat{d}_h\in \Delta (\mathcal{S})$  denote the empirical distribution over  $(s_h^1,\ldots ,s_h^N)$    
7: end for   
8: return  $(\hat{d}_1,\dots,\hat{d}_H)$    
9: end procedure