SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Perolat, J., Vylder, B. D., Hennes, D., Tarassov, E., Strub, F., de Boer, V., Muller, P., Connor, J. T., Burch, N., Anthony, T., McAleer, S., Elie, R., Cen, S. H., Wang, Z., Gruslys, A., Malysheva, A., Khan, M., Ozair, S., Timbers, F., Pohlen, T., Eccles, T., Rowland, M., Lanctot, M., Lespiau, J.-B., Piot, B., Omidshafiei, S., Lockhart, E., Sifre, L., Beauguerlange, N., Munos, R., Silver, D., Singh, S., Hassabis, D., and Tuyls, K. Mastering the game of strategy with model-free multiagent reinforcement learning. Science, 378(6623):990-996, 2022.
Selten, R. Reexamination of the perfectness concept for equilibrium points in extensive games. In Models of Strategic Rationality, pp. 1-31. Springer, 1988.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., and Hassabis, D. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419): 1140-1144, 2018.
Southey, F., Bowling, M. P., Larson, B., Piccione, C., Burch, N., Billings, D., and Rayner, C. Bayes' bluff: Opponent modelling in poker. arXiv preprint arXiv:1207.1411, 2012.
Steinberger, E., Lerer, A., and Brown, N. Dream: Deep regret minimization with advantage baselines and model-free learning. arXiv preprint arXiv:2006.10410, 2020.
Tammelin, O. Solving large imperfect information games using cfr+. arXiv preprint arXiv:1407.5042, 2014.
Tammelin, O., Burch, N., Johanson, M., and Bowling, M. Solving heads-up limit texas hold'em. In Twenty-fourth international joint conference on artificial intelligence, 2015.
Tuyls, K., Pérolat, J., Lanctot, M., Ostrovski, G., Savani, R., Leibo, J. Z., Ord, T., Graepel, T., and Legg, S. Symmetric decomposition of asymmetric games. *Scientific reports*, 8(1):1-20, 2018.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J. P., Jaderberg, M., Vezhnevets, A. S., Leblond, R., Pohlen, T., Dalibard, V., Budden, D., Sulsky, Y., Molloy, J., Paine, T. L., Gülçehre, C., Wang, Z., Pfaff, T., Wu, Y., Ring, R., Yogatama, D., Wünsch, D., McKinney, K., Smith, O., Schaul, T., Lillicrap, T. P., Kavukcuoglu, K., Hassabis, D., Apps, C., and Silver, D. Grandmaster level in Starcraft II using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
von Neumann, J. Zur theorie der gesellschaftsspiele. Mathematische annalen, 100(1):295-320, 1928.
Von Stengel, B. and Forges, F. Extensive-form correlated equilibrium: Definition and computational complexity. Mathematics of Operations Research, 33(4):1002-1022, 2008.
Zinkevich, M., Johanson, M., Bowling, M., and Piccione, C. Regret minimization in games with incomplete information. Neural Information Processing Systems, 2008.
A Marginals of a CCE May Not Be a CCE
a	b
a	1,0	-1,-1
a	-1,-1	0,1
Here we give an example showing the marginal strategies of a CCE may not form a CCE. Consider  $\mu$  s.t.  $\mu (a,a) = 0.5$  and  $\mu (b,b) = 0.5$ .  $\mu$  is a CCE.  $\mathbb{E}_{\rho \sim \mu}[u_i(\rho)] = 0.5$  for each player. If row  $(r)$  were to player  $a$  and column continues to play according to  $\mu$ , row's utility is 0; if  $r$  plays  $b$  instead, their utility is now  $-0.5$ . Thus  $r$  has no profitable deviations from the CCE recommendations. Column does not either, this can be shown with a symmetric argument.
Row's marginal strategy  $s_r^\mu$  plays  $a$  with probability 0.5 and  $b$  with probability 0.5,  $s_c^\mu$  does likewise.  $u_r(s_r^\mu, s_c^\mu) = u_c(s_r^\mu, s_c^\mu) = -0.25$ . However,  $a$  is a profitable deviation for  $r$  now since  $0 > -0.25$ , thus the decorrelated strategies from the same CCE are also not a CCE.
B Hindsight Rationality With Respect to Action Deviations Does Not Imply Nash
Here we show that hindsight rationality with respect to action deviations does not imply Nash equilibrium in 2 player constant-sum games. We show this with a 1 player game. Consider the agents strategy, shown in blue, which receives utility of 1. Deviating to  $[I_1:b,I_2:a]$  will increase the player's utility to 2, so the blue strategy is not a Nash equilibrium. However, this would require two simultaneous action deviations, one at  $I_{1}$  to  $b$  and one at  $I_{2}$  to  $a$ . Neither of these deviations increases the player's utility on their own, so the player is hindsight rational w.r.t. action deviations.
Figure 3: The marginal strategies of a CCE do not generally form a CCE themselves.
Figure 4: Action deviations in a simple game.
Proposition C.1 (Cai et al. (2016)). In CSP games, for any  $CCE\mu$ , if  $i$  deviates to  $s_i$ , then their expected utility if other players continue to play  $\mu$  is equal to their utility if other players where to play the marginal strategy profile  $s_{-i}^{\mu}$ :
$$
\mathbb {E} _ {\rho \sim \mu} [ u _ {i} (s _ {i}, \rho_ {- i}) ] = u _ {i} (s _ {i}, s _ {- i} ^ {\mu}) \forall s _ {i} \in S _ {i}
$$
C.1 Proof of Proposition 4.3
Proof. Since  $\mu$  is an  $\epsilon$ -CCE,  $\forall i \in N$ , we have
$$
\max _ {\rho_ {i} ^ {\prime} \in \mathrm {P} _ {i}} \mathbb {E} _ {\rho \sim \mu} \left[ u _ {i} (\rho_ {i} ^ {\prime}, \rho_ {- i}) \right] - \mathbb {E} _ {\rho \sim \mu} \left[ u _ {i} (\rho) \right] \leq \epsilon
$$
which implies (by Proposition C.1) that  $\forall i\in N$
$$
\begin{array}{l} \max _ {\rho_ {i} ^ {\prime} \in \mathrm {P} _ {i}} u _ {i} (\rho_ {i} ^ {\prime}, s _ {- i} ^ {\mu}) - \mathbb {E} _ {\rho \sim \mu} [ u _ {i} (\rho) ] \leq \epsilon \\ \Longrightarrow \max  _ {\rho_ {i} ^ {\prime} \in \mathrm {P} _ {i}} u _ {i} \left(\rho_ {i} ^ {\prime}, s _ {- i} ^ {\mu}\right) \leq \epsilon + \mathbb {E} _ {\rho \sim \mu} \left[ u _ {i} (\rho) \right]. \\ \end{array}
$$
Summing over  $N$ , we get
$$
\begin{array}{l} \sum_ {i \in N} \max  _ {\rho_ {i} ^ {\prime} \in \mathrm {P} _ {i}} u _ {i} \left(\rho_ {i} ^ {\prime}, s _ {- i} ^ {\mu}\right) \leq \sum_ {i \in N} \left(\epsilon + \mathbb {E} _ {\rho \sim \mu} [ u _ {i} (\rho) ]\right) (1) \\ = \sum_ {i \in N} \epsilon + \sum_ {i \in N} \mathbb {E} _ {\rho \sim \mu} [ u _ {i} (\rho) ] (2) \\ = \sum_ {i \in N} \epsilon + \mathbb {E} _ {\rho \sim \mu} \left[ \sum_ {i \in N} u _ {i} (\rho) \right] (3) \\ = n \epsilon + c (4) \\ = n \epsilon + \sum_ {i \in N} u _ {i} \left(s ^ {\mu}\right). (5) \\ \end{array}
$$
Where (4) and (5) use the fact that  $\forall \rho \in \mathrm{P},\sum_{i\in N}u_i(\rho) = c$  for some constant. The above inequalities give us
$$
\sum_ {i \in N} \max  _ {\rho_ {i} ^ {\prime} \in \mathrm {P} _ {i}} u _ {i} \left(\rho_ {i} ^ {\prime}, s _ {- i} ^ {\mu}\right) \leq n \epsilon + \sum_ {i \in N} u _ {i} \left(s ^ {\mu}\right).
$$
Rearranging, we get
$$
\sum_{i\in N}\underbrace{\max_{\rho^{\prime}_{i}\in\mathrm{P}_{i}}u_{i}(\rho^{\prime}_{i},s^{\mu}_{-i}) - u_{i}(s^{\mu})}_{\geq 0}\leq n\epsilon .
$$
All terms in the sum are non-negative because  $\rho_i^\prime$  is a best-response to  $s_{-i}^{\mu}$ . Then any particular term in the summation is upper bounded by  $n\epsilon$ .
C.2 Proof of Theorem 4.6
Proposition C.3. In two-player constant-sum games, for any  $\epsilon$ -Nash equilibrium  $s$  and player  $i$ , we have
$$
u _ {i} (s) - \min _ {s _ {- i} ^ {\prime} \in S _ {- i}} u _ {i} (s _ {i}, s _ {- i} ^ {\prime}) \leq \epsilon .
$$
The proof of Proposition C.3 is immediate from the fact that no player can gain more than  $\epsilon$  utility by deviating and the game is constant-sum.
Proof. First we show 1. Any marginal strategy  $s^\mu$  of a CCE  $\mu$  is a Nash equilibrium of  $G$  Cai et al. (2016). Then,
$$
\begin{array}{l} \operatorname {V u l} _ {i} \left(s ^ {\mu}, S _ {- i}\right) \doteq u _ {i} \left(s ^ {\mu}\right) - \min  _ {s _ {- i} \in S _ {- i}} u _ {i} \left(s _ {i} ^ {\mu}, s _ {- i}\right) \\ = \sum_ {(i, j) \in E _ {i}} u _ {i j} \left(s _ {i} ^ {\mu}, s _ {j} ^ {\mu}\right) - \min  _ {s _ {- i} \in S _ {- i}} \left(\sum_ {(i, j) \in E _ {i}} u _ {i j} \left(s _ {i} ^ {\mu}, s _ {j}\right)\right) \\ = \sum_ {(i, j) \in E _ {i}} u _ {i j} (s _ {i} ^ {\mu}, s _ {j} ^ {\mu}) - \sum_ {(i, j) \in E _ {i}} \min  _ {s _ {j} \in S _ {j}} u _ {i j} (s _ {i} ^ {\mu}, s _ {j}). \\ \end{array}
$$
Where the last line uses the fact that  $-i$  minimize  $i^{\prime}s$  utility, so can do so without coordinating since  $G$  is polymatrix. Continuing,
$$
\begin{array}{l} = \sum_ {(i, j) \in E _ {i}} \left(u _ {i j} \left(s _ {i} ^ {\mu}, s _ {j} ^ {\mu}\right) - \min  _ {s _ {j} \in S _ {j}} u _ {i} \left(s _ {i} ^ {\mu}, s _ {j}\right)\right) \\ \leq \sum_ {(i, j) \in E _ {i}} \gamma \\ \leq | E _ {i} | \gamma , \\ \end{array}
$$
where by  $(0,\gamma)$ -subgame stability of each  $G_{ij}$ ,  $(s_i^\mu, s_i^\mu)$  is a  $\gamma$ -Nash of  $G_{ij}$ . By Proposition C.3, we have  $u_{ij}(s_i^\mu, s_j^\mu) - \min_{s_j \in S_j} u_i(s_i^\mu, s_j) \leq \gamma$ .
C.3 Proof of Proposition 4.8
Proposition 4.8. In a  $\delta$ -CSP game  $G$  the following hold
1. Any  $CCE$  of  $G$  is a  $2\delta$ - $CCE$  of any  $\tilde{G} \in \mathrm{CSP}_{\delta}(G)$ .
2. The marginalized strategy profile of any CCE of  $G$  is a 2nδ-Nash equilibrium of any  $\tilde{G} \in \mathrm{CSP}_{\delta}(G)$ .
3. The marginalized strategy profile of any  $CCE$  is a  $2(n + 1)\delta$ -Nash equilibrium of  $G$
Proof. First we prove claim 1. Let  $\check{u}_i$  denote the utility function of  $i$  in  $\tilde{G}$ . Note that  $\forall \rho \in \mathrm{P}$  we have  $|\check{u}_i(\rho) - u_i(\rho)| \leq \delta \forall i \in N$ . Let  $\mu$  be any CCE of  $G$ . The definition of CCE states
$$
\mathbb {E} _ {\rho \sim \mu} \left[ u _ {i} \left(\rho_ {i} ^ {\prime}, \rho_ {- i}\right) - u _ {i} (\rho) \right] \leq 0 \quad \forall i \in N, \rho_ {i} ^ {\prime} \in \mathrm {P} _ {i}.
$$
It is sufficient to consider only player  $i$ . We can preserve the inequality by substituting  $\check{u}_i(\rho_i', \rho_{-i}) - \delta$  in place of  $u_i(\rho_i', \rho_{-i})$  and  $\check{u}_i(\rho) + \delta$  in place of  $u_i(\rho)$ . This gives us
$$
\begin{array}{l} \mathbb {E} _ {\rho \sim \mu} [ \tilde {u} _ {i} (\rho_ {i} ^ {\prime}, \rho_ {- i}) - \delta - (\tilde {u} _ {i} (\rho) + \delta) ] \leq 0 \quad \forall \rho_ {i} ^ {\prime} \in \mathrm {P} _ {i} \\ \Longrightarrow \mathbb {E} _ {\rho \sim \mu} \left[ \tilde {u} _ {i} \left(\rho_ {i} ^ {\prime}; \rho_ {- i}\right) - \tilde {u} _ {i} (\rho) \right] \leq 2 \delta \quad \forall \rho_ {i} ^ {\prime} \in \mathrm {P} _ {i}. \\ \end{array}
$$
Thus claim 1 is shown. Claim 2 is an immediate corollary of claim 1 and Proposition 4.3. Lastly, we show claim 3. By claim 2, we have the marginalized strategy of  $\mu$ ,  $s^{\mu}$ , is a  $2n\delta$ -Nash equilibrium of  $\check{G} \in \mathrm{CSP}_{\delta}(G)$ . That is for any  $i \in N$ ,
$$
\tilde {u} _ {i} \left(\rho_ {i} ^ {\prime}, s _ {- i} ^ {\mu}\right) - \tilde {u} _ {i} \left(s _ {i} ^ {\mu}, s _ {- i} ^ {\mu}\right) \leq 2 n \delta \quad \forall \rho_ {i} ^ {\prime} \in \mathrm {P} _ {i}.
$$
However, since  $G$  is  $\delta$ -CSP, we may substitute  $u_{i}(\rho_{i}^{\prime}, s_{-i}^{\mu}) - \delta$  in place of  $\check{u}_{i}(\rho_{i}^{\prime}, s_{-i}^{\mu})$  and  $u_{i}(s_{i}^{\mu}, s_{-i}^{\mu}) + \delta$  in place of  $\check{u}_{i}(s_{i}^{\mu}, s_{-i}^{\mu})$  as preserve the inequality.
$$
\left(u _ {i} \left(\rho_ {i} ^ {\prime}, s _ {- i} ^ {\mu}\right) - \delta\right) - \left(u _ {i} \left(s _ {i} ^ {\mu}, s _ {- i} ^ {\mu}\right) + \delta\right)) \leq 2 n \delta \quad \forall \rho_ {i} ^ {\prime} \in \mathrm {P} _ {i}.
$$
Rearranging, this gives us
$$
u _ {i} \left(\rho_ {i} ^ {\prime}, s _ {- i} ^ {\mu}\right) - u _ {i} \left(s _ {i} ^ {\mu}, s _ {- i} ^ {\mu}\right) \leq 2 n \delta + 2 \delta = 2 (n + 1) \delta \quad \forall \rho_ {i} ^ {\prime} \in \mathrm {P} _ {i}.
$$
C.4 Proof of Theorem 4.9
$$
\operatorname {V u l} _ {i} \left(s ^ {\mu}, S _ {- i}\right) \leq \left| E _ {i} \right| \gamma + 2 \delta \leq (n - 1) \gamma + 2 \delta ,
$$
The proof is largely the same as Theorem 4.6, with added approximation since  $G$  is no longer CSP.
Proof. Let  $\check{G}$  be a polymatrix game that is  $(2n\delta, \gamma)$ -subgame stable such that  $\check{G} \in \mathrm{CSP}_{\delta}(G)$ . Let  $\check{u}_i$  denote the utility function of  $i$  in  $\check{G}$ . By Proposition 4.8,  $\mu$  is a  $2n\delta$ -Nash equilibrium of  $\check{G}$ . Then,
$$
\begin{array}{l} \operatorname {V u l} _ {i} \left(s ^ {\mu}, S _ {- i}\right) \doteq u _ {i} \left(s ^ {\mu}\right) - \min  _ {s _ {- i} ^ {\prime} \in S _ {- i}} u _ {i} \left(s _ {i} ^ {\mu}, s _ {- i} ^ {\prime}\right) \\ \leq \check {u} _ {i} (s ^ {\mu}) - \min  _ {s _ {- i} ^ {\prime} \in S _ {- i}} \check {u} _ {i} (s _ {i} ^ {\mu}, s _ {- i} ^ {\prime}) + 2 \delta , \\ \end{array}
$$
since  $G$  is  $\delta$ -CSP. Then expanding  $\tilde{u}_i$  across  $i$ 's subgames we have
$$
\begin{array}{l} \sum_ {(i, j) \in E _ {i}} \check {u} _ {i j} \left(s _ {i} ^ {\mu}, s _ {j} ^ {\mu}\right) - \min  _ {s _ {- i} ^ {\prime} \in S _ {- i}} \sum_ {(i, j) \in E _ {i}} \check {u} _ {i} \left(s _ {i} ^ {\mu}, s _ {j}\right) + 2 \delta \\ = \sum_ {(i, j) \in E _ {i}} \check {u} _ {i j} \left(s _ {i} ^ {\mu}, s _ {j} ^ {\mu}\right) - \sum_ {(i, j) \in E _ {i}} \min  _ {s _ {j} ^ {\prime} \in S _ {j}} \check {u} _ {i} \left(s _ {i} ^ {\mu}, s _ {j} ^ {\prime}\right) + 2 \delta . \\ \end{array}
$$
Where, as in Theorem 4.6, the last line uses the fact that  $\check{G}$  is polymatrix,  $G_{ij}$  is constant-sum and  $-i$  minimize  $i^{\prime}s$  utility and can do so by without coordinating. Continuing, we have
$$
\begin{array}{l} \sum_ {(i, j) \in E _ {i}} \check {u} _ {i j} (s _ {i} ^ {\mu}, s _ {j} ^ {\mu}) - \sum_ {(i, j) \in E _ {i}} \min _ {s _ {j} ^ {\prime} \in S _ {j}} \check {u} _ {i} (s _ {i} ^ {\mu}, s _ {j} ^ {\prime}) + 2 \delta \\ = \sum_ {(i, j) \in E _ {i}} \left(\check {u} _ {i j} \left(s _ {i} ^ {\mu}, s _ {j} ^ {\mu}\right) - \min  _ {s _ {j} ^ {\prime} \in S _ {j}} \check {u} _ {i} \left(s _ {i} ^ {\mu}, s _ {j} ^ {\prime}\right)\right) + 2 \delta \\ \leq \sum_ {(i, j) \in E _ {i}} \gamma + 2 \delta \\ = \left| E _ {i} \right| \gamma + 2 \delta \\ \leq (n - 1) \gamma + 2 \delta . \\ \end{array}
$$
Where by  $(2n\delta, \gamma)$ -subgame stability of each  $G_{ij}$ ,  $(s_i^\mu, s_i^\mu)$  is a  $\gamma$ -Nash of  $G_{ij}$ . By Proposition C.3,  $s_i^\mu$  can lose at most  $\gamma$  to a worst case opponent  $s_j'$  in each subgame, since  $\tilde{G}_{ij}$  is two-player constant-sum.
C.5 Proof of Theorem 5.3
Theorem 5.3. If  $G$  is  $\delta$ -CSP in the neighborhood of  $S^{\times}(\mathcal{A}_N)$  and  $\exists \tilde{G} \in \mathrm{CSP}_{\delta}(G, S^{\times}(\mathcal{A}_N))$  that is  $\gamma$ -subgame stable in the neighborhood of  $S(\mathcal{A}_i)$ , then for any  $s \in S(\mathcal{A}_i)$
Proof. The proof is very similar to Theorem 4.9. Writing the definition of vulnerability we have
$$
\operatorname {V u l} _ {i} (s, S (\mathcal {A})) \doteq u _ {i} (s) - \min  _ {s _ {- i} ^ {\prime} \in S _ {- i} ^ {\times} (\mathcal {A} _ {N})} u _ {i} (s, s _ {- i} ^ {\prime}), \tag {6}
$$
since  $G$  is  $\delta$ -CSP in the neighborhood of  $S^{\times}(\mathcal{A}_N)$ . Swapping out the utility of  $u_{i}$  for  $\check{u}$ , we have
$$
(6) \leq \check {u} _ {i} (s) - \min  _ {s _ {- i} ^ {\prime} \in S _ {- i} ^ {\times} (\mathcal {A} _ {N})} \check {u} _ {i} \left(s _ {i}, s _ {- i} ^ {\prime}\right) + 2 \delta
$$
Since  $\tilde{G}$  is a polymatrix game,
$$
\begin{array}{l} \check {u} _ {i} (s) - \min  _ {s _ {- i} ^ {\prime} \in S _ {- i} ^ {\times} \left(\mathcal {A} _ {N}\right)} \check {u} _ {i} \left(s _ {i}, s _ {- i} ^ {\prime}\right) + 2 \delta (7) \\ = \sum_ {(i, j) \in E _ {i}} \check {u} _ {i j} \left(s _ {i}, s _ {j}\right) - \min  _ {s _ {- i} ^ {\prime} \in S _ {- i} ^ {\times} \left(\mathcal {A} _ {N}\right)} \sum_ {(i, j) \in E _ {i}} \check {u} _ {i j} \left(s _ {i}, s _ {j}\right) + 2 \delta (8) \\ = \left(\sum_ {(i, j) \in E _ {i}} \check {u} _ {i j} \left(s _ {i}, s _ {j}\right) - \min  _ {s _ {j} ^ {\prime} \in S _ {j} \left(A _ {j}\right)} \check {u} _ {i j} \left(s _ {i}, s _ {j} ^ {\prime}\right)\right) + 2 \delta . (9) \\ \end{array}
$$
Where, as in Theorem 4.6 and Theorem 4.9, the last line uses the fact that  $\check{G}$  is polymatrix,  $G_{ij}$  is constant-sum and  $-i$  minimize  $i$ 's utility and can do so by without coordinating.
Since  $\check{G}$  is  $\gamma$ -subgame stable in the neighborhood of  $S(\mathcal{A}_i)$  and  $s \in S(\mathcal{A}_i)$ , then means  $(s_i, s_j)$  is a  $\gamma$ -Nash for each subgame  $\check{G}_{ij}$ , so has bounded vulnerability within that subgame.
$$
\begin{array}{l} (9) \leq \left(\sum_ {(i, j) \in E _ {i}} \gamma\right) + 2 \delta \\ \leq \left| E _ {i} \right| \gamma + 2 \delta \\ \leq (n - 1) \gamma + 2 \delta \\ \end{array}
$$
D Normal-Form Algorithms
D.1 Computing Subgame Stability
Let  $\gamma$  be the minimum  $\gamma$  such that a CSP game  $G$  is  $(0, \gamma)$ -subgame stable. How do we compute  $\gamma$ ? Does it involve computing all equilibria of  $G$  and checking their subgame stability? The answer is no, it can be done in polynomial time in the number of pure strategies. We next provide an algorithm for computing  $\gamma$ . The algorithm involves solving a linear program for each edge in the graph and each pure strategy of those players. This linear program takes a pure strategy  $\rho_i'$ , and finds a Nash equilibrium of  $G$  that maximizes  $i$ 's incentive to deviate to  $\rho_i'$  when only considering their utility in  $G_{ij}$ ; call this quantity  $\gamma_{ij}^{\rho_i'}$ . If there are no such Nash equilibria the solver returns "infeasible". If the solver does not return "infeasible", we update  $\gamma = \max_{(i,j) \in E_i} \max_{\rho_i' \in P_i} \gamma_{ij}^{\rho_i'}$ .
Algorithm 2 Compute  $\gamma$
Input:  $G = (N,E,\mathrm{P},u)$  , a polymatrix game  $\gamma \leftarrow -\infty$    
for  $(i,j)\in E$  do for  $\rho_i^\prime \in \mathrm{P}_i$  do if LP1(i,j,  $\rho_{i}^{\prime}$  ) not infeasible then  $\gamma_{ij}^{\rho_i^{\prime}}\gets \mathrm{LP1}(i,j,\rho_i^{\prime})$ $\gamma \leftarrow \max (\gamma ,\gamma_{ij}^{\rho_i^{\prime}})$  end if end for for  $\rho_j^\prime \in \mathrm{P}_j$  do if LP1(j,i,  $\rho_j^\prime$  ) not infeasible then  $\gamma_{ji}^{\rho_j^{\prime}}\gets \mathrm{LP1}(j,i,\rho_j^{\prime})$ $\gamma \leftarrow \max (\gamma ,\gamma_{ji}^{\rho_j^{\prime}})$  end if end for end for return  $\gamma$
Let  $a_{i}(\rho_{i}^{\prime},\mu)$  be the advantage of deviating to  $\rho_i^\prime$  from a joint distribution over pure strategies:
$$
a _ {i} \left(\rho_ {i} ^ {\prime}, \mu\right) \doteq \sum_ {(i, j) \in E _ {i}} \underbrace {u _ {i j} \left(\rho_ {i} ^ {\prime} , s _ {j} ^ {\mu}\right)} _ {(a)} - \underbrace {\mathbb {E} _ {\rho \sim \mu} \left[ \sum_ {(i , j) \in E _ {i}} u _ {i j} \left(\rho_ {i} , \rho_ {j}\right) \right]} _ {(b)}
$$
Note that  $(a)$  is a linear function of  $\mu$ , since  $s_j^\mu$  is a marginal strategy.  $(b)$  is also a linear function of  $\mu$ , and so  $a_i(\rho_i', \mu)$  is a linear function of  $\mu$ . Likewise, let
$$
a _ {i j} \left(\rho_ {i} ^ {\prime}, \mu\right) \doteq u _ {i j} \left(\rho_ {i} ^ {\prime}, s _ {j} ^ {\mu}\right) - \mathbb {E} _ {\rho \sim \mu} \left[ u _ {i j} \left(\rho_ {i}, \rho_ {j}\right) \right].
$$
be the advantage of  $\rho_i'$  in the subgame between  $i$  and  $j$ .  $\mathrm{LP1}(i,j,\rho_i')$  is given below. The decision variables are the weights of  $\mu$  for each  $\rho \in \mathbb{P}$  and  $\gamma_{ij}^{\rho_i'}$ .
$$
\max  \gamma_ {i j} ^ {\rho_ {i} ^ {\prime}}
$$
s.t.  $a_{i}(\rho_{i},\mu)\leq 0\quad \forall i\in N,\rho_{i}\in \mathrm{P}_{i}$
$$
a _ {i j} \left(\rho_ {i} ^ {\prime}, \mu\right) \geq \gamma_ {i j} ^ {\rho_ {i} ^ {\prime}}
$$
$$
\sum_ {\rho \in \mathrm {P}} \mu (\rho) = 1
$$
$$
\mu (\rho) \in [ 0, 1 ] \quad \forall \rho \in \mathrm {P}
$$
We can get away with computing a CCE rather than targeting Nash equilibria because the marginals of any CCE are Nash equilibria in CSP games Cai et al. (2016).
The whole procedure runs in polynomial time in the size of the game. We need to solve an LP, which takes polynomial time, at most  $n^2 \max_{i \in N} |\mathrm{P}_i|$  times.
D.2 Finding Constant-Sum Polymatrix Decomposition
Projecting a game into the space of CSP games with minimum  $\delta$  can be done with a linear program. Let  $\underline{\delta}$  be the minimum  $\delta$  such that  $G$  is  $\delta$ -CSP. We give a linear program that finds  $\underline{\delta}$  and returns a CSP game  $\check{G} \in \mathrm{CSP}_{\underline{\delta}}(G)$ . The decision variables are the values of  $\check{u}_{ij}(\rho)$  for all  $i \neq j \in N, \rho \in \mathrm{P}, \underline{\delta}$  and constants for each subgame  $c_{ij}$ , for all  $i \neq j$ .
LP 2
$$
\min \underline {{\delta}}
$$
s.t.  $u_{i}(\rho) - \sum_{j\in -i}\check{u}_{ij}(\rho_{i},\rho_{j})\leq \underline{\delta}\quad \forall i\in N,\rho \in \mathrm{P}$
$$
u _ {i} (\rho) - \sum_ {j \in - i} \check {u} _ {i j} \left(\rho_ {i}, \rho_ {j}\right) \geq - \underline {{\delta}} \quad \forall i \in N, \rho \in P
$$
$$
\check {u} _ {i j} \left(\rho_ {i}, \rho_ {j}\right) + \check {u} _ {j i} \left(\rho_ {i}, \rho_ {j}\right) = c _ {i j} \quad \forall i \neq j \in N, \left(\rho_ {i}, \rho_ {j}\right) \in \mathrm {P} _ {i j},
$$
E Extensive-Form Games
Section 4 developed notions of approximately CSP and subgame stable in the context of normal-form games. Here, we apply these concepts to extensive-form games. While any extensive-form game has an equivalent induced normal-form game, analysing properties of an EFG through its induced normal is intractable for moderately-sized EFGs, since the size the normal-form representation is exponentially larger.
After background on extensive-form games, we introduce a novel "extensive-form version" of normal-form polymatrix games, which we call poly-EFGs. The major benefit of poly-EFGs over normal-form polymatrix games is their efficiently: poly-EFGs are exponentially more compact than an equivalent normal-form polymatrix game. The results of this section extend the theory of Sections 4 and 5 using this more efficient representation. We also give a proof-of-concept showing that poly-EFGs can be used to efficiently decompose extensive-form games by giving an algorithm for decomposing a perfect information EFG into a poly-EFG.
E.1 Background on Extensive-Form Games
We use the imperfect information extensive-form game (EFG) as a model for sequential multi-agent strategic situations. An imperfect information extensive-form game is a 10-tuple  $(N, \mathcal{A}, H, Z, A, P, u, \mathcal{I}, c, \pi_c)$  where  $N$  is a set of players;  $\mathcal{A}$  is a set of actions;  $H$  is a set of sequences of actions, called histories;  $Z \subseteq H$  is a set of terminal histories;  $A: H \to \mathcal{A}$  is a function that maps a history to available actions;  $P: H \to N$  is the player function, which assigns a player to choose an action at each non-terminal history;  $\mathfrak{u} = \{u_i\}_{i \in N}$  is a set of utility functions where  $u_i: Z \to \mathbb{R}$  is the utility function for player  $i$ ;  $\mathcal{I} = \{\mathcal{I}_i\}_{i \in N}$  where  $\mathcal{I}_i$  is a partition of the set  $\{h \in H: P(h) = i\}$  such that if  $h, h' \in I \in \mathcal{I}_i$  then  $A(h) = A(h')$ . We call an element  $I \in \mathcal{I}_i$  an information set. The chance player  $c$  has a function  $\pi_c(a, h) \forall h: P(h) = c$  which returns the probability of random nature events  $a \in \mathcal{A}$ . Let  $N_c = N \cup \{c\}$  be the set of players including chance.