SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For some history  $h$ , the  $j$ th action in  $h$  is written  $h_j$ . A sub-history of  $h$  from the  $j$ th to  $k$ th actions is denoted  $h_{j:k}$  and we use  $h_{:k}$  as a short-hand for  $h_{0:k}$ . If a history  $h'$  is a prefix of history  $h$ , we write  $h' \sqsubseteq h$  and if  $h'$  is a proper prefix of  $h$ , we write  $h' \sqsubset h$ .
A pure strategy in an EFG is a deterministic choice of actions for the player at every decision point. We use  $\rho_{i}:\mathcal{I}_{i}\to \mathcal{A}$  to denote a pure strategy of player  $i$ , and the set of all pure strategies as  $\mathrm{P}_i$ . Likewise,  $s_i\in \Delta (\mathrm{P}_i) = S_i$  is a mixed strategy, where  $\Delta (X)$  denotes the set of probability distributions over a domain  $X$ .
There are an exponential number of pure strategies in the number of information sets. A behavior strategy is a compact representation of the behavior of an agent that assigns a probability distribution over actions to each information set. We use  $\pi_i \in \Pi_i = (\Delta(A(I)))_{I \in \mathcal{I}_i}$  to denote a behavior strategy of player  $i$  and  $\pi_i(a, I)$  as the probability of playing action  $a$  at  $I$ . Let  $I(h)$  be the unique information set such that  $h \in I$ . We overload  $\pi_i(a, h) = \pi_i(a, I(h))$ . We use  $\rho \in \mathrm{P}, s \in S$  and  $\pi \in \Pi$  to denote pure, mixed and behavior strategy profiles, respectively. Note that  $\mathrm{P}$  is a subset of both  $\Pi$  and  $S$ .
Given a behavior strategy profile, let
$$
p_{i}(h_{1},h_{2},\pi_{i})\doteq \prod_{h_{1}\sqsubseteq ha\sqsubseteq h_{2},P(ha) = i}\pi_{i}(a,h)
$$
$$
p \left(h _ {1}, h _ {2}, \pi\right) \doteq \prod_ {i \in N _ {c}} p _ {i} \left(h _ {1}, h _ {2}, \pi_ {i}\right)
$$
$$
p _ {- i} (h _ {1}, h _ {2}, \pi_ {- i}) \doteq \prod_ {j \in N _ {c} \backslash \{i \}} p _ {j} (h _ {1}, h _ {2}, \pi_ {j})
$$
be the probability of transitioning from history  $h_1$  to  $h_2$  according to  $\pi_i, \pi$  and  $\pi_{-i}$ , respectively. Let  $p_i(z, \pi_i), p_{-i}(z, \pi_i)$  and  $p(z, \pi_i)$  be short-hands for  $p_i(\emptyset, z, \pi_i), p_{-i}(\emptyset, z, \pi_{-i})$  and  $p(\emptyset, z, \pi)$ , respectively, where  $\emptyset$  is the empty history.
We define the utility of a behavior strategy as:
$$
u _ {i} (\pi) \dot {=} \mathbb {E} _ {z \sim \pi} [ u _ {i} (z) ] = \sum_ {z \in Z} p (z, \pi) u _ {i} (z) = \sum_ {z \in Z} \left(\prod_ {i \in N _ {c}} p _ {i} (z, \pi_ {i})\right) u _ {i} (z).
$$
Perfect recall is a common assumption made on the structure of information sets in EFGs that prevents players from forgetting information they once possessed. Formally, for any  $h \in I$  let  $X_{i}(h)$  denote the set of  $(I, a)$  s.t.  $I \in \mathcal{I}_i$  and  $\exists h' \in I$  and  $h'a \sqsubseteq h$ . Let  $X_{-i}(h)$  be defined analogously for  $-i$  and  $X(h)$  for all players.
Definition E.1 (Perfect recall). If  $\forall I\in \mathcal{I}_i,\forall h,h'\in I,X_i(h) = X_i(h')$  then  $i$  has perfect recall. If all players possess perfect recall in some EFG  $G$ , we call  $G$  a game of perfect recall.
In games of perfect recall, the set of behavior strategies and mixed strategies are equivalent: any behavior strategy can be converted into a mixed strategy which is outcome equivalent over the set of terminal histories (i.e. has the same distribution over  $\bar{Z}$ ) and vice-versa.
Theorem E.2 (Kuhn (1953)). In games of perfect recall, any behavior strategy  $\pi_i$  has an equivalent mixed strategy  $s_i$  (and vice versa), such that
$$
p _ {i} (z, \pi_ {i}) = \mathbb {E} _ {\rho_ {i} \sim s _ {i}} \left[ p _ {i} (z, \rho_ {i}) \right].
$$
Theorem E.2 establishes a connection between equilibria in behavior and mixed strategies: a Nash equilibrium behavior strategy profile implies the equivalent mixed strategy profile is also a Nash equilibrium in mixed strategies and vice-versa.
We may also reduce any extensive-form game into an equivalent normal-form game.
Definition E.3 (Induced normal-form). The induced normal-form of an extensive-form game  $G$  (with utility functions  $u_{i}$ ) is a normal-form game  $G' = (N, P, u')$  such that  $u_i'(\rho) = u_i(\rho)$
The induced normal-form of an EFG has players making all decisions up-front. It is not always practical to construct an induced normal-form, but the concept is useful for proving things about EFGs.
E.2 Poly-EFGs
What is the appropriate extension of polymatrix games to EFGs? Given some  $n$ -player EFG  $G$ , what should the subgame between  $i$  and  $j$  be in the graph? Unlike in normal form games, players act sequentially, and  $i$  and  $j$  may either observe actions or have their utility impacted by other players. There is additional structure present in EFGs beyond the players' pure strategies.
The approach we take is to have an EFG for each pair of players  $i$  and  $j$ . For simplicity, we assume that each subgame  $G_{ij}^{\prime}$  shares the same structure as some  $n$ -player game  $G$ , but information sets where  $P(I) \notin \{i,j\}$  now belong to the chance player  $c$ .
Definition E.4 (Subgame). Let  $G = (N, \mathcal{A}, H, Z, A, P, u, \mathcal{I}, c, \pi_c)$  be some EFG. We define a subgame  $G_{ij}' = (\{i, j\}, \mathcal{A}, H, Z, A, P', (u_{ij}', u_{ji}', \mathcal{I}, c, \pi_c')$  as a structurally identical game to  $G$  between  $i$  and  $j$  with player function  $P'$  and utility functions  $(u_{ij}', u_{ji}')$ , then
$$
P ^ {\prime} (h) \doteq \left\{ \begin{array}{l l} P (h) & \quad \text {i f} P (h) \in \{i, j \} \\ c & \quad \text {o . w .} \end{array} \right.
$$
and let  $\pi_c^\prime$  be the strategy of the chance player in  $G_{ij}^{\prime}$ . We put no restrictions on  $\pi_c^\prime$ .
Note that  $u_{ij}^{\prime}, u_{ji}^{\prime}$  are not necessarily defined in the above definition. They may take any values. We merely want the subgame  $G_{ij}^{\prime}$  to share the structure of  $G$ . In  $G_{ij}^{\prime}$ ,  $i$  and  $j$ 's utility only depends on their strategies  $\pi_i, \pi_j$  and chance's actions  $\pi_c^{\prime}$ :
$$
\begin{array}{l} u _ {i j} ^ {\prime} \left(\pi_ {i}, \pi_ {j}\right) \doteq \mathbb {E} _ {z \sim \left(\pi_ {i}, \pi_ {j}, \pi_ {j}\right)} \left[ u _ {i j} ^ {\prime} (z) \right] \\ = \sum_ {z \in Z} p _ {i} (z, \pi_ {i}) p _ {j} (z, \pi_ {j}) p _ {c} (z, \pi_ {c} ^ {\prime}) u _ {i j} ^ {\prime} (z). \\ \end{array}
$$
What should  $\pi_c^\prime$  be defined as? This turns out to not matter very much. Given any subgame  $G_{ij}^{\prime}$  with chance strategy  $\pi_c^\prime$  and utility functions  $u_{ij}^{\prime}, u_{ji}^{\prime}$ , for any  $G_{ij}^{\prime \prime}$  with chance strategy  $\pi_c^{\prime \prime}$ , we can find  $u_{ij}^{\prime \prime}, u_{ji}^{\prime \prime}$  so that the utility of players between the two games will always be equal for any strategy profile  $(\pi_i, \pi_j)$ .
Definition E.5. We say  $\pi_c$  is fully mixed if  $\pi_c'(a,h) > 0 \forall h \in \{h \in H \mid P(h) = c\}, a \in A(h)$ .
Proposition E.6. Let  $G$  be an EFG and  $G_{ij}'$  a subgame between  $i$  and  $j$  with utility functions  $u_{ij}', u_{ji}'$  and fully mixed chance strategy  $\pi_c'$ . Given  $G_{ij}'$ , a subgame between  $i$  and  $j$  with fully mixed chance strategy  $\pi_c''$ , we may find  $u_{ij}', u_{ji}''$  such that  $\forall \pi_i, \pi_j$  we have  $u_{ij}'(\pi_i, \pi_j) = u_{ij}''(\pi_i, \pi_j)$  and  $u_{ji}'(\pi_i, \pi_j) = u_{ji}''(\pi_i, \pi_j)$ .
Proof. Note that  $p_c(z, \pi_c'), p_c(z, \pi_c'') \neq 0$ . Then  $\forall z \in Z$ , define  $u_{ij}'(z) = \frac{p_c(z, \pi_c')}{p_c(z, \pi_c'')} u_{ij}'(z)$  and  $u_{ji}'(z) = \frac{p_c(z, \pi_c')}{p_c(z, \pi_c'')} u_{ji}'(z)$ . Then
$$
\begin{array}{l} u _ {i j} ^ {\prime \prime} (\pi_ {i}, \pi_ {j}) = \sum_ {z \in Z} p _ {i} (z, \pi_ {i}) p _ {j} (z, \pi_ {j}) p _ {c} (z, \pi_ {c} ^ {\prime \prime}) u _ {i j} ^ {\prime \prime} (z) \\ = \sum_ {z \in Z} p _ {i} (z, \pi_ {i}) p _ {j} (z, \pi_ {j}) p _ {c} (z, \pi_ {c} ^ {\prime \prime}) \frac {p _ {c} (z , \pi_ {c} ^ {\prime})}{p _ {c} (z , \pi_ {c} ^ {\prime \prime})} u _ {i j} ^ {\prime} (z) \\ = \sum_ {z \in Z} p _ {i} (z, \pi_ {i}) p _ {j} (z, \pi_ {j}) p _ {c} (z, \pi_ {c} ^ {\prime}) u _ {i j} ^ {\prime} (z) \\ = u _ {i j} ^ {\prime} \left(\pi_ {i}, \pi_ {j}\right). \\ \end{array}
$$
For the remainder of this work, when defining a subgame between  $i$  and  $j$  given an EFG  $G$ , we define the subgame chance player's strategy to be equal to  $\pi_c$  in  $G$  at information sets  $I$  where  $P(I) = c$  in  $G$  and uniform randomly otherwise.
Having defined subgames, we may now define our representation of extensive-form polymatrix games.
Definition E.7 (Poly-EFG). A poly-EFG  $(N,E,\mathcal{G})$  is defined by a graph with nodes  $N$ , one for each player, a set of edges  $E$  and a set of games  $\mathcal{G} = \{G_{ij}\mid \forall (i,j)\in E\}$  where  $G_{ij}\in \mathcal{G}$  is a two player EFG between  $i$  and  $j$  and all  $G_{ij}\in \mathcal{G}$  are a subgame between  $i$  and  $j$  and all subgames are defined with respect to some EFG  $G$ .
Let  $G_{ij} \in \mathcal{G}$  denote the subgame between  $i$  and  $j$ . We use subscript  $ij$  (e.g.  $Z_{ij}$ ) to refer to parts of  $G_{ij}$ .
Since each subgame of the poly-EFG is the subgame of the same  $G$ , the space of pure, mixed and behavior strategies is the same for each subgame for any player. A player chooses a strategy (whether pure, mixed or behavior) and plays this strategy in each subgame. A player's global utility is the sum of their utility in each of their subgames. We have
$$
u _ {i} (\pi) = \sum_ {(i, j) \in E _ {i}} u _ {i j} (\pi_ {i}, \pi_ {j}),
$$
where  $E_{i} \subseteq E$  is the set of edges connected to  $i$  in the graph and  $u_{ij}$  is the utility function of  $i$  in subgame  $G_{ij}$ .
E.3 Constant-Sum and Subgame Stable Poly-EFGs
Here, we give definitions of constant-sum and subgame stability for poly-EFGs. These definitions are largely identical to their normal-form counterparts, we merely provide them here for completeness.
Poly-EFGs are constant-sum if, for each subgame, the utilities at the terminals add up to a constant.
Definition E.8 (Constant-sum). We say a poly-EFG  $G$  is constant-sum if  $\forall G_{ij} \in \mathcal{G}, z \in Z_{ij}$ ,  $u_{ij}(z) + u_{ji}(z) = c_{ij}$  where  $Z_{ij}$  is the set of terminal histories of  $\forall G_{ij}$  and  $c_{ij}$  is a constant.
We may also define approximate poly-EFGs in the same way as in normal-form games.
Definition E.9 ( $\delta$ -constant sum poly-EFG). An EFG  $G$  is  $\delta$ -constant sum poly-EFG ( $\delta$ -CSP) if there exists a constant-sum poly-EFG  $\check{G}$  with global utility function  $\check{u}$  such that  $\forall i \in N, \pi \in \Pi, |u_i(\pi) - \check{u}_i(\pi)| \leq \delta$ . We denote the set of such CSP games as  $\mathrm{CSP}_{\delta}(G)$ .
Finally, we define subgame stability for poly-EFGs. Our definitions are near-identical from the normal-form definitions.
Definition E.10 (Subgame stable profile). Let  $G$  be a poly-EFG. We say a strategy profile  $\pi$  is  $\gamma$ -subgame stable if  $\forall (i,j) \in E$ , we have  $(\pi_i, \pi_j)$  is a  $\gamma$ -Nash of  $G_{ij}$ .
Definition E.11 (Subgame stable game). Let  $G$  be a poly-EFG. We say  $G$  is  $(\epsilon, \gamma)$ -subgame stable if for any  $\epsilon$ -Nash equilibrium  $\pi$  of  $G$ ,  $\pi$  is  $\gamma$ -subgame stable.
E.4 Theoretical Results For Poly-EFGs
Our theoretical results from Sections 4 and 5 continue to hold in poly-EFGs. The idea is to use the induced normal-form of a poly-EFG. We assume perfect recall.
First, we will characterize what self-play will produce in EFGs. These are called marginal behavior strategies.
Definition E.12 (Marginal behavior strategy). Given some mediated equilibrium  $(\mu, (\Phi_i)_{i=1}^N)$ , let  $\pi_i^\mu$  be the marginal behavior strategy for  $i$  where  $\pi_i^\mu(a, I)$  is defined arbitrarily if  $\sum_{\rho_i' \in \mathrm{P}_i(I)} s_i^\mu(\rho_i') = 0$  and otherwise
$$
\pi_ {i} ^ {\mu} (a, I) \doteq \frac {\sum_ {\rho_ {i} \in \mathrm {P} _ {i} (a , I)} s _ {i} ^ {\mu} (\rho_ {i})}{\sum_ {\rho_ {i} ^ {\prime} \in \mathrm {P} _ {i} (I)} s _ {i} ^ {\mu} (\rho_ {i} ^ {\prime})} \quad \forall I \in \mathcal {I} _ {i}, a \in A (I),
$$
where  $s_i^\mu (\rho_i)\doteq \sum_{\rho_{-i}\in \mathrm{P}_{-i}}\mu (\rho_i,\rho_{-i})$
Definition E.13 (Marginal behavior strategy profile). Given some mediated equilibrium  $(\mu, (\Phi_i)_{i=1}^N)$ , let  $\pi^\mu$  be a marginal behavior strategy profile, where  $\pi_i^\mu$  is a marginal behavior strategy  $\forall i \in N$ .
Definition E.14 (Induced normal-form polymatrix game). Given a poly-EFG  $G = (N, E, \mathcal{G})$ , the induced normal-form polymatrix game is a polymatrix game  $G' = (N, E, P, u')$  such that  $P_i$  is equal to  $i$ 's set of pure strategies in each  $G_{ij}$  and  $u_{ij}'(\rho_i, \rho_j) = u_{ij}(\rho_i, \rho_j) = \sum_{z \in Z} p_i(z, \rho_i) p_j(z, \rho_j) p_c(z, \pi_c') u_{ij}(z)$  where  $u_{ij}$  is the utility function of  $i$  in  $G_{ij}$ .
In games of perfect recall, every behavior strategy  $\pi_i$  has an equivalent mixed strategy  $s_i^{\pi_i}$  (by Kuhn's Theorem), which means for any perfect recall EFG  $G$ , we can use the poly-EFG representation instead of turning  $G$  into a normal-form game then using a normal-form polymatrix game to get the same vulnerability bounds on  $G$ . Given  $\pi$ , let  $s^{\pi}$  be a profile of equivalent mixed strategies.
From Kuhn's Theorem and an assumption that each  $G_{ij} \in \mathcal{G}$  has perfect recall, we derive two immediate corollaries.
Corollary E.15. If  $\pi$  is  $\gamma$  subgame stable for a poly-EFG  $\tilde{G}$  where each subgame has perfect recall, then  $s^{\pi}$  is  $\gamma$  subgame stable in the induced normal-form polymatrix game of  $\tilde{G}$ .
Corollary E.16. For EFG of perfect recall  $G$ , if  $G$  is  $\delta$ -CSP-EFG then the induced normal form of  $G$  is  $\delta$ -CSP.
An extension of Theorem 4.9 holds for poly-EFGs. Let  $\Pi^{\mu}$  be the set of marginal behavior strategy profiles for any CCE of  $G$  and  $\Pi_i^\mu$  be the set of marginal behavior strategies for  $i$ .
Proposition E.17. If an  $EFG$  is  $\delta$ -CSP and  $\exists \tilde{G} \in \mathrm{CSP}_{\delta}(G)$  that is  $(2n\delta, \gamma)$ -subgame stable and  $\mu$  is a  $CCE^5$  of  $G$ , then
$$
\operatorname {V u l} _ {i} \left(\pi^ {\mu}, \Pi_ {- i}\right) \leq \left| E _ {i} \right| \gamma + 2 \delta \leq (n - 1) \gamma + 2 \delta ,
$$
Proof. Transform  $\tilde{G}$  into its induced normal-form polymatrix game  $\tilde{G}'$ . By Corollaries E.15 and E.16 the induced normal form of  $G$  is  $\delta$ -CSP and  $(2n\delta, \gamma)$ -subgame stable. By perfect recall, we can convert  $\pi^{\mu}$  to an equivalent mixed strategy profile  $s^{\mu}$  and do likewise with any  $\pi_{-i} \in \Pi_{-i}$ . Then apply Theorem 4.9 using  $s^{\mu}$ ,  $S_{-i}$  and the induced normal-form polymatrix game of  $\tilde{G}$  to bound vulnerability on  $G$ 's induced normal form, and hence  $G$ .
E.5 Vulnerability Against Self-Taught Agents in EFGs
Next we show an analogue of Theorem 5.3 for extensive-form games. In Section 5 we defined  $S(\mathcal{A})$  as the set of marginal strategy profiles for a no-regret learning algorithm  $\mathcal{A}$ . Many algorithms for EFGs will compute behavior strategies, so we use  $\Pi (\mathcal{A})\doteq \{\pi^{\mu}\mid (\mu ,(\Phi_{i})_{i\in N})\in \mathcal{M}(\mathcal{A})\}$  as the set of marginal behavior strategy profiles of  $\mathcal{M}(\mathcal{A})$  (recall that  $\mathcal{M}(\mathcal{A})$  is the set of mediated equilibria reachable by learning algorithm  $\mathcal{A}$ ). Then let  $\Pi_i(\mathcal{A})\doteq \{\pi_i\mid \pi \in \Pi (\mathcal{A})\}$  be the set of  $i$ 's marginal strategies from  $\Pi (\mathcal{A})$ . For example, if  $\mathcal{A}$  is CFR,  $\mathcal{M}(\mathcal{A})$  is the set of observably sequentially rational CFCCE Morrill et al. (2021b) and  $\Pi (\mathcal{A})$  is the set of behavior strategy profiles computable by CFR.
5 Note that "CCE" refers to a normal-form CCE (NFCCE) in the language of Farina et al. (2020).
Next, suppose each player  $i$  learns with their own self-play algorithm  $\mathcal{A}_i$ . Let  $\mathcal{A}_N \doteq (\mathcal{A}_1, \dots, \mathcal{A}_n)$  be the profile of learning algorithms;  $\Pi^{\times}(\mathcal{A}_N) \doteq \times_{i \in N} \Pi_i(\mathcal{A}_i)$  be the set of all possible match-ups between strategies learned in self-play by those learning algorithms and  $\Pi_{-i}^{\times}(\mathcal{A}_N) \doteq \times_{j \in -i} \Pi_j(\mathcal{A}_j)$  be the profiles of  $-i$  amongst these match-ups.
Definition E.18. We say a game  $G$  is  $\delta$ -CSP in the neighborhood of  $\Pi' \subseteq \Pi$  if there exists a constant sum poly-EFG  $\check{G}$  such that  $\forall \pi \in \Pi'$  we have  $|u_i(\pi) - \check{u}_i(\pi)| \leq \delta$ . We denote the set of such CSP games as  $\mathrm{CSP}_{\delta}(G, \Pi')$ .
Definition E.19. We say a poly-EFG game  $G$  is  $\gamma$ -subgame stable in the neighborhood of  $\Pi'$  if  $\forall \pi \in \Pi', \forall (i,j) \in E$  we have that  $(\pi_i, \pi_j)$  is a  $\gamma$ -Nash of  $G_{ij}$ .
Proposition E.20. If  $G$  is  $\delta$ -CSP in the neighborhood of  $\Pi^{\times}(\mathcal{A}_N)$  and  $\exists \check{G} \in \mathrm{CSP}_{\delta}(G, \Pi^{\times}(\mathcal{A}_N))$  that is  $\gamma$ -subgame stable in the neighborhood of  $\Pi(\check{\mathcal{A}}_i)$ , then for any  $\Pi \in \Pi(\mathcal{A}_i)$
$$
\operatorname {V u l} _ {i} \left(\pi , \Pi_ {- i} ^ {\times} (\mathcal {A} _ {N})\right) \leq | E _ {i} | \gamma + 2 \delta \leq (n - 1) \gamma + 2 \delta .
$$
The proof goes the same way as in Corollary E.17. Use the induced normal-form polymatrix game of  $\tilde{G}$  and Theorem 5.3 to derive bounds for the induced normal form of  $G$ , which then apply to  $G$ .
E.6 Leveraging the Poly-EFG Representation for Computing CSP Decompositions
The poly-EFG representation gives rise to more efficient algorithms for computing a poly-EFG decomposition. As a proof of concept, we show that in perfect information EFGs, we can write a linear program to compute the optimal polymatrix decomposition for an EFG that is exponentially smaller than LP 2 from Section D.2. Recall that  $\underline{\delta}$  is the minimum value of  $\delta$  such that a game is  $\delta$ -CSP. In an perfect information EFG, we can compute  $\underline{\delta}$  with the following LP. The decision variables are  $\underline{\delta}$ , the values of  $\check{u}_{ij}(z)\forall z\in Z,(i,j)\in E$  and  $c_{ij}\forall (i,j)\in E$ .
LP3
min  $\underline{\delta}$
s.t.  $u_{i}(z) - \sum_{j\in -i}\check{u}_{ij}(z)\leq \underline{\delta}\quad \forall i\in N,z\in Z$
$$
u _ {i} (z) - \sum_ {j \in - i} \check {u} _ {i j} (z) \geq - \underline {{\delta}} \quad \forall i \in N, z \in Z
$$
$$
\check {u} _ {i j} (z) + \check {u} _ {j i} (z) = c _ {i j} \quad \forall i \neq j \in N, z \in Z
$$
The trick is that in perfect information EFGs, each pure strategy profile leads to a single terminal. Hence, rather than having a constraint for each pure strategy profile, a constraint for each terminal will suffice. This leads to an exponential reduction in the number of constraints over LP 2.
F Details of SGDecompose
Next, we give the full details of SGDecompose. We give details here using our poly-EFG representation, since this is the representation we use in our experiments.
For each subgame  $\hat{G}_{ij}$  we store a single vector  $\check{u}_{ij}$  where the entry  $\check{u}_{ij}(z)$  gives the value of the utility for corresponding terminal  $z$ . We additionally store a constant  $\tilde{c}_{ij}$  for each subgame. Player  $i$  will use  $\check{u}_{ij}$  when computing  $\check{u}_{ij}(\pi_i, \pi_j)$ :
$$
\check {u} _ {i j} \left(\pi_ {i}, \pi_ {j}\right) = \sum_ {z \in Z} p _ {i} \left(z, \pi_ {i}\right) p _ {j} \left(z, \pi_ {j}\right) p _ {c} \left(z, \pi_ {c}\right) \check {u} _ {i j} (z).
$$
Whereas we compute  $\check{u}_{ji}(\pi_i,\pi_j)$  as follows.
$$
\check {u} _ {j i} \left(\pi_ {i}, \pi_ {j}\right) = \sum_ {z \in Z} p _ {i} (z, \pi_ {i}) p _ {j} (z, \pi_ {j}) p _ {c} (z, \pi_ {c}) \left(\check {c} _ {i j} - \check {u} _ {i j} (z)\right).
$$
For simplicity, let  $\tilde{u}$  denote the stacked vector of all  $\tilde{u}_{ij}$  and  $\check{c}_{ij}$ . We additionally initialize  $\tilde{G}$  as a fully connected graph.
The overall loss function which we minimize has two components: first,  $\mathcal{L}^{\delta}$  is the error between the utility functions of  $G$  and  $\tilde{G}$ ; it is a proxy for  $\delta$  in  $\delta$ -CSP.
$$
\begin{array}{l} \mathcal {L} ^ {\delta} (\pi ; \check {u}, u) \doteq \sum_ {i \in N} | \check {u} _ {i} (\pi) - u _ {i} (\pi) | \\ = \sum_ {i \in N} \left| \left(\sum_ {(i, j) \in E _ {i}} \check {u} _ {i j} (\pi_ {i}, \pi_ {j})\right) - u _ {i} (\pi) \right|. \\ \end{array}
$$
The other component of the overall loss function,  $\mathcal{L}^{\gamma}$ , measures the subgame stability. First, we define  $\mathcal{L}_{ij}^{\gamma}$ , which only applies to a single subgame.
$$
\begin{array}{l} \mathcal {L} _ {i j} ^ {\gamma} \left(\pi_ {i j}, \pi_ {i j} ^ {*}; \check {u}\right) \doteq \max  \left(\check {u} _ {i j} \left(\pi_ {i} ^ {*}, \pi_ {j}\right) - \check {u} _ {i j} \left(\pi_ {i j}\right), 0\right) \\ + \max  \left(\check {u} _ {j i} \left(\pi_ {i}, \pi_ {j} ^ {*}\right) - \check {u} _ {j i} \left(\pi_ {i j}\right), 0\right). \\ \end{array}
$$
Where  $\pi_{ij} = (\pi_i,\pi_j)$  is a profile and  $\pi_{ij}^{*} = (\pi_{i}^{*},\pi_{j}^{*})$  is a profile of deviations. Then
$$
\mathcal {L} ^ {\gamma} \left(\pi , \pi^ {*}; \check {u}\right) \doteq \sum_ {(i, j) \in E} \mathcal {L} _ {i j} ^ {\gamma} \left(\pi_ {i j}, \pi_ {i j} ^ {*}; \check {u}\right).
$$
Algorithm 3 shows how to compute a subgame stable constant-sum polymatrix decomposition via SGD. As input, the algorithm receives a game  $G$ , a finite set of strategy profiles  $\Pi'$ , learning rate  $\eta$ , number of training epochs  $T$ , hyperparameter  $\lambda \in [0,1]$  and batch size  $B$ . First, we initialize  $\Pi^{\times}$  as the set of all match-ups amongst strategies in  $\Pi'$ .
We then repeat the following steps for each epoch. First, we compute a best-response (for example, via sequence-form linear programming) to each strategy  $\pi_i'$  in  $\Pi'$  in each subgame; the full process is shown in Algorithm 4. After computing these best-responses for the current utility function of  $\check{G}$ , we try to fit  $\check{u}$  to be nearly CSP in the neighborhood of  $\Pi^{\times}$  and subgame stable in the neighborhood of  $\Pi'$ . Since  $\Pi^{\times}$  is exponentially larger than  $\Pi'$ , we partition it into batches, then use batch gradient descent. We use the following batch loss function, which computes the average values of  $\mathcal{L}^{\delta}$  and  $\mathcal{L}^{\gamma}$  over the batches, then weights the losses with  $\lambda$ . Let  $\Pi^b$  denote a batch of strategy profiles from  $\Pi^{\times}$  with size  $B$ , the overall loss function is
$$
\mathcal {L} \left(\Pi^ {b}, \Pi^ {\prime}, \Pi^ {*}; \check {u}, u\right) \doteq \frac {\lambda}{B} \sum_ {\pi \in \Pi^ {b}} \mathcal {L} ^ {\delta} (\pi ; \check {u}, u) + \frac {(1 - \lambda)}{| \Pi^ {\prime} |} \sum_ {\pi \in \Pi^ {\prime}} \sum_ {\pi^ {*} \in \Pi^ {*}} \mathcal {L} ^ {\gamma} (\pi , \pi^ {*}; \check {u}).
$$
We take this loss and find its gradient with respect to  $\tilde{u}$ , then update  $\tilde{u}$ :
$$
\check {u} \leftarrow \check {u} - \eta \cdot \nabla_ {\tilde {u}} \mathcal {L} \left(\Pi^ {b}, \Pi^ {\prime}, \Pi^ {*}; \check {u}, u\right).
$$
We found that in practise the gradient can be quite large relative to  $\check{u}$ , which has the potential to destabilize optimization. This is alleviated by normalizing the gradient by its  $l^2$  norm.
$$
g \leftarrow \nabla_ {\tilde {u}} \mathcal {L} \left(\Pi^ {b}, \Pi^ {\prime}, \Pi^ {*}; \check {u}, u\right)
$$
$$
\check {u} \leftarrow \check {u} - \eta \cdot \frac {g}{\| g \| _ {2}}
$$
The codebase for our experiments is available at https://github.com/RevanMacQueen/Self-Play-Polymatrix.
G.1 Leduc Poker