SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given a pair  $(t, t')$ , we use  $\mathbf{M}_{t,t'}$  to denote the full-dimensional  $(m \times n)$  matrix obtained from the matrix  $\mathbf{M}$  by keeping all entries indexed by  $\mathcal{I}_{\mathcal{X}}^{(t)}$  and  $\mathcal{I}_{\mathcal{Y}}^{(t')}$ , and zeroing out the rest. When in place of  $t$  or  $t'$  we use “:”, it corresponds to keeping as non-zero all rows (for the first index) or all columns (for the second index). In particular,  $\mathbf{M}_{t,:}$  is the matrix that keeps all rows of  $\mathbf{M}$  indexed by  $\mathcal{I}_{\mathcal{X}}^{(t)}$  intact and zeros out the rest. Further, notation  $\mathbf{M}_{t',t:s}$  is used to indicate that we select rows indexed by  $\mathcal{I}_{\mathcal{X}}^{(t')}$  and all columns of  $\mathbf{M}$  indexed by  $\mathcal{I}_{\mathcal{Y}}^{(t)}, \mathcal{I}_{\mathcal{Y}}^{(t+1)}, \ldots, \mathcal{I}_{\mathcal{Y}}^{(s)}$ , while we zero out the rest; similarly for  $\mathbf{M}_{t:s,t'}$ . Notation  $\mathbf{M}_{t',1:t}$  is used to indicate that we select rows indexed by  $\mathcal{I}_{\mathcal{X}}^{(t')}$  and all columns of  $\mathbf{M}$  indexed by  $\mathcal{I}_{\mathcal{Y}}^{(1)}, \mathcal{I}_{\mathcal{Y}}^{(2)}, \ldots, \mathcal{I}_{\mathcal{Y}}^{(t)}$ , while we zero out the rest; similarly for  $\mathbf{M}_{1:t,t'}$ . Given a vector  $\mathbf{x} \in \mathcal{X}$ ,  $\mathbf{x}^{(t)}$  denotes the entries of  $\mathbf{x}$  indexed by the elements of  $\mathcal{I}_{\mathcal{X}}^{(t)}$ ; similarly, for  $\mathbf{y} \in \mathcal{Y}$ ,  $\mathbf{y}^{(t)}$  denotes the entries of  $\mathbf{y}$  indexed by the elements of  $\mathcal{I}_{\mathcal{Y}}^{(t)}$ .
Additionally, we use  $\mathbf{M}^{(t,t^{\prime})}$  to denote the submatrix of  $\mathbf{M}$  obtained by selecting rows indexed by  $\mathcal{J}_{\mathcal{X}}^{(t)}$  and columns indexed by  $\mathcal{J}_{\mathcal{Y}}^{(t^{\prime})}$ .  $\mathbf{M}^{(t,t^{\prime})}$  is  $(p\times q)$ -dimensional, for  $p = \sum_{j\in \mathcal{J}_{\mathcal{X}}^{(t)}}|A_j|$  and  $q = \sum_{j\in \mathcal{J}_{\mathcal{Y}}^{(t^{\prime})}}|A_j|$ . Notation “:” has the same meaning as in the previous paragraph.
Dilated Regularizers. We assume access to strongly convex functions  $\phi : \mathcal{X} \to \mathbb{R}$  and  $\psi : \mathcal{Y} \to \mathbb{R}$  with known strong convexity parameters  $c_{\phi} > 0$  and  $c_{\psi} > 0$ , and that are continuously differentiable on the interiors of their respective domains. We further assume that these functions are nice as defined by Farina et al. [14]: their gradients and the gradients of their convex conjugates can be computed in time linear (or nearly linear) in the dimension of the treeplex.
A dilated regularizer is a framework for constructing nice regularizing functions for treexes. It makes use of the inductive characterization of a treeplex via Cartesian product and convex hull operations to generalize from the local simplex structure of the sequence-form polytope at a decision point to the entire sequence-form polytope. In particular, given a local "nice" regularizer  $\phi^j$  for each decision point  $j$ , a dilated regularizer for the treeplex can be defined as  $\phi(\mathbf{x}) = \sum_{j \in \mathcal{J}_X} x^{p_j} \phi^j \left( \frac{\mathbf{x}^j}{x^{p_j}} \right)$ .
The key property of these dilated regularizing functions is that the prox computations of the form  $\mathbf{x}_k = \operatorname{argmin}_{\mathbf{x} \in \mathcal{X}} \{\langle \mathbf{h}, \mathbf{x} \rangle + D_{\phi}(\mathbf{x}_k, \mathbf{x}_{k-1})\}$  decompose into bottom-up updates, where, up to a scaling factor, each set of coordinates from set  $\mathcal{J}_{\mathcal{X}}^{(t)}$  can be computed solely based on the coordinates of  $\mathbf{x}_k$  from sets  $\mathcal{J}_{\mathcal{X}}^{(1)}, \ldots, \mathcal{J}_{\mathcal{X}}^{(t-1)}$  and coordinates of  $\mathbf{g}$  from sets  $\mathcal{J}_{\mathcal{X}}^{(1)}, \ldots, \mathcal{J}_{\mathcal{X}}^{(t)}$ . Concretely, the recursive structure of the prox update is as follows (this was originally shown by [23], here we show a variation from Farina et al. [13]):
Proposition 2.2 (Farina et al. [13]). A prox update to compute  $\mathbf{x}_k$ , with gradient  $\mathbf{h}$  and center  $\mathbf{x}_{k-1}$  on a treeplex  $\mathcal{X}$  using a Bregman divergence constructed from a dilated DGF  $\phi$  can be decomposed into local prox updates at each decision point  $j \in \mathcal{J}_{\mathcal{X}}$  as follows:
$$
\mathbf{x}_{k}^{j} = \mathbf{x}_{k}^{p_{j}}\cdot \operatorname *{argmin}_{\mathbf{b}^{j}\in \Delta^{n_{j}}}\Bigg\{\left\langle \mathbf{h}^{j} + \hat{\mathbf{h}}^{j},\mathbf{b}^{j}\right\rangle +D_{\phi^{j}}\bigg(\mathbf{b}^{j},\frac{\mathbf{x}_{k - 1}^{j}}{\mathbf{x}_{k - 1}^{p_{j}}}\bigg)\Bigg\},
$$
$$
\hat {h} ^ {(j, a)} = \sum_ {j ^ {\prime} \in \mathcal {C} _ {j, a}} \left[ \phi^ {\downarrow j ^ {\prime *}} \big (- \mathbf {h} ^ {\downarrow j} + \nabla \phi^ {\downarrow j ^ {\prime}} (x _ {k - 1} ^ {\downarrow j ^ {\prime}}) \big) - \phi^ {j ^ {\prime}} \left(\frac {\mathbf {x} _ {k - 1} ^ {j ^ {\prime}}}{x _ {k - 1} ^ {(j , a)}}\right) + \left\langle \nabla \phi^ {j ^ {\prime}} \left(\frac {\mathbf {x} _ {k - 1} ^ {j ^ {\prime}}}{x _ {k - 1} ^ {(j , a)}}\right), \frac {\mathbf {x} _ {k - 1} ^ {j ^ {\prime}}}{x _ {k - 1} ^ {(j , a)}} \right\rangle \right].
$$
3 Extrapolated Cyclic Algorithm
Our extrapolated cyclic primal-dual algorithm is summarized in Algorithm 1. As discussed in Section 2, under the block partition and ordering that respects the treeplex ordering, the updates for  $\mathbf{x}_k^{(t)}$  in Line 9 (respectively,  $\mathbf{y}_k^{(t)}$  in Line 12), up to scaling by the value of their respective parent sequences, can be carried out using only the information about  $\frac{\mathbf{x}_k^j}{x_k^{p_j}}$  and  $\mathbf{h}_k^j$  (respectively,  $\frac{\mathbf{y}_k^j}{y_k^{p_j}}$  and  $\mathbf{g}_k^j$ ) for infosets  $j$  that are "lower" on the treeplex. The specific choices of the extrapolation sequences  $\widetilde{\mathbf{x}}_k$  and  $\widetilde{\mathbf{y}}_k$  that only utilize the information from prior cycles and the scaled values of  $\frac{\mathbf{x}_k^j}{x_k^{p_j}}$  and  $\frac{\mathbf{y}_k^j}{y_k^{p_j}}$  for infosets  $j$  updated up to the block  $t$  updates for  $\mathbf{x}_k$  and  $\mathbf{y}_k$  are what crucially enables us to decompose the updates for  $\mathbf{x}_k$  and  $\mathbf{y}_k$  into local block updates carried out in the bottom-up manner. At the end of the cycle, once  $\frac{\mathbf{x}_k^j}{x_k^{p_j}}$  and  $\frac{\mathbf{y}_k^j}{y_k^{p_j}}$  has been updated for all infosets, we can carry out a top-to-bottom update to fully determine vectors  $\mathbf{x}_k$  and  $\mathbf{y}_k$ , as summarized in the last two for loops in Algorithm 1. We present an implementation-specific version of the algorithm in Appendix D, which explicitly demonstrates that our algorithm's runtime does not have a dependence on the number of blocks used. In our analysis of the implementation-specific version of the algorithm, we argue that the per-iteration complexity of our algorithm matches that of MP. To support this analysis, we compare the empirical runtimes of our algorithm with MP and  $\mathrm{CFR}^+$  variants in Section 4.
Our convergence argument is built on the decomposition of the relaxed gap  $\mathrm{Gap}^{\mathbf{u},\mathbf{v}}(\mathbf{x}_k,\mathbf{v}_k)$  for arbitrary but fixed  $(\mathbf{u},\mathbf{v})\in \mathcal{X}\times \mathcal{Y}$  into telescoping and non-positive terms, which is common in first-order methods. The first idea that enables leveraging cyclic updates lies in replacing vectors  $\mathbf{M}\mathbf{x}_k$  and  $\mathbf{M}^\top \mathbf{y}_k$  by "extrapolated" vectors  $\mathbf{g}_k$  and  $\mathbf{h}_k$  that can be partially updated in a blockwise fashion as a cycle of the algorithm progresses, as stated in Proposition 3.1. To our knowledge, this basic idea originates in Song and Diakonikolas [39]. Unique to our work are the specific choices of  $\mathbf{g}_k$  and  $\mathbf{h}_k$ , which leverage all the partial information known to the algorithm up to the current iteration and block update. Crucially, we leverage the treeplex structure to show that our chosen updates are sufficient to bound the error sequence  $\mathcal{E}_k$  and obtain the claimed convergence bound in Theorem 3.2. Due to space constraints, the proof is deferred to Appendix C.
To simplify the exposition, we introduce the following notation:
$$
\mathbf {M} _ {\mathbf {x}} := \sum_ {t = 1} ^ {s - 1} \mathbf {M} _ {t, t + 1: s}, \quad \mathbf {M} _ {\mathbf {y}} := \mathbf {M} - \mathbf {M} _ {\mathbf {x}} = \sum_ {t = 1} ^ {s} \mathbf {M} _ {t: s, t}; \tag {3.1}
$$
$$
\mu_ {x} := \left\| \mathbf {M} _ {\mathbf {x}} \right\| _ {*} + \left\| \mathbf {M} _ {\mathbf {y}} \right\| _ {*}, \quad \mu_ {y} := \left\| \mathbf {M} _ {\mathbf {x}} ^ {\top} \right\| _ {*} + \left\| \mathbf {M} _ {\mathbf {y}} ^ {\top} \right\| _ {*}.
$$
When the norm of the space is  $\| \cdot \| = \| \cdot \|_1$ , both  $\mu_x$  and  $\mu_y$  are bounded above by  $2\max_{i,j} |M_{ij}|$ .
The next proposition decomposes the relaxed gap into an error term and telescoping terms. The proposition is independent of the specific choices of extrapolated vectors  $\mathbf{g}_k, \mathbf{h}_k$ .
Algorithm 1 Extrapolated Cyclic Primal-Dual EFG Solver (ECyclicPDA)
1: Initialization:  $\mathbf{x}_0\in \mathcal{X},\mathbf{y}_0\in \mathcal{Y},\eta_0 = H_0 = 0,\eta = \frac{\sqrt{C_\phi C_\psi}}{\mu_x + \mu_y}$ $\bar{\mathbf{x}}_0 = \mathbf{x}_0,\bar{\mathbf{y}}_0 = \mathbf{y}_0,\mathbf{g}_0 = \mathbf{0},\mathbf{h}_0 = \mathbf{0}$    
2: for  $k = 1:K$  do   
3: Choose  $\eta_k\leq \eta ,H_k = H_{k - 1} + \eta_k$    
4:  $\mathbf{g}_k = \mathbf{g}_{k - 1},\mathbf{h}_k = \mathbf{h}_{k - 1}$    
5:  $\widetilde{\mathbf{x}}_k = \mathbf{x}_{k - 1} + \frac{\eta_{k - 1}}{\eta_k} (\mathbf{x}_{k - 1} - \mathbf{x}_{k - 2}),\widetilde{\mathbf{y}}_k = \mathbf{y}_{k - 1} + \frac{\eta_{k - 1}}{\eta_k} (\mathbf{y}_{k - 1} - \mathbf{y}_{k - 2})$    
6: for  $t = 1:s$  do   
7:  $\begin{array}{r}\mathbf{h}_k^{(t)} = (\mathbf{M}^{(:,t)})^\top \widetilde{\mathbf{y}}_k \end{array}$    
8:  $\mathbf{x}_k^{(t)} = \left[\operatorname *{argmin}_{\mathbf{x}\in \mathcal{X}}\left\{\eta_k\langle \mathbf{x},\mathbf{h}_k\rangle +D_\phi (\mathbf{x},\mathbf{x}_{k - 1})\right\} \right]^{(t)}$    
9:  $\widetilde{\mathbf{x}}_k^{(t)} = \left[\frac{\mathbf{x}_k^j}{x_k^{p_j}} x_{k - 1}^{p_j} + \frac{\eta_{k - 1}}{\eta_k}\left(\mathbf{x}_k^j -\frac{\mathbf{x}_{k - 1}^j}{x_{k - 1}^{p_j}} x_{k - 2}^{p_j}\right)\right]_{j\in \mathcal{J}_X^{(t)}}$    
10:  $\begin{array}{rl} & {\mathbf{g}_k^{(t)} = \mathbf{M}^{(t,:)}\widetilde{\mathbf{x}}_k}\\ & {\mathbf{y}_k^{(t)} = \left[\operatorname *{argmax}_{\mathbf{v}\in \mathcal{V}}\left\{\eta_k\langle \mathbf{g}_k,\mathbf{v}\rangle -D_\psi (\mathbf{v},\mathbf{y}_{k - 1})\right\} \right]^{(t)}}\\ & {\widetilde{\mathbf{y}}_k^{(t)} = \left[\frac{\mathbf{y}_k^j}{y_k^{p_j}} y_{k - 1}^{p_j} + \frac{\eta_{k - 1}}{\eta_k}\left(\mathbf{y}_k^j -\frac{\mathbf{y}_{k - 1}^j}{y_{k - 1}^{p_j}} y_{k - 2}^{p_j}\right)\right]_{j\in \mathcal{J}_Y^{(t)}}} \end{array}$    
13: for  $j\in J_{X}$  do   
14:  $\begin{array}{r}\mathbf{x}_k^j = x_k^{p_j}\cdot (\frac{\mathbf{x}_k^j}{x_k^{p_j}}) \end{array}$    
15: for  $j\in J_{Y}$  do   
16:  $\begin{array}{r}\mathbf{y}_k^j = y_k^{p_j}\cdot (\frac{\mathbf{y}_k^j}{y_k^{p_j}}) \end{array}$    
17:  $\overline{\mathbf{x}}_k = \frac{H_k - \eta_k}{H_k}\overline{\mathbf{x}}_{k - 1} + \frac{\eta_k}{H_k}\mathbf{x}_k,\overline{\mathbf{y}}_k = \frac{H_k - \eta_k}{H_k}\overline{\mathbf{y}}_{k - 1} + \frac{\eta_k}{H_k}\mathbf{y}_k$    
18: Return:  $\overline{\mathbf{x}}_K,\overline{\mathbf{y}}_K$
Proposition 3.1. Let  $\mathbf{x}_k, \mathbf{y}_k$  be the iterates of Algorithm 1 for  $k \geq 1$ . Then, for all  $k \geq 1$ ,  $\mathbf{x}_k \in \mathcal{X}$ ,  $\mathbf{y}_k \in \mathcal{Y}$ , we have
$$
\eta_ {k} \operatorname {G a p} ^ {\mathbf {u}, \mathbf {v}} \left(\mathbf {x} _ {k}, \mathbf {y} _ {k}\right) \leq \mathcal {E} _ {k} - D _ {\phi} (\mathbf {u}, \mathbf {x} _ {k}) + D _ {\phi} (\mathbf {u}, \mathbf {x} _ {k - 1}) - D _ {\psi} (\mathbf {v}, \mathbf {y} _ {k}) + D _ {\psi} (\mathbf {v}, \mathbf {y} _ {k - 1}),
$$
where the error sequence  $\mathcal{E}_k$  is defined by
$$
\mathcal {E} _ {k} := \eta_ {k} \left\langle \mathbf {M} \mathbf {x} _ {k} - \mathbf {g} _ {k}, \mathbf {v} - \mathbf {y} _ {k} \right\rangle - \eta_ {k} \left\langle \mathbf {u} - \mathbf {x} _ {k}, \mathbf {M} ^ {\top} \mathbf {y} _ {k} - \mathbf {h} _ {k} \right\rangle - D _ {\psi} (\mathbf {y} _ {k}, \mathbf {y} _ {k - 1}) - D _ {\phi} (\mathbf {x} _ {k}, \mathbf {x} _ {k - 1}).
$$
To obtain our main result, we leverage the blockwise structure of the problem, the bilinear structure of the objective, and the treeplex structure of the feasible sets to control the error sequence  $\mathcal{E}_k$ . A key property that enables this result is that normalized entries  $\mathbf{x}_k^j / x_{k-1}^{p_j}$  from the same information set belong to a probability simplex. This property is crucially used in controlling the error of the extrapolation vectors. The main result is summarized in the following theorem.
Theorem 3.2. Consider the iterates  $\mathbf{x}_k, \mathbf{y}_k$  for  $k \geq 1$  in Algorithm 1 and the output primal-dual pair  $\bar{\mathbf{x}}_K, \bar{\mathbf{y}}_K$ . Then,  $\forall k \geq 1$ ,
$$
\frac {\mu_ {x} D _ {\phi} \left(\mathbf {x} ^ {*} , \mathbf {x} _ {K}\right) + \mu_ {y} D _ {\psi} \left(\mathbf {y} ^ {*} , \mathbf {y} _ {K}\right)}{\mu_ {x} + \mu_ {y}} \leq D _ {\phi} \left(\mathbf {x} ^ {*}, \mathbf {x} _ {0}\right) + D _ {\psi} \left(\mathbf {y} ^ {*}, \mathbf {y} _ {0}\right), a n d, f u r t h e r,
$$
$$
\operatorname {G a p}(\bar{\mathbf{x}}_{K},\bar{\mathbf{y}}_{K}) = \sup_{\mathbf{u}\in \mathcal{X},\mathbf{v}\in \mathcal{Y}}\{\langle \mathbf{M}\bar{\mathbf{x}}_{K},\mathbf{v}\rangle -\langle \mathbf{M}\mathbf{u},\bar{\mathbf{y}}_{K}\rangle \} \leq \frac{\sup_{\mathbf{u}\in\mathcal{X},\mathbf{v}\in\mathcal{Y}}\left\{D_{\phi}(\mathbf{u},\mathbf{x}_{0}) + D_{\psi}(\mathbf{v},\mathbf{y}_{0})\right\}}{H_{K}}.
$$
In the above bound, if  $\forall k\geq 1$ $\eta_{k} = \eta = \frac{\sqrt{c_{\phi}c_{\psi}}}{\mu_{x} + \mu_{y}}$  then  $H_{K} = K\eta$  . As a consequence, for any  $\epsilon >0$ $\mathrm{Gap}(\overline{\mathbf{x}}_K,\overline{\mathbf{y}}_K)\leq \epsilon$  after at most  $\left\lceil \frac{(\mu_x + \mu_y)(\sup_{\mathbf{u}\in\mathcal{X},\mathbf{v}\in\mathcal{Y}}\{D_\phi(\mathbf{u},\mathbf{x}_0) + D_\psi(\mathbf{v},\mathbf{y}_0)\})}{\sqrt{c_\phi c_\psi}\epsilon}\right\rceil$  iterations.
4 Experimental Evaluation and Discussion
We evaluate the performance of ECyclicPDA instantiated with three different dilated regularizers: dilated entropy [25], dilatable global entropy [14], and dilated  $\ell^2$  [13]. In the case of the dilated  $\ell^2$
regularizer, we use dual averaging of the "extrapolated" vectors  $\mathbf{g}_k$  and  $\mathbf{h}_k$  in our algorithm, since otherwise we have no guarantee that the iterates would remain in the relative interior of the domain of the dilated DGF, and the Bregman divergence may become undefined. We compare our method to MP, which is state-of-the-art among first-order methods for EFG solving. We test ECyclicPDA and MP with three different averaging schemes: uniform, linear, and quadratic averaging since Gao et al. [20] suggest that these different averaging schemes can lead to faster convergence in practice. We also compare against empirical state-of-the-art  $\mathrm{CFR}^+$  variants:  $\mathrm{CFR}^+$  [41], and the predictive  $\mathrm{CFR}^+$  variant (PCFR $^+$ ) [15]. We emphasize that our method achieves the same  $O\left(\frac{1}{T}\right)$  average-iterate convergence rate as MP, and that all the  $\mathrm{CFR}^+$  variants have the same suboptimal  $O\left(\frac{1}{\sqrt{T}}\right)$  average-iterate convergence rate. We experiment on four standard benchmark games for EFG solving: Goofspiel (4 ranks), Liar's Dice, Leduc (13 ranks), and Battleship. In all experiments, we run for 10,000 full (or equivalent) gradient computations. This corresponds to 5,000 iterations of ECyclicPDA,  $\mathrm{CFR}^+$ , and  $\mathrm{PCFR}^+$ , and 2,500 iterations of MP. $^2$  A description of all games is provided in Appendix E. Additional experimental details are provided in Appendix G.
For each instantiation of ECyclicPDA considered on a given game (choice of regularizer, averaging, and block construction strategy) the stepsize is tuned by taking power of 2 multiples of  $\eta$  ( $2^l \cdot \eta$  for  $l \in \mathbb{N}$ ), where  $\eta$  is the theoretical stepsize stated in Theorem 3.2, and then choosing the stepsize  $\eta^*$  among these multiples of  $\eta$  that has the best performance. Within the algorithm, we use a constant stepsize, letting  $\eta_k = \eta_0$  for all  $k$ . We apply the same tuning scheme for MP stepsizes (for a given choice of regularizer and averaging). Note that this stepsize tuning is coarse, and so it is possible that better results can be achieved for ECyclicPDA and MP using finer stepsize tuning.
We test our algorithm with four different block construction strategies. The single block construction strategy puts every decision point in a single block, and thus it corresponds to the non-block-based version of ECyclicPDA. The children construction strategy iterates through the decision points of the treeplex bottom-up (by definition, this will respect the treeplex ordering), and placing each set of decision points that have parent sequences starting from the same decision point in its own block. The postorder construction strategy iterates through the decision points bottom-up (again, by definition, this will respect the treeplex ordering). The order is given by a postorder traversal of the treeplex, treating all decision points that have the same parent sequence as a single node (and when the node is processed, all decision points are sequentially added to the block). It greedily makes blocks as large as possible, while only creating a new block if it causes a parent decision point and child decision point to end up in the same block. The infosets construction strategy places each decision point in its own block. We provide further description of the block construction strategies in Appendix F.
We show the results of different block construction strategies in Figure 1. For each block construction strategy, ECyclicPDA is instantiated with the choice of regularizer and averaging that yields the fastest convergence among all choices of parameters. We can see that the different block construction strategies do not make a significant difference in Goofspiel (4 ranks) or in Leduc (13 ranks). However, we see benefits of using blocks in Liar's Dice and Battleship. In Liar's Dice, children and postorder have a clear advantage, and children outperforms the other block construction strategies in Battleship.
We show the results of comparing our algorithm against MP,  $\mathrm{CFR}^+$ , and  $\mathrm{PCFR}^+$  in Figure 2. ECyclicPDA is instantiated with the choice of regularizer, averaging, and block construction strategy that yields the fastest convergence among all choices for ECyclicPDA, and MP is instantiated with the choice of regularizer and averaging that yields the fastest convergence among all choices for MP. We see that ECyclicPDA performs better than MP in all games besides Goofspiel (4 ranks), where they perform about the same. In Liar's Dice and Battleship, the games where ECyclicPDA benefits from having multiple blocks, we see competitiveness with  $\mathrm{CFR}^+$  and  $\mathrm{PCFR}^+$ . In particular, in Liar's Dice, ECyclicPDA is overtaking  $\mathrm{CFR}^+$  at 10,000 gradient computations. On Battleship, we see that both ECyclicPDA and MP outperform  $\mathrm{CFR}^+$ , and that ECyclicPDA is competitive with  $\mathrm{PCFR}^+$ .
Restarting. We now introduce restarting as a heuristic tool for speeding up EFG solving. While restarting is only known to lead to a linear convergence rate in the case of using the  $\ell_2$  regularizer in certain FOMs [5, 21], we apply restarting as a heuristic across our methods based on dilated regularizers and to CFR-based methods. To the best of our knowledge, restarting schemes have not been empirically evaluated on EFG algorithms such as MP,  $\mathrm{CFR}^+$ , or (obviously), our new method. We implement restarting by resetting the averaging process when the duality gap has halved since the last time the averaging process was reset. After resetting, the initial iterate is set equal to the last
Here we count one gradient evaluation for  $\mathbf{x}$  and one for  $\mathbf{y}$  as two gradient evaluations total.
Figure 1: Duality gap as a function of the number of full (or equivalent) gradient computations for ECyclicPDA with different block construction strategies.
Figure 2: Duality gap as a function of the number of full (or equivalent) gradient computations for ECyclicPDA, MP,  $\mathrm{CFR}^+$ ,  $\mathrm{PCFR}^+$ .
iterate we saw before restarting. Since the restarting heuristic is one we introduce, we distinguish restarted variants in the plots by preponding the name of the algorithm with "r". For example, when restarting is applied to  $\mathrm{CFR}^+$ , the label used is  $\mathbf{rCFR}^+$ .
We show the results of different block construction strategies when restarting is used on ECyclicPDA in Figure 3. As before, we take the combination of regularizer and averaging scheme that works best. Again, we can see that the different block construction strategies do not make a significant difference in Goofspiel (4 ranks) or in Leduc (13 ranks), while making a difference for Liar's Dice and Battleship. However, with restarting, the benefit of the children and postorder for Liar's Dice and Battleship is even more pronounced relative to the other block construction strategies; the gap is several orders of magnitude after  $10^{4}$  gradient computations. Note that while children performed worse than single block for Battleship previously, with restarting, children performs much better.
Finally, we compare the performance of the restarted version of our algorithm, with restarted versions of MP,  $\mathrm{CFR}^{+}$ , and  $\mathrm{PCFR}^{+}$  in Figure 4. As before, we take the combination of regularizer, averaging scheme, and block construction strategy that works best for ECyclicPDA, and the combination of regularizer and averaging scheme that works best for MP. Firstly, we note that the scale of the y-axis is different from Figure 2 for all games besides Leduc (13 ranks), because restarting tends to hit much higher levels of precision. We see that restarting provides significant benefits for  $\mathrm{PCFR}^{+}$  in Goofspiel (4 ranks) allowing it to converge to numerical precision, while the other algorithms do not benefit much. In Liar's Dice, restarted  $\mathrm{CFR}^{+}$  and  $\mathrm{PCFR}^{+}$  converge to numerical precision within 200 gradient computations, and restarted ECyclicPDA converges to numerical precision at  $10^{4}$  gradient computations. Additionally, restarted MP achieves a much lower duality gap. For Battleship, ECyclicPDA, MP, and  $\mathrm{PCFR}^{+}$  all benefit from restarting, and restarted ECyclicPDA is competitive with restarted  $\mathrm{PCFR}^{+}$ . Similar to the magnification in benefit of using blocks versus not using blocks when restarting in Liar's Dice and Battleship, we see that restarted ECyclicPDA achieves significantly better duality gap than MP in these games.
Wall-clock time experiments. In Table 1, we show the average wall-clock time per iteration of our algorithm instantiated with different block construction strategies as well as MP,  $\mathrm{CFR}^+$ , and  $\mathrm{PCFR}^+$ . It is clear from this table that the runtimes are pretty similar to each other, and this is without extensive optimization of our particular implementation. Clearly, our algorithm is at least as fast as MP per "full" gradient computation. Since the computational bottleneck of gradient and prox computations becomes apparent in bigger games, Battleship demonstrates the speed of our algorithm relative to MP best.