SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Block-Coordinate Methods and Restarting for Solving Extensive-Form Games*
Darshan Chakrabarti
IEOR Department
New York, NY 10025
dc3595@columbia.edu
Jelena Diakonikolas
Madison, WI 53706
jelena@cs.wisc.edu
Christian Kroer
ck2945@columbia.edu
Coordinate descent methods are popular in machine learning and optimization for their simple sparse updates and excellent practical performance. In the context of large-scale sequential game solving, these same properties would be attractive, but until now no such methods were known, because the strategy spaces do not satisfy the typical separable block structure exploited by such methods. We present the first cyclic coordinate-descent-like method for the polytope of sequence-form strategies, which form the strategy spaces for the players in an extensive-form game (EFG). Our method exploits the recursive structure of the proximal update induced by what are known as dilated regularizers, in order to allow for a pseudo block-wise update. We show that our method enjoys a  $O(1 / T)$  convergence rate to a two-player zero-sum Nash equilibrium, while avoiding the worst-case polynomial scaling with the number of blocks common to cyclic methods. We empirically show that our algorithm usually performs better than other state-of-the-art first-order methods (i.e., mirror prox), and occasionally can even beat  $\mathrm{CFR}^+$ , a state-of-the-art algorithm for numerical equilibrium computation in zero-sum EFGs. We then introduce a restarting heuristic for EFG solving. We show empirically that restarting can lead to speedups, sometimes huge, both for our cyclic method, as well as for existing methods such as mirror prox and predictive  $\mathrm{CFR}^+$ .
Extensive-form games (EFGs) are a broad class of game-theoretic models which are played on a tree. They can compactly model both simultaneous and sequential moves, private and/or imperfect information, and stochasticity. Equilibrium computation for a two-player zero-sum EFG can be formulated as the following bilinear saddle-point problem (BSPP)
$$
\min  _ {\mathbf {x} \in \mathcal {X}} \max  _ {\mathbf {y} \in \mathcal {Y}} \left\langle \mathbf {M x}, \mathbf {y} \right\rangle . \tag {PD}
$$
Here, the set of strategies  $\mathcal{X},\mathcal{Y}$  for the  $\mathbf{x}$  and  $\mathbf{y}$  players are convex polytopes known as sequence-form polytopes [44]. The (PD) formulation lends itself to first-order methods (FOMs) [14, 25], linear
*Authors are ordered alphabetically.
programming [44], and online learning-based approaches [6, 8, 15, 17, 42, 49], since the feasible sets are convex and compact polytopes, and the objective is bilinear.
A common approach for solving BSPPs is by using first-order methods, where local gradient information is used to iteratively improve the solution in order to converge to an equilibrium asymptotically. In the game-solving context, such methods rely on two oracles: a first-order oracle that returns a (sub)gradient at the current pair of strategies, and a pair of prox oracles for the strategy spaces  $\mathcal{X},\mathcal{Y}$ , which allow one to perform a generalized form of projected gradient descent steps on  $\mathcal{X},\mathcal{Y}$ . These prox oracles are usually constructed through the choice of an appropriate regularizer. For EFGs, it is standard to focus on regularizers for which the prox oracle can be computed in linear time with respect to the size of the polytope, which is only known to be achievable through what is known as dilated regularizers [23]. Most first-order methods for EFGs require full-tree traversals for the first-order oracle, and full traversals of the decision sets for the prox computation, before making a strategy update for each player. For large EFGs these full traversals, especially for the first-order oracle, can be very expensive, and it may be desirable to make strategy updates before a full traversal has been performed, in order to more rapidly incorporate partial first-order information.
In other settings, one commonly used approach for solving large-scale problems is through coordinate methods (CMs) [34, 46]. These methods involve computing the gradient for a restricted set of coordinates at each iteration of the algorithm, and using these partial gradients to construct descent directions. The convergence rate of these methods typically is able to match the rate of full gradient methods. However, in some cases they may exhibit worse runtime due to constants introduced by the method. In spite of this, they often serve practical benefits of being more time and space efficient, and enabling distributed computation [2, 3, 11, 19, 22, 28, 30, 32, 34, 47, 48].
Generally, coordinate descent methods assume that the problem is separable, i.e., there exists a partition of the coordinates into blocks so that the feasible set can be decomposed as a Cartesian product of feasible sets, one for each block. This assumption is crucial, as it allows the methods to perform block-wise updates without worrying about feasibility, and it simplifies the convergence analysis. Extending CDMs to EFGs is non-trivial because the constraints of the sequence-form polytope do not possess this separable structure; instead the strategy space is such that the decision at a given decision point affects all variables that occur after that decision. We are only aware of a couple examples in the literature where separability is not assumed [1, 10], but those methods require strong assumptions which are not applicable in EFG settings.
Contributions. We propose the Extrapolated Cyclic Primal-Dual Algorithm (ECyclicPDA). Our algorithm is the first cyclic coordinate method for the polytope of sequence-form strategies. It achieves a  $O(1 / T)$  convergence rate to a two-player zero-sum Nash equilibrium, with no dependence on the number of blocks; this, is in contrast with the worst-case polynomial dependence on the number of blocks that commonly appears in convergence rate guarantees for cyclic methods. Our method crucially leverages the recursive structure of the prox updates induced by dilated regularizers. In contrast to true cyclic (block) coordinate descent methods, the intermediate iterates generated during one iteration of ECyclicPDA are not feasible because of the non-separable nature of the constraints of sequence-form polytopes. Due to this infeasibility we refer to our updates as being pseudo-block updates. The only information that is fully determined after one pseudo-block update, is the behavioral strategy for all sequences at decision points in the block that was just considered. The behavioral strategy is converted back to sequence-form at the end of a full iteration of our algorithm.
At a very high level, our algorithm is inspired by the CODER algorithm due to Song and Diakonikolas [39]. However, there are several important differences due to the specific structure of the bilinear problem (PD) that we solve. First of all, the CODER algorithm is not directly applicable to our setting, as the feasible set (treeplex) that appears in our problem formulation is not separable. Additionally, CODER only considers Euclidean setups with quadratic regularizers, whereas our work considers more general normed settings; in particular, the  $\ell_1$  setup is of primary interest for our problem setup, since it yields a much better dependence on the game size.
These two issues regarding the non-separability of the feasible set and the more general normed spaces and regularizers are handled in our work by (i) considering dilated regularizers, which allow for blockwise (up to scaling) updates in a bottom-up fashion, respecting the treeplex ordering; and (ii) introducing different extrapolation steps (see Lines 10 and 13 in Algorithm 1) that are unique to our work and specific to the bilinear EFG problem formulation. Additionally, our special problem structure and the choice of the extrapolation sequences  $\widetilde{\mathbf{x}}_k$  and  $\widetilde{\mathbf{y}}_k$  allows us to remove
any nonstandard Lipschitz assumptions used in Song and Diakonikolas [39]. Notably, unlike Song and Diakonikolas [39] and essentially all the work on cyclic methods we are aware of, which pay polynomially for the number of blocks in the convergence bound, our convergence bound in the  $\ell_1$  setting is never worse than the optimal bound of full vector-update methods such as mirror prox [33] and dual extrapolation [35], which we consider a major contribution of our work.
Numerically, we demonstrate that our algorithm performs better than mirror prox (MP), and can be competitive with  $\mathrm{CFR}^+$  and its variants on certain domains. We also propose the use of adaptive restarting as a general heuristic tool for EFG solving: whenever an EFG solver constructs a solution with duality gap at most a constant fraction of its initial value since the last restart, we restart it and initialize the new run with the output solution at restart. Restarting is theoretically supported by the fact that BSPPs possess the sharpness property [5, 18, 21, 43], and restarting combined with certain Euclidean-based FOMs leads to a linear convergence rate under sharpness [5, 21]. We show that with restarting, it is possible for our ECyclicPDA methods to outperform  $\mathrm{CFR}^+$  on some games; this is the first time that a FOM has been observed to outperform  $\mathrm{CFR}^+$  on non-trivial EFGs. Somewhat surprisingly, we then show that for some games, restarting can drastically speed up  $\mathrm{CFR}^+$  as well. In particular, we find that on one game,  $\mathrm{CFR}^+$  with restarting exhibits a linear convergence rate, and so does a recent predictive variant of  $\mathrm{CFR}^+$  [15], on the same game and on an additional one.
Related Work. CMs have been widely studied in the past decade and a half [1-3, 7, 10, 11, 19, 22, 28, 30, 32, 34, 37-39, 46, 48]. CMs can be grouped into three broad classes [38]: greedy methods, which greedily select coordinates that will lead to the largest progress; randomized methods, which select (blocks of) coordinates according to a probability distribution over the blocks; and cyclic methods, which make updates in cyclic orders. Because greedy methods typically require full gradient evaluation (to make the greedy selection), the focus in the literature has primarily been on randomized (RCMs) and cyclic (CCMs) variants. RCMs require separability of the problem's constraints so we focus on CCMs. However, establishing convergence arguments for CCMs through connections with convergence arguments for full gradient methods is difficult. Some guarantees have been provided in the literature, either making restrictive assumptions [37] or by treating the cyclical coordinate gradient as an approximation of a full gradient [7], and thus incurring a linear dependence on the number of blocks in the convergence guarantee. Song and Diakonikolas [39] were the first make an improvement on reducing the dependence on the number of blocks by using a novel extrapolation strategy and introducing new block Lipschitz assumptions. That paper was the main inspiration for our work, but inapplicable to our setting, thus necessitating new technical ideas, as already discussed. While primal-dual coordinate methods for bilinear saddle-point problems have been explored in Carmon et al. [9], their techniques are not clearly extendable to our problem. The  $\ell_1 - \ell_1$  setup they consider is the one which is relevant to the two-player zero-sum game setting we study, but their assumption in that case is that the feasible set for each of the players is a probability simplex, which is a much simpler feasible set than the treeplex considered in our work. It is unclear how to generalize their result to our setting, as their results depend on the simplex structure.
There has also been significant work on FOMs for two-player zero-sum EFG solving. Because this is a BSPP, off-the-shelf FOMs for BSPPs can be applied, with the caveat that proximal oracles are required. The most popular proximal oracles have been based on dilated regularizers [23], which lead to a proximal update that can be performed with a single pass over the decision space, and strong theoretical dependence on game constants [13, 14, 23, 25]. A second popular approach is the counterfactual regret minimization (CFR) framework, which decomposes regret minimization on the EFG decision sets into local simplex-based regret minimization [49]. In theory, CFR-based results have mostly led to an inferior  $T^{-1/2}$  rate of convergence, but in practice the CFR framework instantiated with regret matching $^{+}$  ( $\mathrm{RM^{+}}$ ) [41] or predictive  $\mathrm{RM^{+}}$  ( $\mathrm{PRM^{+}}$ ) [15] is the fastest approach for essentially every EFG setting. The most competitive FOM-based approaches for practical performance are based on dilated regularizers [14, 24], but these have not been able to beat  $\mathrm{CFR^{+}}$  on EFG settings; we show for the first time that it is possible to beat  $\mathrm{CFR^{+}}$  through a combination of block-coordinate updates and restarting, at least on some games. An extended discussion of FOM and CFR approaches to EFG solving is given in Appendix A.
In this section, we provide the necessary background and notation subsequently used to describe and analyze our algorithm. As discussed in the introduction, our focus is on bilinear problems (PD).
2.1 Notation and Optimization Background
We use bold lowercase letters to denote vectors and bold uppercase letters to denote matrices. We use  $\| \cdot \|$  to denote an arbitrary  $\ell_p$  norm for  $p \geq 1$  applied to a vector in either  $\mathbb{R}^m$  or  $\mathbb{R}^n$ , depending on the context. The norm dual to  $\| \cdot \|$  is denoted by  $\| \cdot \|_*$  and defined in the standard way as  $\| \mathbf{z} \|_* = \sup_{\mathbf{x} \neq \mathbf{0}} \frac{\langle \mathbf{z}, \mathbf{x} \rangle}{\|\mathbf{x}\|}$ , where  $\langle \mathbf{z}, \mathbf{x} \rangle$  denotes the standard inner product. In particular, for  $\| \cdot \| = \| \cdot \|_p$  where  $p \geq 1$ , we have  $\| \cdot \|_* = \| \cdot \|_{p^*}$ , where  $\frac{1}{p} + \frac{1}{p^*} = 1$ . We further use  $\| \cdot \|_*$  to denote the induced matrix norm defined by  $\| \mathbf{M} \|_* = \sup_{\mathbf{x} \neq \mathbf{0}} \frac{\|\mathbf{M}\mathbf{x}\|_*}{\|\mathbf{x}\|}$ . In particular, for the Euclidean norm  $\| \cdot \| = \| \cdot \|_2$ , the dual norm  $\| \cdot \|_* = \| \cdot \|_2$  is also the Euclidean norm, and  $\| \mathbf{M} \|_* = \| \mathbf{M}\|_2$  is the matrix operator norm. For the  $\ell_1$  norm  $\| \cdot \| = \| \cdot \|_1$ , the dual norm is the  $\ell_\infty$ -norm,  $\| \cdot \|_* = \| \cdot \|_\infty$ , while the matrix norm is  $\| \mathbf{M} \|_* = \| \mathbf{M}\|_{\infty \to 1} = \sup_{\mathbf{x} \neq \mathbf{0}} \frac{\|\mathbf{M}\mathbf{x}\|_\infty}{\|\mathbf{x}\|_1} = \max_{i,j} |M_{ij}|$ . We use  $\Delta^n = \{\mathbf{x} \in \mathbb{R}^n : \mathbf{x} \geq 0, \langle 1, \mathbf{x} \rangle = 1\}$  to denote the probability simplex in  $n$  dimensions.
Primal-dual Gap. Given  $\mathbf{x} \in \mathbb{R}^d$ , the primal value of the problem (PD) is  $\max_{\mathbf{v} \in \mathcal{X}} \langle \mathbf{M}\mathbf{x}, \mathbf{v} \rangle$ . Similarly, the dual value of (PD) is defined by  $\min_{\mathbf{u} \in \mathcal{Y}} \langle \mathbf{M}\mathbf{u}, \mathbf{y} \rangle$ . Given a primal-dual pair  $(\mathbf{x}, \mathbf{y}) \in \mathcal{X} \times \mathcal{Y}$ , the primal-dual gap (or saddle-point gap) is defined by
$$
\operatorname {G a p} (\mathbf {x}, \mathbf {y}) = \max  _ {\mathbf {v} \in \mathcal {X}} \left\langle \mathbf {M} \mathbf {x}, \mathbf {v} \right\rangle - \min  _ {\mathbf {u} \in \mathcal {Y}} \left\langle \mathbf {M} \mathbf {u}, \mathbf {y} \right\rangle = \max  _ {(\mathbf {u}, \mathbf {v}) \in \mathcal {X} \times \mathcal {Y}} \operatorname {G a p} ^ {\mathbf {u}, \mathbf {v}} (\mathbf {x}, \mathbf {y}),
$$
where we define  $\mathrm{Gap}^{\mathbf{u},\mathbf{v}}(\mathbf{x},\mathbf{y}) = \langle \mathbf{M}\mathbf{x},\mathbf{v}\rangle -\langle \mathbf{M}\mathbf{u},\mathbf{y}\rangle$  . For our analysis, it is useful to work with the relaxed gap  $\mathrm{Gap}^{\mathbf{u},\mathbf{v}}(\mathbf{x},\mathbf{y})$  for some arbitrary but fixed  $\mathbf{u}\in \mathcal{X},\mathbf{v}\in \mathcal{V}$  , and then draw conclusions about a candidate solution by making concrete choices of  $\mathbf{u},\mathbf{v}$
Definitions and Facts from Convex Analysis. In this paper, we primarily work with convex functions  $f: \mathbb{R}^n \to \mathbb{R} \cup \{\pm \infty\}$  that are differentiable on the interior of their domain. We say that  $f$  is  $c_f$ -strongly convex w.r.t. a norm  $\| \cdot \|$  if  $\forall \mathbf{y} \in \mathbb{R}^n, \forall \mathbf{x} \in \operatorname{int} \operatorname{dom} f$ ,
$$
f (\mathbf {y}) \geq f (\mathbf {x}) + \left\langle \nabla f (\mathbf {x}), \mathbf {y} - \mathbf {x} \right\rangle + \frac {c _ {f}}{2} \| \mathbf {y} - \mathbf {x} \| ^ {2}.
$$
We will also need convex conjugates and Bregman divergences. Given an extended real valued function  $f: \mathbb{R}^n \to \mathbb{R} \cup \{\pm \infty\}$ , its convex conjugate is defined by  $f^{*}(\mathbf{z}) = \sup_{\mathbf{z} \in \mathbb{R}^{n}} \{\langle \mathbf{z}, \mathbf{x} \rangle - f(\mathbf{x})\}$ . Let  $f: \mathbb{R}^n \to \mathbb{R} \cup \{\pm \infty\}$  be a function that is differentiable on the interior of its domain. Given  $\mathbf{y} \in \mathbb{R}^n$  and  $\mathbf{x} \in \operatorname{int}\mathrm{dom}f$ , the Bregman divergence  $D_{f}(\mathbf{y},\mathbf{x})$  is defined by  $D_{f}(\mathbf{y},\mathbf{x}) = f(\mathbf{y}) - f(\mathbf{x}) - \langle \nabla f(\mathbf{x}),\mathbf{y} - \mathbf{x} \rangle$ . If the function  $f$  is  $c_{f}$ -strongly convex, then  $D_{f}(\mathbf{y},\mathbf{x}) \geq \frac{c_{f}}{2}\| \mathbf{y} - \mathbf{x}\|^{2}$ .
2.2 Extensive-Form Games: Background and Additional Notation
Extensive form games are represented by game trees. Each node  $v$  in the game tree belongs to exactly one player  $i \in \{1, \dots, n\} \cup \{\mathsf{c}\}$  whose turn it is to move. Player  $\mathsf{c}$  is a special player called the chance player; it is used to denote random events that happen in the game, such as drawing a card from a deck or tossing a coin. At terminal nodes of the game, players are assigned payoffs. We focus on two-player zero-sum games, where  $n = 2$  and payoffs sum to zero. Private information is modeled using information sets (infosets): a player cannot distinguish between nodes in the same infoset, so the set of actions available to them must be the same at each node in the infoset.
Treeplexes. The decision problem for a player in a perfect recall EFG can be described as follows. There exists a set of decision points  $\mathcal{J}$ , and at each decision point  $j$  the player has a set of actions  $A_{j}$  with  $|A_{j}| = n_{j}$  actions in total. These decision points coincide with infosets in the EFG. Without loss of generality, we let there be a single root decision point, representing the first decision the player makes in the game. The choice to play an action  $a \in A_{j}$  for a decision point  $j \in \mathcal{J}$  is represented using a sequence  $(j, a)$ , and after playing this sequence, the set of possible next decision points is denoted by  $\mathcal{C}_{j,a}$  (which may be empty in case the game terminates). The set of decisions form a tree, meaning that  $\mathcal{C}_{j,a} \cap \mathcal{C}_{j',a'} = \emptyset$  unless  $j = j'$  and  $a = a'$ ; this is known as perfect recall. The last sequence (necessarily unique) encountered on the path from the root to decision point  $j$  is denoted by  $p_{j}$ . We define  $\downarrow j$  as the set consisting of all decision points that can be reached from  $j$ . An example of the use of this notation for a player in Kuhn poker [26] can be found in Appendix B.
The set of strategies for a player can be characterized using the sequence-form, where the value of the decision variable assigned to playing the sequence  $(j,a)$  is the product of the decision variable assigned to playing the parent sequence  $p_j$  and the probability of playing action  $a$  when at  $j$  [44].
The set of all sequence-form strategies of a player form a polytope known as the sequence-form polytope. Sequence-form polytopes fall into a class of polytopes known as treeplexes [23], which can be characterized inductively using convex hull and Cartesian product operations:
Definition 2.1 (Treeplex). A treeplex  $\mathcal{X}$  for a player can be characterized recursively as follows, where  $r$  is the root decision point for a player.
$$
\begin{array}{l} \mathcal {X} _ {j, a} = \prod_ {j ^ {\prime} \in \mathcal {C} _ {j, a}} \mathcal {X} _ {\downarrow j ^ {\prime}}, \\ \mathcal {X} _ {\downarrow j} = \left\{ \right.\left( \right.\lambda_ {1}, \ldots , \lambda_ {| A _ {j} |}, \lambda_ {1} \mathbf {x} _ {1}, \ldots , \lambda_ {| A _ {j} |} \mathbf {x} _ {| A _ {j} |}: (\lambda_ {1}, \ldots , \lambda_ {| A _ {j} |}) \in \Delta^ {| A _ {j} |}, \mathbf {x} _ {a} \in \mathcal {X} _ {j, a} \left. \right\}, \\ \mathcal {X} = \{1 \} \times \mathcal {X} _ {\downarrow r}. \\ \end{array}
$$
This formulation allows the expected loss of a player to be formulated as a bilinear function  $\langle \mathbf{M}\mathbf{x},\mathbf{y}\rangle$  of players' strategies  $\mathbf{x},\mathbf{y}$ . This gives rise to the BSPP in Equation (PD), and the set of saddle points of that BSPP are exactly the set of Nash equilibria of the EFG. The payoff matrix  $\mathbf{M}$  is a sparse matrix, whose nonzeroes correspond to the set of leaf nodes of the game tree.
Indexing Notation. A sequence-form strategy of a player can be written as a vector  $\mathbf{v}$ , with an entry for each sequence  $(j, a)$ . We use  $\mathbf{v}^j$  to denote the subset of size  $|A_j|$  of entries of  $\mathbf{v}$  that correspond to sequences  $(j, a)$  formed by taking actions  $a \in A_j$  and let  $\mathbf{v}^{\downarrow j}$  denote the subset of entries of  $\mathbf{v}$  that are indexed by sequences that occur in the subtreeplex rooted at  $j$ . Additionally, we use  $v^{p_j}$  to denote the (scalar) value of the parent sequence of decision point  $j$ . By convention, for the root decision point  $j$ , we let  $v^{p_j} = 1$ . Observe that for any  $j \in \mathcal{J}$ ,  $\mathbf{v}^j / v^{p_j}$  is in the probability simplex.
Given a treeplex  $\mathcal{Z}$  we denote by  $\mathcal{I}_{\mathcal{Z}}$  the set of infosets for this treeplex. We say that a partition of  $\mathcal{I}_{\mathcal{Z}}$  into  $k \leq |\mathcal{I}_{\mathcal{Z}}|$  sets  $\mathcal{I}_{\mathcal{Z}}^{(1)}, \ldots, \mathcal{I}_{\mathcal{Z}}^{(k)}$  respects the treeplex ordering if for any two sets  $\mathcal{I}_{\mathcal{Z}}^{(i)}, \mathcal{I}_{\mathcal{Z}}^{(i')}$  with  $i < i'$  and any two infosets  $j \in \mathcal{I}_{\mathcal{Z}}^{(i)}, j' \in \mathcal{I}_{\mathcal{Z}}^{(i')}$ ,  $j$  does not intersect the path from  $j'$  to the root decision point. The set of infosets for the player  $\mathbf{x}$  is denoted by  $\mathcal{I}_{\mathcal{X}}$ , while the set of infosets for player  $\mathbf{y}$  is denoted by  $\mathcal{I}_{\mathcal{Y}}$ . We assume that  $\mathcal{I}_{\mathcal{X}}$  and  $\mathcal{I}_{\mathcal{Y}}$  are partitioned into  $s$  nonempty sets  $\mathcal{I}_{\mathcal{X}}^{(1)}, \mathcal{I}_{\mathcal{X}}^{(2)}, \ldots, \mathcal{I}_{\mathcal{X}}^{(s)}$  and  $\mathcal{I}_{\mathcal{Y}}^{(1)}, \mathcal{I}_{\mathcal{Y}}^{(2)}, \ldots, \mathcal{I}_{\mathcal{Y}}^{(s)}$ , where  $s \leq \min \{| \mathcal{I}_{\mathcal{X}} |, | \mathcal{I}_{\mathcal{Y}} | \}$  and the ordering of the sets in the two partitions respect the treeplex ordering of  $\mathcal{X}, \mathcal{Y}$ , respectively.