Title: Block-Coordinate Methods and Restarting for Solving Extensive-Form Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: extensive-form games

first-order methods

coordinate descent
Abstract: Coordinate descent methods are popular in machine learning and optimization for their simple sparse updates and excellent practical performance. 
In the context of large-scale sequential game solving, these same properties would be attractive, but until now no such methods were known, because the strategy spaces do not satisfy the typical separable block structure exploited by such methods.
We present the first cyclic coordinate-descent-like method for the polytope of sequence-form strategies, which form the strategy spaces for the players in an extensive-form game (EFG). 
Our method exploits the recursive structure of the proximal update induced by what are known as dilated regularizers, in order to allow for a pseudo block-wise update.
We show that our method enjoys a O(1/T) convergence rate to a two-player zero-sum Nash equilibrium, while avoiding the worst-case polynomial scaling with the number of blocks common to cyclic methods. We empirically show that our algorithm usually performs better than other state-of-the-art first-order methods (i.e., mirror prox), and occasionally can even beat CFR$^+$, a state-of-the-art algorithm for numerical equilibrium computation in zero-sum EFGs. 
We then introduce a restarting heuristic for EFG solving. We show empirically that restarting can lead to speedups, sometimes huge, both for our cyclic method, as well as for existing methods such as mirror prox and predictive CFR$^+$.

Block-Coordinate Methods and Restarting for Solving Extensive-Form Games ∗

Abstract

Coordinate descent methods are popular in machine learning and optimization for their simple sparse updates and excellent practical performance. In the context of large-scale sequential game solving, these same properties would be attractive, but until now no such methods were known, because the strategy spaces do not satisfy the typical separable block structure exploited by such methods. We present the first cyclic coordinate-descent-like method for the polytope of sequence-form strategies, which form the strategy spaces for the players in an extensive-form game (EFG). Our method exploits the recursive structure of the proximal update induced by what are known as dilated regularizers, in order to allow for a pseudo block-wise update. We show that our method enjoys a O (1 /T ) convergence rate to a two-player zero-sum Nash equilibrium, while avoiding the worst-case polynomial scaling with the number of blocks common to cyclic methods. We empirically show that our algorithm usually performs better than other state-of-the-art first-order methods (i.e., mirror prox), and occasionally can even beat CFR + , a state-ofthe-art algorithm for numerical equilibrium computation in zero-sum EFGs. We then introduce a restarting heuristic for EFG solving. We show empirically that restarting can lead to speedups, sometimes huge, both for our cyclic method, as well as for existing methods such as mirror prox and predictive CFR + .

1 Introduction

Extensive-form games (EFGs) are a broad class of game-theoretic models which are played on a tree. They can compactly model both simultaneous and sequential moves, private and/or imperfect information, and stochasticity. Equilibrium computation for a two-player zero-sum EFG can be formulated as the following bilinear saddle-point problem (BSPP)

Here, the set of strategies X , Y for the x and y players are convex polytopes known as sequence-form polytopes [44]. The (PD) formulation lends itself to first-order methods (FOMs) [14, 25], linear

programming [44], and online learning-based approaches [6, 8, 15, 17, 42, 49], since the feasible sets are convex and compact polytopes, and the objective is bilinear.

A common approach for solving BSPPs is by using first-order methods, where local gradient information is used to iteratively improve the solution in order to converge to an equilibrium asymptotically. In the game-solving context, such methods rely on two oracles: a first-order oracle that returns a (sub)gradient at the current pair of strategies, and a pair of prox oracles for the strategy spaces X , Y , which allow one to perform a generalized form of projected gradient descent steps on X , Y . These prox oracles are usually constructed through the choice of an appropriate regularizer . For EFGs, it is standard to focus on regularizers for which the prox oracle can be computed in linear time with respect to the size of the polytope, which is only known to be achievable through what is known as dilated regularizers [23]. Most first-order methods for EFGs require full-tree traversals for the first-order oracle, and full traversals of the decision sets for the prox computation, before making a strategy update for each player. For large EFGs these full traversals, especially for the first-order oracle, can be very expensive, and it may be desirable to make strategy updates before a full traversal has been performed, in order to more rapidly incorporate partial first-order information.

In other settings, one commonly used approach for solving large-scale problems is through coordinate methods (CMs) [34, 46]. These methods involve computing the gradient for a restricted set of coordinates at each iteration of the algorithm, and using these partial gradients to construct descent directions. The convergence rate of these methods typically is able to match the rate of full gradient methods. However, in some cases they may exhibit worse runtime due to constants introduced by the method. In spite of this, they often serve practical benefits of being more time and space efficient, and enabling distributed computation [2, 3, 11, 19, 22, 28, 30, 32, 34, 47, 48].

Generally, coordinate descent methods assume that the problem is separable, i.e., there exists a partition of the coordinates into blocks so that the feasible set can be decomposed as a Cartesian product of feasible sets, one for each block. This assumption is crucial, as it allows the methods to perform block-wise updates without worrying about feasibility, and it simplifies the convergence analysis. Extending CDMs to EFGs is non-trivial because the constraints of the sequence-form polytope do not possess this separable structure; instead the strategy space is such that the decision at a given decision point affects all variables that occur after that decision. We are only aware of a couple examples in the literature where separability is not assumed [1, 10], but those methods require strong assumptions which are not applicable in EFG settings.

Contributions. We propose the Extrapolated Cyclic Primal-Dual Algorithm (ECyclicPDA) . Our algorithm is the first cyclic coordinate method for the polytope of sequence-form strategies. It achieves a O (1 /T ) convergence rate to a two-player zero-sum Nash equilibrium, with no dependence on the number of blocks; this, is in contrast with the worst-case polynomial dependence on the number of blocks that commonly appears in convergence rate guarantees for cyclic methods. Our method crucially leverages the recursive structure of the prox updates induced by dilated regularizers. In contrast to true cyclic (block) coordinate descent methods, the intermediate iterates generated during one iteration of ECyclicPDA are not feasible because of the non-separable nature of the constraints of sequence-form polytopes. Due to this infeasibility we refer to our updates as being pseudo-block updates. The only information that is fully determined after one pseudo-block update, is the behavioral strategy for all sequences at decision points in the block that was just considered. The behavioral strategy is converted back to sequence-form at the end of a full iteration of our algorithm.

At a very high level, our algorithm is inspired by the CODER algorithm due to Song and Diakonikolas [39]. However, there are several important differences due to the specific structure of the bilinear problem (PD) that we solve. First of all, the CODER algorithm is not directly applicable to our setting, as the feasible set (treeplex) that appears in our problem formulation is not separable. Additionally, CODER only considers Euclidean setups with quadratic regularizers, whereas our work considers more general normed settings; in particular, the ℓ 1 setup is of primary interest for our problem setup, since it yields a much better dependence on the game size.

These two issues regarding the non-separability of the feasible set and the more general normed spaces and regularizers are handled in our work by (i) considering dilated regularizers, which allow for blockwise (up to scaling) updates in a bottom-up fashion, respecting the treeplex ordering; and (ii) introducing different extrapolation steps (see Lines 10 and 13 in Algorithm 1) that are unique to our work and specific to the bilinear EFG problem formulation. Additionally, our special problem structure and the choice of the extrapolation sequences ˜ x k and ˜ y k allows us to remove

any nonstandard Lipschitz assumptions used in Song and Diakonikolas [39]. Notably, unlike Song and Diakonikolas [39] and essentially all the work on cyclic methods we are aware of, which pay polynomially for the number of blocks in the convergence bound, our convergence bound in the ℓ 1 setting is never worse than the optimal bound of full vector-update methods such as mirror prox [33] and dual extrapolation [35], which we consider a major contribution of our work.

Numerically, we demonstrate that our algorithm performs better than mirror prox (MP), and can be competitive with CFR + and its variants on certain domains. We also propose the use of adaptive restarting as a general heuristic tool for EFG solving: whenever an EFG solver constructs a solution with duality gap at most a constant fraction of its initial value since the last restart, we restart it and initialize the new run with the output solution at restart. Restarting is theoretically supported by the fact that BSPPs possess the sharpness property [5, 18, 21, 43], and restarting combined with certain Euclidean-based FOMs leads to a linear convergence rate under sharpness [5, 21]. We show that with restarting, it is possible for our ECyclicPDA methods to outperform CFR + on some games; this is the first time that a FOM has been observed to outperform CFR + on non-trivial EFGs. Somewhat surprisingly, we then show that for some games, restarting can drastically speed up CFR + as well. In particular, we find that on one game, CFR + with restarting exhibits a linear convergence rate, and so does a recent predictive variant of CFR + [15], on the same game and on an additional one.

Related Work. CMs have been widely studied in the past decade and a half [1-3, 7, 10, 11, 19, 22, 28, 30, 32, 34, 37-39, 46, 48]. CMs can be grouped into three broad classes [38]: greedy methods, which greedily select coordinates that will lead to the largest progress; randomized methods, which select (blocks of) coordinates according to a probability distribution over the blocks; and cyclic methods, which make updates in cyclic orders. Because greedy methods typically require full gradient evaluation (to make the greedy selection), the focus in the literature has primarily been on randomized (RCMs) and cyclic (CCMs) variants. RCMs require separability of the problem's constraints so we focus on CCMs. However, establishing convergence arguments for CCMs through connections with convergence arguments for full gradient methods is difficult. Some guarantees have been provided in the literature, either making restrictive assumptions [37] or by treating the cyclical coordinate gradient as an approximation of a full gradient [7], and thus incurring a linear dependence on the number of blocks in the convergence guarantee. Song and Diakonikolas [39] were the first make an improvement on reducing the dependence on the number of blocks by using a novel extrapolation strategy and introducing new block Lipschitz assumptions. That paper was the main inspiration for our work, but inapplicable to our setting, thus necessitating new technical ideas, as already discussed. While primal-dual coordinate methods for bilinear saddle-point problems have been explored in Carmon et al. [9], their techniques are not clearly extendable to our problem. The ℓ 1 -ℓ 1 setup they consider is the one which is relevant to the two-player zero-sum game setting we study, but their assumption in that case is that the feasible set for each of the players is a probability simplex, which is a much simpler feasible set than the treeplex considered in our work. It is unclear how to generalize their result to our setting, as their results depend on the simplex structure.

There has also been significant work on FOMs for two-player zero-sum EFG solving. Because this is a BSPP, off-the-shelf FOMs for BSPPs can be applied, with the caveat that proximal oracles are required. The most popular proximal oracles have been based on dilated regularizers [23], which lead to a proximal update that can be performed with a single pass over the decision space, and strong theoretical dependence on game constants [13, 14, 23, 25]. A second popular approach is the counterfactual regret minimization (CFR) framework, which decomposes regret minimization on the EFG decision sets into local simplex-based regret minimization [49]. In theory, CFR-based results have mostly led to an inferior T -1 / 2 rate of convergence, but in practice the CFR framework instantiated with regret matching + (RM + ) [41] or predictive RM + (PRM + ) [15] is the fastest approach for essentially every EFG setting. The most competitive FOM-based approaches for practical performance are based on dilated regularizers [14, 24], but these have not been able to beat CFR + on EFG settings; we show for the first time that it is possible to beat CFR + through a combination of block-coordinate updates and restarting, at least on some games. An extended discussion of FOM and CFR approaches to EFG solving is given in Appendix A.

2 Notation and Preliminaries

In this section, we provide the necessary background and notation subsequently used to describe and analyze our algorithm. As discussed in the introduction, our focus is on bilinear problems (PD).

2.1 Notation and Optimization Background

̸

We use bold lowercase letters to denote vectors and bold uppercase letters to denote matrices. We use ∥ · ∥ to denote an arbitrary ℓ p norm for p ≥ 1 applied to a vector in either R m or R n , depending on the context. The norm dual to ∥ · ∥ is denoted by ∥ · ∥ ∗ and defined in the standard way as ∥ z ∥ ∗ = sup x = 0 ⟨ z , x ⟩ ∥ x ∥ , where ⟨ z , x ⟩ denotes the standard inner product. In particular, for ∥·∥ = ∥·∥ p , where p ≥ 1 , we have ∥ · ∥ ∗ = ∥ · ∥ p ∗ , where 1 p + 1 p ∗ = 1 . We further use ∥ · ∥ ∗ to denote the induced matrix norm defined by ∥ M ∥ ∗ = sup x = 0 ∥ Mx ∥ ∗ ∥ x ∥ . In particular, for the Euclidean norm ∥ · ∥ = ∥ · ∥ 2 , the dual norm ∥ · ∥ ∗ = ∥ · ∥ 2 is also the Euclidean norm, and ∥ M ∥ ∗ = ∥ M ∥ 2 is the matrix operator norm. For the ℓ 1 norm ∥ · ∥ = ∥ · ∥ 1 , the dual norm is the ℓ ∞ -norm, ∥ · ∥ ∗ = ∥ · ∥ ∞ , while the matrix norm is ∥ M ∥ ∗ = ∥ M ∥ ∞→ 1 = sup x = 0 ∥ Mx ∥ ∞ ∥ x ∥ 1 = max i,j | M ij | . We use ∆ n = { x ∈ R n : x ≥ 0 , ⟨ 1 , x ⟩ = 1 } to denote the probability simplex in n dimensions.

̸

Primal-dual Gap. Given x ∈ R d , the primal value of the problem (PD) is max v ∈X ⟨ Mx , v ⟩ . Similarly, the dual value of (PD) is defined by min u ∈Y ⟨ Mu , y ⟩ . Given a primal-dual pair ( x , y ) ∈ X × Y , the primal-dual gap (or saddle-point gap) is defined by

where we define Gap u , v ( x , y ) = ⟨ Mx , v ⟩ - ⟨ Mu , y ⟩ . For our analysis, it is useful to work with the relaxed gap Gap u , v ( x , y ) for some arbitrary but fixed u ∈ X , v ∈ Y , and then draw conclusions about a candidate solution by making concrete choices of u , v .

Definitions and Facts from Convex Analysis. In this paper, we primarily work with convex functions f : R n → R ∪ {±∞} that are differentiable on the interior of their domain. We say that f is c f -strongly convex w.r.t. a norm ∥ · ∥ if ∀ y ∈ R n , ∀ x ∈ int dom f ,

We will also need convex conjugates and Bregman divergences. Given an extended real valued function f : R n → R ∪{±∞} , its convex conjugate is defined by f ∗ ( z ) = sup z ∈ R n {⟨ z , x ⟩-f ( x ) } . Let f : R n → R ∪ {±∞} be a function that is differentiable on the interior of its domain. Given y ∈ R n and x ∈ int dom f , the Bregman divergence D f ( y , x ) is defined by D f ( y , x ) = f ( y ) -f ( x ) -⟨∇ f ( x ) , y -x ⟩ . If the function f is c f -strongly convex, then D f ( y , x ) ≥ c f 2 ∥ y -x ∥ 2 .

2.2 Extensive-Form Games: Background and Additional Notation

Extensive form games are represented by game trees. Each node v in the game tree belongs to exactly one player i ∈ { 1 , . . . , n } ∪ { c } whose turn it is to move. Player c is a special player called the chance player; it is used to denote random events that happen in the game, such as drawing a card from a deck or tossing a coin. At terminal nodes of the game, players are assigned payoffs. We focus on two-player zero-sum games, where n = 2 and payoffs sum to zero. Private information is modeled using information sets (infosets): a player cannot distinguish between nodes in the same infoset, so the set of actions available to them must be the same at each node in the infoset.

Treeplexes. The decision problem for a player in a perfect recall EFG can be described as follows. There exists a set of decision points J , and at each decision point j the player has a set of actions A j with | A j | = n j actions in total. These decision points coincide with infosets in the EFG. Without loss of generality, we let there be a single root decision point, representing the first decision the player makes in the game. The choice to play an action a ∈ A j for a decision point j ∈ J is represented using a sequence ( j, a ) , and after playing this sequence, the set of possible next decision points is denoted by C j,a (which may be empty in case the game terminates). The set of decisions form a tree, meaning that C j,a ∩ C j ′ ,a ′ = ∅ unless j = j ′ and a = a ′ ; this is known as perfect recall . The last sequence (necessarily unique) encountered on the path from the root to decision point j is denoted by p j . We define ↓ j as the set consisting of all decision points that can be reached from j . An example of the use of this notation for a player in Kuhn poker [26] can be found in Appendix B.

The set of strategies for a player can be characterized using the sequence-form , where the value of the decision variable assigned to playing the sequence ( j, a ) is the product of the decision variable assigned to playing the parent sequence p j and the probability of playing action a when at j [44].

̸

The set of all sequence-form strategies of a player form a polytope known as the sequence-form polytope. Sequence-form polytopes fall into a class of polytopes known as treeplexes [23], which can be characterized inductively using convex hull and Cartesian product operations:

Definition 2.1 (Treeplex) . A treeplex X for a player can be characterized recursively as follows, where r is the the root decision point for a player.

This formulation allows the expected loss of a player to be formulated as a bilinear function ⟨ Mx , y ⟩ of players' strategies x , y . This gives rise to the BSPP in Equation (PD), and the set of saddle points of that BSPP are exactly the set of Nash equilibria of the EFG. The payoff matrix M is a sparse matrix, whose nonzeroes correspond to the set of leaf nodes of the game tree.

Indexing Notation. A sequence-form strategy of a player can be written as a vector v , with an entry for each sequence ( j, a ) . We use v j to denote the subset of size | A j | of entries of v that correspond to sequences ( j, a ) formed by taking actions a ∈ A j and let v ↓ j denote the subset of entries of v that are indexed by sequences that occur in the subtreeplex rooted at j . Additionally, we use v p j to denote the (scalar) value of the parent sequence of decision point j . By convention, for the root decision point j, we let v p j = 1 . Observe that for any j ∈ J , v j /v p j is in the probability simplex.

Given a pair ( t, t ′ ) , we use M t,t ′ to denote the full-dimensional ( m × n ) matrix obtained from the matrix M by keeping all entries indexed by J X ( t ) and J Y ( t ′ ) , and zeroing out the rest. When in place of t or t ′ we use ' : ', it corresponds to keeping as non-zeros all rows (for the first index) or all columns (for the second index). In particular, M t, : is the matrix that keeps all rows of M indexed by J X ( t ) intact and zeros out the rest. Further, notation M t ′ ,t : s is used to indicate that we select rows indexed by J X ( t ′ ) and all columns of M indexed by J Y ( t ) , J Y ( t +1) , . . . , J Y ( s ) , while we zero out the rest; similarly for M t : s,t ′ . Notation M t ′ , 1: t is used to indicate that we select rows indexed by J X ( t ′ ) and all columns of M indexed by J Y (1) , J Y (2) , . . . , J Y ( t ) , while we zero out the rest; similarly for M 1: t,t ′ . Given a vector x ∈ X , x ( t ) denotes the entries of x indexed by the elements of J X ( t ) ; similarly, for y ∈ Y , y ( t ) denotes the entries of y indexed by the elements of J Y ( t ) .

Given a treeplex Z we denote by J Z the set of infosets for this treeplex. We say that a partition of J Z into k ≤ |J Z | sets J Z (1) , . . . , J Z ( k ) respects the treeplex ordering if for any two sets J Z ( i ) , J Z ( i ′ ) with i < i ′ and any two infosets j ∈ J Z ( i ) , j ′ ∈ J Z ( i ′ ) , j does not intersect the path from j ′ to the root decision point. The set of infosets for the player x is denoted by J X , while the set of infosets for player y is denoted by J Y . We assume that J X and J Y are partitioned into s nonempty sets J X (1) , J X (2) , . . . , J X ( s ) and J Y (1) , J Y (2) , . . . , J Y ( s ) , where s ≤ min {|J X | , |J Y |} and the ordering of the sets in the two partitions respect the treeplex ordering of X , Y , respectively.

Additionally, we use M ( t,t ′ ) to denote the submatrix of M obtained by selecting rows indexed by J X ( t ) and columns indexed by J Y ( t ′ ) . M ( t,t ′ ) is ( p × q ) -dimensional, for p = ∑ j ∈J X ( t ) | A j | and q = ∑ j ∈J Y ( t ′ ) | A j | . Notation ' : ' has the same meaning as in the previous paragraph.

Dilated Regularizers. We assume access to strongly convex functions ϕ : X → R and ψ : Y → R with known strong convexity parameters c ϕ > 0 and c ψ > 0 , and that are continuously differentiable on the interiors of their respective domains. We further assume that these functions are nice as defined by Farina et al. [14]: their gradients and the gradients of their convex conjugates can be computed in time linear (or nearly linear) in the dimension of the treeplex.

A dilated regularizer is a framework for constructing nice regularizing functions for treeplexes. It makes use of the inductive characterization of a treeplex via Cartesian product and convex hull operations to generalize from the local simplex structure of the sequence-form polytope at a decision point to the entire sequence-form polytope. In particular, given a local 'nice' regularizer ϕ j for each decision point j , a dilated regularizer for the treeplex can be defined as ϕ ( x ) = ∑ j ∈J X x p j ϕ j ( x j x p j ) .

The key property of these dilated regularizing functions is that the prox computations of the form x k = argmin x ∈X {⟨ h , x ⟩ + D ϕ ( x k , x k -1 ) } decompose into bottom-up updates, where, up to a scaling factor, each set of coordinates from set J X ( t ) can be computed solely based on the coordinates of x k from sets J X (1) , . . . , J X ( t -1) and coordinates of g from sets J X (1) , . . . , J X ( t ) . Concretely, the recursive structure of the prox update is as follows (this was originally shown by [23], here we show a variation from Farina et al. [13]):

Proposition 2.2 (Farina et al. [13]) . A prox update to compute x k , with gradient h and center x k -1 on a treeplex X using a Bregman divergence constructed from a dilated DGF ϕ can be decomposed into local prox updates at each decision point j ∈ J X as follows:

3 Extrapolated Cyclic Algorithm

Our extrapolated cyclic primal-dual algorithm is summarized in Algorithm 1. As discussed in Section 2, under the block partition and ordering that respects the treeplex ordering, the updates for x ( t ) k in Line 9 (respectively, y ( t ) k in Line 12), up to scaling by the value of their respective parent sequences, can be carried out using only the information about x j k x p j k and h j k (respectively, y j k y p j k and g j k ) for infosets j that are 'lower' on the treeplex. The specific choices of the extrapolation sequences ˜ x k and ˜ y k that only utilize the information from prior cycles and the scaled values of x j k x p j k and y j k y p j k for infosets j updated up to the block t updates for x k and y k are what crucially enables us to decompose the updates for x k and y k into local block updates carried out in the bottom-up manner. At the end of the cycle, once x j k x p j k and y j k y p j k has been updated for all infosets, we can carry out a top-to-bottom update to fully determine vectors x k and y k , as summarized in the last two for loops in Algorithm 1. We present an implementation-specific version of the algorithm in Appendix D, which explicitly demonstrates that our algorithm's runtime does not have a dependence on the number of blocks used. In our analysis of the implementation-specific version of the algorithm, we argue that the per-iteration complexity of our algorithm matches that of MP. To support this analysis, we compare the empirical runtimes of our algorithm with MP and CFR + variants in Section 4.

Our convergence argument is built on the decomposition of the relaxed gap Gap u , v ( x k , v k ) for arbitrary but fixed ( u , v ) ∈ X × Y into telescoping and non-positive terms, which is common in first-order methods. The first idea that enables leveraging cyclic updates lies in replacing vectors Mx k and M ⊤ y k by 'extrapolated' vectors g k and h k that can be partially updated in a blockwise fashion as a cycle of the algorithm progresses, as stated in Proposition 3.1. To our knowledge, this basic idea originates in Song and Diakonikolas [39]. Unique to our work are the specific choices of g k and h k , which leverage all the partial information known to the algorithm up to the current iteration and block update. Crucially, we leverage the treeplex structure to show that our chosen updates are sufficient to bound the error sequence E k and obtain the claimed convergence bound in Theorem 3.2. Due to space constraints, the proof is deferred to Appendix C.

To simplify the exposition, we introduce the following notation:

When the norm of the space is ∥ · ∥ = ∥ · ∥ 1 , both µ x and µ y are bounded above by 2 max i,j | M ij | .

The next proposition decomposes the relaxed gap into an error term and telescoping terms. The proposition is independent of the specific choices of extrapolated vectors g k , h k .

Proposition 3.1. Let x k , y k be the iterates of Algorithm 1 for k ≥ 1 . Then, for all k ≥ 1 , x k ∈ X , y k ∈ Y , we have where the error sequence E k is defined by

To obtain our main result, we leverage the blockwise structure of the problem, the bilinear structure of the objective, and the treeplex structure of the feasible sets to control the error sequence E k . A key property that enables this result is that normalized entries x j k /x p j k -1 from the same information set belong to a probability simplex. This property is crucially used in controlling the error of the extrapolation vectors. The main result is summarized in the following theorem.

Theorem 3.2. Consider the iterates x k , y k for k ≥ 1 in Algorithm 1 and the output primal-dual pair ¯ x K , ¯ y K . Then, ∀ k ≥ 1 ,

In the above bound, if ∀ k ≥ 1 , η k = η = √ c ϕ c ψ µ x + µ y , then H K = Kη . As a consequence, for any ϵ > 0 , Gap( ¯ x K , ¯ y K ) ≤ ϵ after at most ⌈ ( µ x + µ y )(sup u ∈X , v ∈Y { D ϕ ( u , x 0 )+ D ψ ( v , y 0 ) } ) √ c ϕ c ψ ϵ ⌉ iterations.

4 Experimental Evaluation and Discussion

We evaluate the performance of ECyclicPDA instantiated with three different dilated regularizers: dilated entropy [25], dilatable global entropy [14], and dilated ℓ 2 [13]. In the case of the dilated ℓ 2

regularizer, we use dual averaging of the 'extrapolated' vectors g k and h k in our algorithm, since otherwise we have no guarantee that the iterates would remain in the relative interior of the domain of the dilated DGF, and the Bregman divergence may become undefined. We compare our method to MP, which is state-of-the-art among first-order methods for EFG solving. We test ECyclicPDA and MP with three different averaging schemes: uniform, linear, and quadratic averaging since Gao et al. [20] suggest that these different averaging schemes can lead to faster convergence in practice. We also compare against empirical state-of-the-art CFR + variants: CFR + [41], and the predictive CFR + variant (PCFR + ) [15]. We emphasize that our method achieves the same O ( 1 T ) average-iterate convergence rate as MP, and that all the CFR+ variants have the same suboptimal O ( 1 √ T ) average-iterate convergence rate. We experiment on four standard benchmark games for EFG solving: Goofspiel (4 ranks), Liar's Dice, Leduc (13 ranks), and Battleship. In all experiments, we run for 10 , 000 full (or equivalent) gradient computations. This corresponds to 5,000 iterations of ECyclicPDA, CFR + , and PCFR + , and 2,500 iterations of MP. 2 A description of all games is provided in Appendix E. Additional experimental details are provided in Appendix G.

For each instantiation of ECyclicPDA considered on a given game (choice of regularizer, averaging, and block construction strategy) the stepsize is tuned by taking power of 2 multiples of η ( 2 l · η for l ∈ N ), where η is the theoretical stepsize stated in Theorem 3.2, and then choosing the stepsize η ∗ among these multiples of η that has the best performance. Within the algorithm, we use a constant stepsize, letting η k = η 0 for all k . We apply the same tuning scheme for MP stepsizes (for a given choice of regularizer and averaging). Note that this stepsize tuning is coarse, and so it is possible that better results can be achieved for ECyclicPDA and MP using finer stepsize tuning.

We test our algorithm with four different block construction strategies. The single block construction strategy puts every decision point in a single block, and thus it corresponds to the non-block-based version of ECyclicPDA. The children construction strategy iterates through the decision points of the treeplex bottom-up (by definition, this will respect the treeplex ordering), and placing each set of decision points that have parent sequences starting from the same decision point in its own block. The postorder construction strategy iterates through the decision points bottom-up (again, by definition, this will respect the treeplex ordering). The order is given by a postorder traversal of the treeplex, treating all decision points that have the same parent sequence as a single node (and when the node is processed, all decision points are sequentially added to the block). It greedily makes blocks as large as possible, while only creating a new block if it causes a parent decision point and child decision point to end up in the same block. The infosets construction strategy places each decision point in its own block. We provide further description of the block construction strategies in Appendix F.

We show the results of different block construction strategies in Figure 1. For each block construction strategy, ECyclicPDA is instantiated with the choice of regularizer and averaging that yields the fastest convergence among all choices of parameters. We can see that the different block construction strategies do not make a significant difference in Goofspiel (4 ranks) or in Leduc (13 ranks). However, we see benefits of using blocks in Liar's Dice and Battleship. In Liar's Dice, children and postorder have a clear advantage, and children outperforms the other block construction strategies in Battleship.

We show the results of comparing our algorithm against MP, CFR + , and PCFR + in Figure 2. ECyclicPDA is instantiated with the choice of regularizer, averaging, and block construction strategy that yields the fastest convergence among all choices for ECyclicPDA, and MP is instantiated with the choice of regularizer and averaging that yields the fastest convergence among all choices for MP. We see that ECyclicPDA performs better than MP in all games besides Goofspiel (4 ranks), where they perform about the same. In Liar's Dice and Battleship, the games where ECyclicPDA benefits from having multiple blocks, we see competitiveness with CFR + and PCFR + . In particular, in Liar's Dice, ECyclicPDA is overtaking CFR + at 10 , 000 gradient computations. On Battleship, we see that both ECyclicPDA and MP outperform CFR + , and that ECyclicPDA is competitive with PCFR + .

Restarting. We now introduce restarting as a heuristic tool for speeding up EFG solving. While restarting is only known to lead to a linear convergence rate in the case of using the ℓ 2 regularizer in certain FOMs [5, 21], we apply restarting as a heuristic across our methods based on dilated regularizers and to CFR-based methods. To the best of our knowledge, restarting schemes have not been empirically evaluated on EFG algorithms such as MP, CFR + , or (obviously), our new method. We implement restarting by resetting the averaging process when the duality gap has halved since the last time the averaging process was reset. After resetting, the initial iterate is set equal to the last

10 2

10 4

Gradient computations

10

-

2

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

4

10

-

1

Liar's Dice

10 2

10 4

Gradient computations

10

-

3

10

-

1

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

5

10

-

2

Battleship

ECyclicPDA single block

ECyclicPDA postorder

ECyclicPDA children

ECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

4

10

-

1

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

6

10

-

2

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 1

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

5

10

-

1

Battleship

ECyclicPDA

MP

CFR

+

PCFR

+

1

iterate we saw before restarting. Since the restarting heuristic is one we introduce, we distinguish restarted variants in the plots by prepending the name of the algorithm with 'r'. For example, when restarting is applied to CFR + , the label used is rCFR + .

We show the results of different block construction strategies when restarting is used on ECyclicPDA in Figure 3. As before, we take the combination of regularizer and averaging scheme that works best. Again, we can see that the different block construction strategies do not make a significant difference in Goofspiel (4 ranks) or in Leduc (13 ranks), while making a difference for Liar's Dice and Battleship. However, with restarting, the benefit of the children and postorder for Liar's Dice and Battleship is even more pronounced relative to the other block construction strategies; the gap is several orders of magnitude after 10 4 gradient computations. Note that while children performed worse than single block for Battleship previously, with restarting, children performs much better.

Finally, we compare the performance of the restarted version of our algorithm, with restarted versions of MP, CFR + , and PCFR + in Figure 4. As before, we take the combination of regularizer, averaging scheme, and block construction strategy that works best for ECyclicPDA, and the combination of regularizer and averaging scheme that works best for MP. Firstly, we note that the scale of the y-axis is different from Figure 2 for all games besides Leduc (13 ranks), because restarting tends to hit much higher levels of precision. We see that restarting provides significant benefits for PCFR + in Goofspiel (4 ranks) allowing it to converge to numerical precision, while the other algorithms do not benefit much. In Liar's Dice, restarted CFR + and PCFR + converge to numerical precision within 200 gradient computations, and restarted ECyclicPDA converges to numerical precision at 10 4 gradient computations. Additionally, restarted MP achieves a much lower duality gap. For Battleship, ECyclicPDA, MP, and PCFR + all benefit from restarting, and restarted ECyclicPDA is competitive with restarted PCFR + . Similar to the magnification in benefit of using blocks versus not using blocks when restarting in Liar's Dice and Battleship, we see that restarted ECyclicPDA achieves significantly better duality gap than MP in these games.

Wall-clock time experiments. In Table 1, we show the average wall-clock time per iteration of our algorithm instantiated with different block construction strategies as well as MP, CFR + , and PCFR + . It is clear from this table that the runtimes are pretty similar to each other, and this is without extensive optimization of our particular implementation. Clearly, our algorithm is at least as fast as MP per 'full' gradient computation. Since the computational bottleneck of gradient and prox computations becomes apparent in bigger games, Battleship demonstrates the speed of our algorithm relative to MP best.

10 2

10 4

Gradient computations

10

-

2

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Battleship

rECyclicPDA single block

rECyclicPDA postorder

rECyclicPDA children

rECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

10

10

-

4

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 1

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Battleship

rECyclicPDA

rMP

rCFR

+

rPCFR

+

1

Name

Goofspiel (4 ranks)

Liar's Dice

Leduc (13 ranks)

Battleship

Discussion. We develop the first cyclic block-coordinate-like method for two-player zero-sum EFGs. Our algorithm relies on the recursive nature of the prox updates for dilated regularizers, cycling through blocks that respect the partial order induced on decision points by the treeplex, and extrapolation to conduct pseudo-block updates, produce feasible iterates, and achieve O ( 1 T ) ergodic convergence. Furthermore, the runtime of our algorithm has no dependence on the number of blocks. We present empirical evidence that our algorithm generally outperforms MP, and is the first FOM to compete with CFR + and PCFR + on non-trivial EFGs. Finally, we introduce a restarting heuristic for EFG solving, and demonstrate often huge gains in convergence rate. An open question raised by our work is understanding what makes restarting work for methods used with regularizers besides the ℓ 2 regularizer (the only setting for which there exist linear convergence guarantees). This may be challenging because existing proofs require upper bounding the corresponding Bregman divergence (for a given nonℓ 2 regularizer) between iterates by the distance to optimality. This is difficult for entropy or any dilated regularizer because the initial iterate used by the algorithm after restarting may have entries arbitrarily close to zero even if they are guaranteed to not exactly be zero (as is the case for entropy). Relatedly, both our block-coordinate method and restarting have a much bigger advantage in some numerical instances (Battleship, Liar's Dice) than others (Leduc and Goofspiel); a crucial question is to understand what type of game structure drives this behavior.

References

Aviad Aberdam and Amir Beck. An accelerated coordinate gradient descent algorithm for non-separable composite optimization. Journal of Optimization Theory and Applications , 193 (1-3):219-246, 2021.

Ahmet Alacaoglu, Quoc Tran Dinh, Olivier Fercoq, and Volkan Cevher. Smooth primal-dual coordinate descent algorithms for nonsmooth convex optimization. In Advances in Neural Information Processing Systems , 2017.

Zeyuan Allen-Zhu, Zheng Qu, Peter Richtárik, and Yang Yuan. Even faster accelerated coordinate descent using non-uniform sampling. In Proceedings of International Conference on Machine Learning , 2016.

Ioannis Anagnostides, Gabriele Farina, and Tuomas Sandholm. Near-optimal ϕ -regret learning in extensive-form games. arXiv preprint arXiv:2208.09747 , 2022.

David Applegate, Oliver Hinder, Haihao Lu, and Miles Lubin. Faster first-order primal-dual methods for linear programming using restarts and sharpness. arXiv preprint arXiv:2105.12715 , 2022.

Yu Bai, Chi Jin, Song Mei, Ziang Song, and Tiancheng Yu. Efficient phi-regret minimization in extensive-form games via online mirror descent. In Advances in Neural Information Processing Systems , 2022.

Amir Beck and Luba Tetruashvili. On the convergence of block coordinate descent type methods. SIAM Journal on Optimization , 23(4):2037-2060, 2013.

Noam Brown and Tuomas Sandholm. Solving imperfect-information games via discounted regret minimization. In Proceedings of the AAAI Conference on Artificial Intelligence , 2019.

Yair Carmon, Yujia Jin, Aaron Sidford, and Kevin Tian. Coordinate methods for matrix games. arXiv preprint arXiv:2009.08447 , 2020.

Flavia Chorobura and Ion Necoara. Random coordinate descent methods for nonseparable composite optimization. arxiv preprint arXiv:2203.14368 , 2022.

Jelena Diakonikolas and Lorenzo Orecchia. Alternating randomized block coordinate descent. In Proceedings of International Conference on Machine Learning , 2018.

Gabriele Farina, Christian Kroer, Noam Brown, and Tuomas Sandholm. Stable-predictive optimistic counterfactual regret minimization. In Proceedings of the International Conference on Machine Learning , 2019.

Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Optimistic regret minimization for extensive-form games via dilated distance-generating functions. In Advances in Neural Information Processing Systems , 2019.

Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Better regularization for sequential decision spaces: Fast convergence rates for Nash, correlated, and team equilibria. In Proceedings of the ACM Conference on Economics and Computation , 2021.

Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Faster game solving via predictive blackwell approachability: Connecting regret matching and mirror descent. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

Gabriele Farina, Ioannis Anagnostides, Haipeng Luo, Chung-Wei Lee, Christian Kroer, and Tuomas Sandholm. Near-optimal no-regret learning dynamics for general convex games. Advances in Neural Information Processing Systems , 2022.

Gabriele Farina, Chung-Wei Lee, Haipeng Luo, and Christian Kroer. Kernelized multiplicative weights for 0/1-polyhedral games: Bridging the gap between learning in extensive-form and normal-form games. In Proceedings of the International Conference on Machine Learning , 2022.

Olivier Fercoq. Quadratic error bound of the smoothed gap and the restarted averaged primaldual hybrid gradient. arXiv preprint arXiv:2206.03041 , 2023.

Jerome Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of statistical software , 33(1):1, 2010.

Yuan Gao, Christian Kroer, and Donald Goldfarb. Increasing iterate averaging for solving saddle-point problems. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

Andrew Gilpin, Javier Pena, and Tuomas Sandholm. First-order algorithm with O (ln(1 /ϵ )) convergence for ϵ -equilibrium in two-person zero-sum games. Mathematical programming , 133 (1):279-298, 2012.

Mert Gürbüzbalaban, Asuman Ozdaglar, Pablo A Parrilo, and N Denizcan Vanli. When cyclic coordinate descent outperforms randomized coordinate descent. In Advances in Neural Information Processing Systems , 2017.

Samid Hoda, Andrew Gilpin, Javier Pena, and Tuomas Sandholm. Smoothing techniques for computing Nash equilibria of sequential games. Mathematics of Operations Research , 35(2): 494-512, 2010.

Christian Kroer, Gabriele Farina, and Tuomas Sandholm. Solving large sequential games with the excessive gap technique. In Advances in Neural Information Processing Systems , 2018.

Christian Kroer, Kevin Waugh, Fatma Kılınç-Karzan, and Tuomas Sandholm. Faster algorithms for extensive-form game solving via improved smoothing functions. Mathematical Programming , pages 1-33, 2020.

Harold William Kuhn and Albert William Tucker, editors. 11. Extensive Games and the Problem of Information , pages 193-216. Princeton University Press, 2016.

Chung-Wei Lee, Christian Kroer, and Haipeng Luo. Last-iterate convergence in extensive-form games. In Advances in Neural Information Processing Systems , 2021.

Qihang Lin, Zhaosong Lu, and Lin Xiao. An accelerated randomized proximal coordinate gradient method and its application to regularized empirical risk minimization. SIAM Journal on Optimization , 25(4):2244-2273, 2015.

Viliam Lisý, Marc Lanctot, and Michael Bowling. Online monte carlo counterfactual regret minimization for search in imperfect information games. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems , AAMAS '15, page 27-36. International Foundation for Autonomous Agents and Multiagent Systems, 2015. ISBN 9781450334136.

Ji Liu, Stephen J. Wright, Christopher Ré, Victor Bittorf, and Srikrishna Sridhar. An asynchronous parallel stochastic coordinate descent algorithm. arXiv preprint arxiv:1311.1873 , 2014.

Mingyang Liu, Asuman Ozdaglar, Tiancheng Yu, and Kaiqing Zhang. The power of regularization in solving extensive-form games. In Proceedings of the International Conference on Learning Representations , 2023.

Rahul Mazumder, Jerome H Friedman, and Trevor Hastie. Sparsenet: Coordinate descent with nonconvex penalties. Journal of the American Statistical Association , 106(495):1125-1138, 2011.

Arkadi Nemirovski. Prox-method with rate of convergence o (1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15(1):229-251, 2004.

Yu. Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization , 22(2):341-362, 2012.

Yurii Nesterov. Dual extrapolation and its applications to solving variational inequalities and related problems. Mathematical Programming , 109(2-3):319-344, 2007.

Sheldon M. Ross. Goofspiel - the game of pure strategy. Journal of Applied Probability , 8(3): 621-625, 1971.

Ankan Saha and Ambuj Tewari. On the nonasymptotic convergence of cyclic coordinate descent methods. SIAM Journal on Optimization , 23(1):576-601, 2013.

Hao-Jun Michael Shi, Shenyinying Tu, Yangyang Xu, and Wotao Yin. A primer on coordinate descent algorithms. arXiv preprint arXiv:1610.00040 , 2016.

Chaobing Song and Jelena Diakonikolas. Fast cyclic coordinate dual averaging with extrapolation for generalized variational inequalities. arXiv preprint arXiv:2102.13244 , 2021.

Finnegan Southey, Michael P. Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings, and Chris Rayner. Bayes' bluff: Opponent modelling in poker. arXiv preprint arXiv:1207.1411 , 2012.

Oskari Tammelin. Solving large imperfect information games using CFR+. arXiv preprint arXiv:1407.5042 , 2014.

Oskari Tammelin, Neil Burch, Michael Johanson, and Michael Bowling. Solving heads-up limit Texas hold'em. In Twenty-Fourth International Joint Conference on Artificial Intelligence , 2015.

Paul Tseng. On linear convergence of iterative methods for the variational inequality problem. Journal of Computational and Applied Mathematics , 60(1-2):237-252, 1995.

Bernhard von Stengel. Efficient computation of behavior strategies. Games and Economic Behavior , 14(2):220-246, 1996.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In Proceedings of International Conference on Learning Representations , 2021.

Stephen J. Wright. Coordinate descent algorithms. Mathematical Programming , 151(1):3-34, 2015.

Tong Tong Wu and Kenneth Lange. Coordinate descent algorithms for lasso penalized regression. The Annals of Applied Statistics , 2(1):224-244, 2008.

Yuchen Zhang and Xiao Lin. Stochastic primal-dual coordinate method for regularized empirical risk minimization. In Proceedings of International Conference on Machine Learning , 2015.

Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In Advances in Neural Information Processing Systems , 2007.

A Additional Related Work

There has been significant work on FOMs for two-player zero-sum EFG solving. Because this is a BSPP, off-the-shelf FOMs for BSPPs can be applied, with the caveat that proximal oracles are required. The standard Euclidean distance has been used in some cases [21], but it requires solving a projection problem that takes O ( n log 2 n ) time, where n is the dimension of a player's decision space [16, 21]. While this is 'nearly' linear time, such projections have not been used much in practice. Proximal oracles have instead been based on dilated regularizers [23], which lead to a proximal update that can be performed with a single pass over the decision space. With the dilated entropy regularizer, this can be performed in linear time, and this regularizer leads to the strongest bounds on game constants that impact the convergence rate of proximal-oracle-based FOMs [14, 25]. More recently, it has been shown that a specialized kernelization can be used to achieve linear-time proximal updates and stronger convergence rates specifically for the dilated entropy with optimistic online mirror descent through a correspondence with optimistic multiplicative weights on the exponentially-many vertices of the decision polytope [6, 17]. Yet this approach was shown to have somewhat disappointing numerical performance in Farina et al. [17], and thus is less important practically despite its theoretical significance.

A completely different approach for first-order-based updates is the CFR framework, which decomposes regret minimization on the EFG decision sets into local simplex-based regret minimization [49]. In theory, CFR-based results have mostly led to an inferior T -1 / 2 rate of convergence, but in practice the CFR framework instantiated with regret matching + (RM + ) [41] or predictive RM + (PRM + ) [15] is the fastest approach for essentially every EFG setting. RM + is often fastest for 'poker-like' EFGs, while PRM + is often fastest for other classes of games [15]. Improved rates on the order of T -3 / 4 [12] and log T/T [4] have been achieved within the CFR framework, but only while using regret minimizers that lead to significantly worse practical performance (in particular, numerically these perform worse than the best 1 /T FOMs such as mirror prox with appropriate stepsize tuning).

In the last few years there has been a growing literature on last-iterate convergence in EFGs. There, the goal is to show that one can converge to an equilibrium without averaging the iterates generated by a FOM or CFR-based method. It has long been known that with the Euclidean regularizer, it is possible to converge at a linear rate in last iterate with e.g., the extragradient method (a.k.a. mirror prox with the Euclidean regularizer) on BSPPs with polyhedral decision sets, as they are in EFGs [21, 43, 45]. More recently, it has been shown that a linear rate can be achieved with certain dilated regularizers [27], with the kernelization approach of Farina et al. [17], and in a regularized CFR setup [31]. At this stage, however, these last-iterate results are of greater theoretical significance than practical significance, because the linear rate often does not occur until after quite many iterations, and typically the methods do not match the performance of ergodic methods at reasonable time scales. For this reason, we do not compare to last-iterate algorithms in our experiments.

B Treeplex Example

As an example, consider the treeplex of Kuhn poker [26] adapted from [13] shown in Figure 5. Kuhn poker is a game played with a three card deck: jack, queen, and king. In this case, for example, we have J X = { 0 , 1 , 2 , 3 , 4 , 5 , 6 } , p 0 = ∅ , p 1 = p 2 = p 3 = (0 , start ) , p 4 = (1 , check ) , p 5 = (2 , check ) , p 6 = (3 , check ) , A 0 = { start } , A 1 = A 2 = A 3 = { check, raise } , A 4 = A 5 = A 6 = { fold , call } , C (0 , start ) = { 1 , 2 , 3 } , C (1 , raise ) = C (1 , raise ) = C (2 , raise ) = C (3 , raise ) = C (4 , fold ) = C (5 , fold ) = C (6 , fold ) = C (4 , call ) = C (5 , call ) = C (6 , call ) = ∅ , ↓ 0 = J X , ↓ 1 = { 1 , 4 } , ↓ 2 = { 2 , 5 } , ↓ 3 = { 3 , 6 } , ↓ 4 = { 4 } , ↓ 5 = { 5 } , ↓ 6 = { 6 } . In this case, ∅ represents the empty sequence.

C Proofs

C.1 Proof of Proposition 3.1

Proof. The claim that x k ∈ X , y k ∈ Y is immediate from the algorithm description, as both are solutions to constrained optimization problems with these same constraints.

start

queen

king

1

IX

X2

X3

check

raise

check

raise

check

raise

check raise

checkraise

checkraise

X5

X6

/

call

1

fold

call

1

For the remaining claim, observe first that

Recall from Algorithm 1 that

Define the function under the max defining y k by Ψ k . Then as Ψ k ( · ) is the sum of -ψ ( · ) and linear terms, we have D Ψ k ( · , y ) = -D ψ ( · , y ) , for any y . Further, as Ψ k is maximized by y k , we have Ψ k ( v ) ≤ Ψ k ( y k ) -D ψ ( v , y k ) . Thus, it follows that

Using the same ideas for the primal side, we have

Combining (C.1) and (C.2),

To complete the proof, it remains to combine the definition of E k from the proposition statement with the last inequality.

C.2 Proof of Theorem 3.2

For notational convenience, in this proof we define vectors ˆ x k and ˆ y k by ˆ x j k = x j k x p j k x p j k -1 for

To prove the theorem, we first prove the following auxiliary lemma which bounds the inner product terms appearing in the error terms E k .

and

Proof. Observe first that, by Algorithm 1,

Additionally, by definition (see Eq. (3.1)), ∑ s t =1 M t, 1: t = M -M x = M y . Hence,

The first two terms in (C.4) telescope, so we focus on bounding -η k -1 ⟨ M y ( x k -1 -ˆ x k -1 ) , y k -1 -y k ⟩ . By definition of ˆ x k , for all j ∈ J X ,

By the definition of a treeplex, each vector x j k -1 x p j k -1 belongs to a probability simplex of the appropriate size. This further implies that

where the notation [ a j ] j ∈J X is used to denote the vector with entries a j , for j ∈ J X . The first inequality in (C.5) holds for any ℓ p norm ( p ≥ 1 ), by its definition and 〈 1 , x j k -1 〉 = 1 , ∀ j. Thus, applying the definitions of the norms from the preliminaries,

where the last line is by Young's inequality and holds for any α > 0 .

On the other hand, recalling that M x = ∑ s -1 t =1 M t,t +1: s , we also have

Observe that in (C.7) the first two terms telescope and thus we only need to focus on bounding the last term. Applying the definitions of dual and matrix norms from Section 2 and using Young's inequality, we have that for any α > 0 ,

Hence, combining (C.3)-(C.8), we can conclude that

completing the proof of the first claim.

Similarly, we observe from Algorithm 1 that

Observing that ∑ s t =1 M 1: t -1 ,t = M x and ∑ s t =1 M t : s,t = M y , using the same sequence of arguments as for bounding (C.3), we can conclude that for any β > 0 ,

completing the proof.

Proof Theorem 3.2. Recalling the definition of E k , by Lemma C.1,

Recalling that ψ is c ψ -strongly convex, ϕ is c ϕ -strongly convex, setting α = β = √ c ϕ c ψ , and using

Telescoping (C.10) and recalling that η 0 = 0 , we now have

Observe that Gap u , v ( · , · ) is linear in both its arguments. Hence, Gap u , v ( ¯ x K , ¯ y K ) = 1 H K ∑ K k =1 η k Gap u , v ( x k , y k ) . Applying Proposition 3.1 and combining with (C.11), we now have

Let ( x ∗ , y ∗ ) ∈ X × Y be any primal-dual solution to (PD). Then Gap ( x ∗ , y ∗ ) ( ¯ x K , ¯ y K ) ≥ 0 and we can conclude that

To complete bounding the gap, it remains to argue that the right-hand side of (C.12) is bounded by D ϕ ( u , x 0 ) + D ψ ( v , y 0 ) -µ x µ x + µ y D ϕ ( u , x K ) -µ y µ x + µ y D ψ ( v , y K ) . This is done using the same sequence of arguments as in bounding E k and is omitted.

Further, using that D ϕ ( · , · ) ≥ 0 , D ψ ( · , · ) ≥ 0 , we can also conclude that

Finally, setting η k = √ c ϕ c ψ µ x + µ y for all k ≥ 1 immediately leads to the conclusion that H K = K √ c ϕ c ψ µ x + µ y , as H K = ∑ K k =1 η k , by definition. The last bound is by setting sup u ∈X , v ∈Y { D ϕ ( u , x 0 )+ D ψ ( v , y 0 ) } H K ≤ ϵ, and solving for K.

D Algorithm Implementation Details

In Algorithm 2, we present an implementation-specific version of ECyclicPDA, in order to make it clear that our algorithm can be implemented without any extra computation compared to the

computation needed for gradient and prox updates in MP. Note that MP performs two gradient computations and two prox computations per player, due to how it achieves 'extrapolation'; we want to argue that we perform an equivalent number operations as needed for a single gradient computation and prox computation per player. Note that the overall complexity of first-order methods when applied to EFGs is dominated by the gradient and prox update computations; this is why we compare our algorithm to MP on this basis. The key differences from Algorithm 1 are that we explicitly use ˆ x k and ˆ y k to represent the behavioral strategy that is computed via the partial prox updates (which are then scaled at the end of a full iteration of our method to x k and y k ), and that we use ˆ h j k and ˆ g j k to accumulate gradient contributions from decision points that occur underneath j , to make the partial prox update explicit.

In Lines 8 and 13, we are only dealing with the columns and rows, respectively, of the payoff matrix that correspond to the current block number t , which means that as t ranges from 1 to s , for the computation of the gradient, we will only consider each column and row, respectively, once, as would have to be done in a full gradient computation for MP.

The more difficult aspect of the implementation is ensuring that we do the same number of operations for the prox computation in ECyclicPDA as an analogous single prox computation in MP. We achieve this by applying the updates in Proposition 2.2 only for the decision points in the current block, in Lines 9 to 12 for x and 14 to 17 for y .

We focus on the updates for x ; the argument is analogous for y . When applying this local prox update for decision point j ∈ J X ( t ) , we have already correctly computed ˆ h j , the contributions to the gradient for the local prox update that originate from the children of j , again because the blocks represent the treeplex ordering; in particular, whenever we have encountered a child decision point of j in the past, we accumulate its contribution to the gradient for its parent at ˆ h p j . Since the prox update decomposition from Proposition 2.2 has to be applied for every single decision point in J X in a full prox update (as done in MP), we again do not incur any dependence on the number of blocks.

E Description of EFG Benchmarks

We provide game descriptions of the games we run our experiments on below. Our game descriptions are adapted from Farina et al. [15]. In Table 2, we provide the number of sequences for player x ( n ), the number of sequences for player y ( m ), and the number of leaves in the game (NNZ ( M ) ).

E.1 Goofspiel (4 ranks)

Goofspiel is a card-based game that is a standard benchmark in the EFG-solving community [36]. In the version that we test on, there are 4 unique cards (ranks), and there are 3 copies of each rank, divided into 3 separate decks. Each player gets a deck, and the third deck is known as the prize deck. Cards are randomly drawn from the prize deck, and each player submits a bid for the drawn card by submitting a card from one of their respective decks, the value of which represents their bid. Whoever submits the higher bid wins the card from the prize deck. Once all the cards from the prize deck have been drawn, bid on, and won by one of the players, the game terminates, and the payoffs for players are given by the sum of the prize cards they won.

E.2 Liar's Dice

Liar's Dice is another standard benchmark in the EFG-solving community [29]. In the version that we test on, each player rolls an unbiased six-sided die, and they take turns either calling higher bids or challenging the other player. A bid consists of a combination of a value v between one and six, and a number of dice between one and two, n , representing the number of dice between the two players that has v pips showing. A higher bid involves either increasing n holding v fixed, increasing v holding n fixed, or both. When a player is challenged (or the highest possible bid of 'two dice each showing six pips' is called), the dice are revealed, and whoever is correct wins 1 (either the challenger if the bid is not true, or the player who last called a bid, if the bid is true), and the other player receives a utility of -1.

E.3 Leduc (13 ranks)

Leduc is yet another standard benchmark in the EFG-solving community [40] and is a simplified version of Texas Hold'Em. In the version we test on, there are 13 unique cards (ranks), and there are 2 copies of each rank (half the size of a standard 52 card deck). There are two rounds of betting that take place, and before the first round each player places an ante of 1 into the pot, and is dealt a single pocket (private) card. In addition, two cards are placed face down, and these are community cards that will be used to form hands. The two hands that can be formed with the community cards are pair, and highest card.

During the first round of betting, player 1 acts first. There is a max of two raises allowed in each round of betting. Each player can either check, raise, or fold. If a player folds, the other player immediately wins the pots and the game terminates. If a player checks, the other player has an opportunity to raise if they have not already previously checked or raised, and if they previously checked, the game moves on to the next round. If a player raises, the other player has an opportunity to raise if they have not already previously raised. The game then moves on the second round, during which one of the community cards is placed face up, and then similar betting dynamics as the first round take place. After the second round terminates, there is a showdown, and whoever can form the better hand (higher ranked pair, or highest card) with the community cards takes the pot.

E.4 Battleship

This is an instantiation of the classic board game, Battleship, in which players take turns shooting at the opposing player's ships. Before the game begins, the players place two ships of length 2 and value 4, on a grid of size 2 by 3. The ships need to be placed in a way so that the ships take up exactly four spaces within the grid (they do not overlap with each other, and are contained entirely in the grid). Each player gets three shots, and players take turns firing at cells of their opponent's grid. A ship is sunk when the two cells it has been placed on have been shot at. At the end of the game, the utility for a player is the difference between the cumulative value of the opponent's sunk ships and the cumulative value of the player's sunk ships.

F Block Construction Strategies

As discussed in the main paper, the postorder block construction strategy can be viewed as traversing the decision points of the treeplex in postorder, treating decision points with the same parent sequence as the same node, and then greedily putting decision points in the same block until we reach a decision point that has a child decision point in the current block (at which point we start a new block). We make this postorder traversal and greedy block construction explicit in Algorithm 3.

In Algorithm 4 we provide pseudocode for constructing blocks using the children block construction strategy. As discussed in the main paper, the children block construction strategy corresponds to placing decision points with the same parent decision point (same decision point at which their parent sequences start at) in the same block. In our implementation, instead of doing a bottom-up traversal, we do a top down implementation, and at the end, reverse the order of the blocks (this allows us to respect the treeplex ordering).

In both Algorithm 3 and Algorithm 4, ∅ represents the empty sequence.

Algorithm 3 Postorder Block Construction

Algorithm 4 Children Block Construction

Wecan now illustrate each of the block construction strategies on the treeplex for player 1 in Kuhn that was presented in Appendix B. If we use single block, then we have J X (1) = J X = { 0 , 1 , 2 , 3 , 4 , 5 , 6 } . If we use infosets, then we have J X ( i ) = { 7 -i } for i ∈ { 1 , 2 , 3 , 4 , 5 , 6 , 7 } (we have to subtract in order to label the infosets in a manner that respects the treeplex ordering). If we use children, then we have J X (1) = { 4 } , J X (2) = { 5 } , J X (3) = { 6 } , J X (4) = { 1 , 2 , 3 } , and J X (5) = { 0 } . If we use postorder, then we have J X (1) = { 4 , 5 , 6 } , J X (2) = { 1 , 2 , 3 } , and J X (3) = { 0 } .

Note that in the implementation of our algorithm, it is not actually important that the number of blocks for both players are the same; if one player has more blocks than the other, for iterations of our algorithm that correspond to block numbers that do not exist for the other player, we just do not do anything for the other player. Nevertheless, the output of the algorithm does not change if we combine all the blocks for the player with more blocks after the minimum number of blocks between the two players is exceeded, into one block. For example, if player 1 has s 1 blocks, and player 2 has s 2 blocks, with s 1 < s 2 , we can actually combine blocks s 1 +1 , . . . , s 2 all into the same block for player 2, and this would not change the execution of the algorithm. This is what we do in our implementation.

Additionally, given a choice of a partition of decision points into blocks, there may exist many permutations of decision points within the blocks which satisfy the treeplex ordering of the decision points. Unless the game that is being tested upon possesses some structure which leads to a single canonical ordering of the decision points (which respects the treeplex ordering), an arbitrary decision needs to be made regarding what order is used.

G Experiments

G.1 Additional Experimental Details

Block Construction Strategy Comparison. In this section, we provide additional plots (Figures 6 to 14) comparing different block construction strategies for our algorithm, for specific choices of regularizer and averaging scheme. Note that for the games for which there is a benefit to using blocks (Liar's Dice and Battleship), the benefit is generally apparent across different regularizers and averaging schemes. Furthermore, when there is not a benefit for a particular regularizer and averaging scheme, there is no significant cost either (using blocks does not lead to worse performance).

Block Construction Strategy Comparison with Restarts. We repeat a similar analysis as above (comparing the block construction strategies holding a regularizer and averaging scheme fixed) but this time with the adaptive restarting heuristic applied to our algorithm: the plots can be seen in (Figures 15 to 23). As mentioned in Section 4, we prepend the algorithm name with 'r' to denote the restarted variant of the algorithm in the plots.

As discussed in the main body, the trend of the benefit of using blocks being more pronounced with restarting (for games for which blocks are beneficial) holds generally even when holding the regularizer and averaging scheme fixed. This can be seen by comparing each of the restarted block construction strategy comparison plots with the corresponding non-restarted block construction strategy comparison plot.

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

1

10 0

Battleship

ECyclicPDA single block

ECyclicPDA postorder

ECyclicPDA children

ECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Battleship

ECyclicPDA single block

ECyclicPDA postorder

ECyclicPDA children

ECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

2

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Battleship

ECyclicPDA single block

ECyclicPDA postorder

ECyclicPDA children

ECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

3

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

1

10 0

Battleship

ECyclicPDA single block

ECyclicPDA postorder

ECyclicPDA children

ECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

2

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

3

10

-

1

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

3

10 0

Battleship

ECyclicPDA single block

ECyclicPDA postorder

ECyclicPDA children

ECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

2

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

3

10 0

Battleship

ECyclicPDA single block

ECyclicPDA postorder

ECyclicPDA children

ECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

4

10

-

1

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

1

10 0

Battleship

ECyclicPDA single block

ECyclicPDA postorder

ECyclicPDA children

ECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

2

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

3

10

-

1

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

5

10

-

2

Battleship

ECyclicPDA single block

ECyclicPDA postorder

ECyclicPDA children

ECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

2

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

4

10

-

1

Battleship

ECyclicPDA single block

ECyclicPDA postorder

ECyclicPDA children

ECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Liar's Dice

10 2

10 4

Gradient computations

10

-

1

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

1

10 0

Battleship

rECyclicPDA single block

rECyclicPDA postorder

rECyclicPDA children

rECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

2

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

9

10

-

3

Battleship

rECyclicPDA single block

rECyclicPDA postorder

rECyclicPDA children

rECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

2

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

3

10 0

Battleship

rECyclicPDA single block

rECyclicPDA postorder

rECyclicPDA children

rECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Liar's Dice

10 2

10 4

Gradient computations

10

-

1

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

1

10 0

Battleship

rECyclicPDA single block

rECyclicPDA postorder

rECyclicPDA children

rECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

2

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Battleship

rECyclicPDA single block

rECyclicPDA postorder

rECyclicPDA children

rECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

2

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

3

10 0

Battleship

rECyclicPDA single block

rECyclicPDA postorder

rECyclicPDA children

rECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Liar's Dice

10 2

10 4

Gradient computations

10

-

1

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

1

10 0

Battleship

rECyclicPDA single block

rECyclicPDA postorder

rECyclicPDA children

rECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

3

10

-

1

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Battleship

rECyclicPDA single block

rECyclicPDA postorder

rECyclicPDA children

rECyclicPDA infosets

1

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

2

10 0

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10

-

1

10 0

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

3

10 0

Battleship

rECyclicPDA single block

rECyclicPDA postorder

rECyclicPDA children

rECyclicPDA infosets

1

Regularizer Comparison. In this section (Figures 24 to 26) we compare the performance of ECyclicPDA and MP instantiated with different regularizers for each averaging scheme, against the performance of CFR + and PCFR + .

It is apparent from these plots, that our algorithm generally outperforms MP, holding the averaging scheme and regularizer fixed. This can be seen by examining the corresponding figure for a choice of averaging scheme, and noting that for any given regularizer, the corresponding MP line is generally above the corresponding ECyclicPDA line.

Regularizer Comparisons with Restarts. We repeat a similar analysis in this section (Figures 27 to 29), instead now comparing the performance of ECyclicPDA and MP instantiated with different regularizers for each averaging scheme, against the performance of CFR + and PCFR + , when all methods are restarted. The trend noted above of our method generally beating MP, even holding the regularizer and averaging scheme fixed, still holds even when restarting.

10 2

10 4

Gradient computations

10

-

4

10

-

1

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

6

10

-

2

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 1

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

5

10

-

1

Battleship

ECyclicPDA dilated entropy

ECyclicPDA dilatable global entropy

ECyclicPDA dilated

/lscript

2

MP dilated entropy

MP dilatable global entropy

MP dilated

/lscript

2

CFR

+

PCFR

+

1

10 2

10 4

Gradient computations

10

-

4

10

-

1

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

6

10

-

2

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 1

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

5

10

-

1

Battleship

ECyclicPDA dilated entropy

ECyclicPDA dilatable global entropy

ECyclicPDA dilated

/lscript

2

MP dilated entropy

MP dilatable global entropy

MP dilated

/lscript

2

CFR

+

PCFR

+

1

10 2

10 4

Gradient computations

10

-

4

10

-

1

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

6

10

-

2

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 1

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

5

10

-

1

Battleship

ECyclicPDA dilated entropy

ECyclicPDA dilatable global entropy

ECyclicPDA dilated

/lscript

2

MP dilated entropy

MP dilatable global entropy

MP dilated

/lscript

2

CFR

+

PCFR

+

1

10 2

10 4

Gradient computations

10

-

10

10

-

4

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 1

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Battleship

rECyclicPDA dilated entropy

rECyclicPDA dilatable global entropy

rECyclicPDA dilated

/lscript

2

rMP dilated entropy

rMP dilatable global entropy

rMP dilated

/lscript

2

rCFR

+

rPCFR

+

1

10 2

10 4

Gradient computations

10

-

10

10

-

4

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 1

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Battleship

rECyclicPDA dilated entropy

rECyclicPDA dilatable global entropy

rECyclicPDA dilated

/lscript

2

rMP dilated entropy

rMP dilatable global entropy

rMP dilated

/lscript

2

rCFR

+

rPCFR

+

1

10 2

10 4

Gradient computations

10

-

10

10

-

4

Duality gap

Goofspiel (4 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Liar's Dice

10 2

10 4

Gradient computations

10

-

2

10 1

Leduc (13 ranks)

10 2

10 4

Gradient computations

10

-

10

10

-

4

Battleship

rECyclicPDA dilated entropy

rECyclicPDA dilatable global entropy

rECyclicPDA dilated

/lscript

2

rMP dilated entropy

rMP dilatable global entropy

rMP dilated

/lscript

2

rCFR

+

rPCFR

+

1

Additional wall-clock time experiments In Table 3, we show the wall-clock time required to reach a duality gap of 10 -4 . It is clear that our algorithm is competitive with MP: MP and its restarted variant time out on Leduc, and MP times out on Battleship (while our algorithm does not even take close to 30 seconds). Furthermore, we are outperforming CFR + and its restarted variant in Battleship.

Peer reviews:
comments: SUMMARY:
This work proposes a cyclic coordinate descent method to solve the two-player zero-sum extended form game (EFG) and derives the convergence.

STRENGTHS:
To me, solving problems with non-separable constraints by coordinate-descent-type methods is novel and interesting. Therefore, I believe that this is indeed a contribution.

WEAKNESSES:
I'm not familiar with this area so please forgave me if I made some mistakes.

1. Possibly expensive computational cost per iteration. Following the literature survey in the manuscript, I read reference [1] which studies coordinate descent for solving optimization problems with non-separable non-smooth objective functions. I found that in both the algorithm in [1] and the algorithm in this paper, although only partial gradient is needed at each update, the proximal step with respect to the whole nonsmooth function is needed. More specifically, in Lines 8 and 11, the argmin step might be difficult and computationally expensive since the argmin step is over the whole X and Y and involve all the blocks.

2.  This concern follows the first one but focuses on the experiments: the comparison with non-coordinate algorithms may be unfair. The current comparison is in terms of the number of full gradient computations. However, gradient computation is not the only computation cost, Lines 8 and 11 also cause computation costs. I would suggest a comparison in terms of wall clock time.

QUESTIONS:
no

LIMITATIONS:
I do not see any potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper combines the local prox update technique with the extrapolated cyclic algorithm and proposes the ECyclicPDA algorithm. While the local prox update technique is well understood, the authors reinterpret it as a coordinate method (CM). Then, the method is combined with a new extrapolated method. Theoretical analysis shows that the proposed algorithm can converge at a rate of O(1/T).\n\nSTRENGTHS:\n1. This paper presents a new understanding of the local update rules of OMD with dilated DGF. The combination of local OMD updates and extrapolated updates is original and interesting.\n2. The theoretical results seem sound.\n3. The paper is well-written.\n\nWEAKNESSES:\n1. It is hard to understand why we want to reinterpret the well-known local update rules of OMD with dilated DGF to CM. As we know, people already compute the strategy in a bottom-up fashion in previous work [1, 2, 3]. So, I think the main contribution is the combination of the local update method and the extrapolated cyclic method.\n2. However, the ECyclicPDA algorithm seems less efficient than the traditional MP algorithm. For every infoset, an extra traversal of the subgame rooted at the infoset is needed to compute the “extrapolated” vector. This could be infeasible for large-scale games.\n3. The experimental results show that the proposed ECyclicPDA performs better than MP. The results are not surprising when considering that ECyclicPDA can be very time-consuming.\n4. The algorithm does not compare with related optimistic OMD [4] algorithms for EFGs.\n\n\n[1] Farina, Gabriele, Christian Kroer, and Tuomas Sandholm. \"Optimistic regret minimization for extensive-form games via dilated distance-generating functions.\" Advances in neural information processing systems 32 (2019).\n[2] Farina, Gabriele, Christian Kroer, and Tuomas Sandholm. \"Better regularization for sequential decision spaces: Fast convergence rates for Nash, correlated, and team equilibria.\" arXiv preprint arXiv:2105.12954 (2021).\n[3] Liu, Weiming, et al. \"Equivalence analysis between counterfactual regret minimization and online mirror descent.\" International Conference on Machine Learning. PMLR, 2022.\n[4] Lee, Chung-Wei, Christian Kroer, and Haipeng Luo. \"Last-iterate convergence in extensive-form games.\" Advances in Neural Information Processing Systems 34 (2021): 14293-14305.\n\nQUESTIONS:\nnone\n\nLIMITATIONS:\nThe authors have stated the limitations properly. No potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper introduces ECyclicPDA, a new first order method for solving extensive form games.  The main idea is to implement something in the spirit of coordinate descent where improving directions can be found by considering a part of the current iterate in isolation.  Empirical results show it generally outperforms other first-order methods and approaches the performance of CFR+.  Additionally, a heuristic to restart the process is introduced which improves the performance of both ECyclicPDA and CFR+

STRENGTHS:
ECyclicPDA is a nice contribution that takes the growing collection of FOMs for EFGs in a new direction.  The analysis seems to involve non-trivial technical innovations.  The empirical results are convining and show continued progress toward cloasing the gap with CFR-based approaches.

The restarting heuristic is also very interesting, and the ability to get performance improvements with CFR+ is particularly nice (and intuitive in hindsight).

WEAKNESSES:
As far as I can tell, the details of how restarting is implemented are never clearly explained.  The clearest explanation I can find is in the introduction on lines 88-91, but even taking that as the full specification, I don’t see where the tuning of the parameter about when to restart is specified.  It would also be nice to have a bit of explanation / intuition for how restarting is benefiting various algorithms.  For CFR+ I can see how resetting the regret sums and the averaging process could be beneficial.  For FOMs I’m less clear why it is useful.  Is it purely from resetting the averaging process?

QUESTIONS:
Please comment on the questions about resetting.

LIMITATIONS:
Adequate
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe draft considers solving the extended-form game (EFG) with extrapolated block-coordinate descent methods, proving that it achieves O(1/T) convergence. The authors further show that with a restarting strategy, the proposed algorithm may be comparable sometimes to state-of-the-art algorithms like CFR+. In my understanding, the proposed algorithm is essentially a generalization of the CODER algorithm proposed by Song [37] in a way from coordinate descent (CD) to blockwise CD and specializes the domain to be linear functional and treeplex. I.e., the algorithm is essentially the same (line-by-line corresponded) as CODER but generalizes the scalar coordinate to blocks of \"separable\" variables.  For the specialization, the authors focus on the context of EFG with bilinear score function and 2-player treeplex (linear probability simplex), where the prox is taken on L1-norm instead of the L2-norm by CODER. Implementation-wise, the proposed algorithm uses the recursive structure of the treeplex for efficient linear time computation.\n\nSTRENGTHS:\nThe major contribution of the draft is that it provides an instance to efficiently implement the CODER algorithm on EFG with blockwise L1 proximal and shows that with restarting, such implementation may be comparable with the state-of-the-art algorithm like CFR+. It also analyzes the scenario and provides an O(1/T) convergence rate without dependence on the number of variables. To summarize, the draft shows that a simple CODER extrapolation works well when taking L1 specialization on EFG.\n\nWEAKNESSES:\nHowever, I would like to claim that the original CODER method may already cover the blockwise updates. I.e., if you look into the proof of CODER, it generally does not use the scalar property of the proximal and treats variables as blocks. They are using the $d$-dimension notation for simplicity and already claiming they are doing block CD in their abstract. In the EFG case, the d=2, so naturally (if the proximal works), the convergence complexity doesn't depend on the number of variables since blocks=2 is a constant. So the draft's theoretical result (independence on the number of variables) is not surprising. And the proposed method does not relax the \"separability assumption\" since the 2-player treeplex is separable as two blocks. The major difference/contribution here is that it performs the analysis on the L1 proximal and does the analysis.\n\nFurther, although it's good to see that the simple method works with restarting, the restarting part doesn't have theoretical support. I.e., without restarting, the theoretically supported method is not comparable to the state-of-the-art. And with restarting, the \"full\" method is comparable but not really much better than CFR+. Thus these factors put the paper in a borderline condition. However, the analysis seems solid, and I would give a borderline acceptance.\n\nQUESTIONS:\n1. (Figure 2) Only doing the step-size optimization for your method and not the baseline is tricky for the experiment. For fairness, it should be either all constant step-size or all step-size tuned for the baseline. Otherwise, the reader cannot be sure whether the proposed method is better than MP.\n2. (Figure 4) The vertical drop in the curves looks like either a numerical issue or a wrong optimal value.\n3. (Algorithm 1) Adding parenthesis and denoting what you maintain in the computation may be good for clarity.\n\nLIMITATIONS:\nNot applicable since the paper does not have a limitation section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method for solving Extensive-Form Games based on a block-coordinate approach. The authors motivate and explain their idea and experimentally evaluate its performance in terms of primal-dual gap in four different games. The method features a favorable theoretical convergence rate but the empirical results are often comparable or worse than existing methods in terms of the duality gap.\n\nSTRENGTHS:\n- The paper is well written and free of grammar and stylistic issues.\n- The introduction to the problem is very approachable even to domain non-experts.\n- Experimental validation of the block construction strategy choice.\n- Favorable converge properties.\n\nWEAKNESSES:\n- It seems that in empirical studies the method is consistently worse than PCFR+ (Fig 2 and 4). I have a difficulty to find a complete justification for this in the paper.\n- Only a theoretical convergence rate and not an actual computation cost measured in units of time is reported.\n- The plots lack error bars (unless no variance is possible to obtain).\n- The contributions are explained indirectly. They could be listed in a more compact and more explicit form (e.g., a numbered list) to allow for critical assessment.\n\nQUESTIONS:\n- How does the run-time of the tested methods compare in practice?\n\n*** Rebuttal Acknowledgment ***\n\nI have read the author's rebuttal. The answer covers my questions well and the explanation the authors provides logical justification of the potential shortcomings. Therefore, I am of the opinion that the paper is sound and I increased my rating accordingly. I do not go higher due to my very limited familiarity with the subfield.\n\nLIMITATIONS:\n- The discussion of limitation is not very thorough and it is mostly limited to a mention of a lack of understanding of the game type effect on the performance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a cyclic block-coordinate-descent-like method for two-player zero-sum extensive-form games (EFG). Such methods for EFG are  difficult due to non-separable nature of block structure of the problem. The decision problem for a player in a EFG can be formulated using Treeplex, for which regularizing functions can be constructed through the framework of dilated regularizers. These dilated regularizing functions allow recursive prox computations. This paper utilizes this frame work to develop an extrapolated cyclic algorithm to perform pseudo-block updates.  They demonstrate O(1/T) convergence rate for two-player zero-sum Nash equilibrium  using this method, and provide a specific algorithmic implementation which shows that runtime of the proposed method  is independent of the number of blocks. \n\nExperimental evaluation on EFG benchmark games is performed using three different dilated regularizers -dilated entropy, dilated global entropy, and dilated $\\ell_2$ with different block construction strategies. Experiments demonstrate the benefit of using blocks over non-block based approach in  some games, and no significant diffference between different block construction methods in others.  The results show improved  performance over the state of the art first order method (Mirror-Prox). Further, they introduce a restarting heuristic which speeds up the proposed method as well as the baselines.\n\nSTRENGTHS:\nThe paper is generally well written, describing necessary background to help readers understand the paper.\n\nThe proposed algorithm seems novel, though I must admit I am not familiar with the literature.\n\nThe experimental evaluation seems convincing, the proposed method outperforms SOTA FOM\n\nWEAKNESSES:\nline 105-``As discussed before RCMs are not applicable to our setting''. It is not discussed anywhere in the paper why randomized coordinate methods are not applicable.\n\nThe paper could include a discussion in the related work about [i]  which develops primal-dual coordinate methods for solving bilinear saddle-point problems.\n\n[i] Carmon et al. Coordinate Methods for Matrix Games\n\nQUESTIONS:\n-\n\nLIMITATIONS:\nThe authors discuss open question of why restarting work with regularizers  other than $\\ell_2$. The authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed Extrapolated Cyclic Primal-Dual Algorithm (ECyclicPDA) is a solution technique for large-scale extensive-form games (EFG) that resembles first-order coordinate descent. To enable pseudo block-wise updates, it takes use of the recursive nature of the proximal update caused by dilated regularizers. A restarting heuristic for EFG solution is also presented by the authors, and it has the potential to significantly speed up both their cyclic technique as well as other current methods like mirror prox and (predictive) CFR+.\n\nSTRENGTHS:\nIn summary, the ECyclicPDA algorithm introduces a novel and effective solution technique for tackling large-scale sequential games, demonstrating superior performance compared to current state-of-the-art techniques in some scenarios. The paper examines prior research on coordinate descent methods and first-order techniques for solving EFGs, highlighting the originality and benefits of their proposed ECyclicPDA approach.\n\nWEAKNESSES:\nThe execution times and computational demands of the ECyclicPDA method in comparison to other cutting-edge approaches are not thoroughly analyzed in the paper. It is essential to evaluate such a comparison in order to ascertain the proposed algorithm's applicability and practical effectiveness. Although it is mentioned that the runtime of ECyclicPDA is independent of the number of blocks, a comprehensive evaluation is still lacking. Additionally, the paper overlooks competitive alternatives like DCFR, despite making references to these methods.\n\nQUESTIONS:\n1) The authors claim, \"For the first time, a first-order method has surpassed CFR+ in performance on non-trivial EFGs.\" However, it seems that in the paper titled \"Equivalence Analysis between Counterfactual Regret Minimization and Online Mirror Descent\" by Liu et al., the authors also demonstrate instances where first-order OMD methods can outperform CFR+.\n\n2) How strong is the assumption that the strongly convex functions are nice, see page 5, paragraph  \"Dilated Regularizers\"?\n\n3) In the appendix, the last three figures share the same caption. Do these plots refer to linear, quadratic, and uniform averaging?\n\nLIMITATIONS:\nDuring the experiments, ECyclicPDA demonstrated superior performance compared to CFR+ solely in the Battleship game. This stands as the only instance where the results were competitive with PCFR+, which still remains the overall best-performing strategy. However, it is not entirely clear from the overall text why ECyclicPDA outperformed CFR+ specifically in the Battleship game.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}