Last-iterate Convergence in Extensive-Form Games

1 Introduction . Extensive-form games ( EFGs ) are an important class of games in game theory and artificial intelligence which can model imperfect information and sequential interactions . EFGs are typically solved by finding or approximating a Nash equilibrium . Regret-minimization algorithms are among the most popular approaches to approximate Nash equilibria . The motivation comes from a classical result which says that in two-player zero-sum games , when both players use no-regret algorithms , the average strategy converges to Nash equilibrium [ Freund and Schapire , 1999 , Hart and Mas-Colell , 2000 , Zinkevich et al. , 2007 ] . Counterfactual Regret Minimization ( CFR ) [ Zinkevich et al. , 2007 ] and it variants such as CFR+ [ Tammelin , 2014 ] are based on this motivation . However , due to their ergodic convergence guarantee , theoretical convergence rates of regretminimization algorithms are typically limited to O ( 1/ √ T ) or O ( 1/T ) for T rounds , and this is also the case in practice [ Brown and Sandholm , 2019a , Burch et al. , 2019 ] . In contrast , it is known that linear convergence rates are achievable for certain other first-order algorithms [ Tseng , 1995 , Gilpin et al. , 2008 ] . Additionally , the averaging procedure can create complications . It not only increases the computational and memory overhead [ Bowling et al. , 2015 ] , but also makes things difficult when incorporating neural networks in the solution process , where averaging is usually not possible . Indeed , to address this issue , Brown et al . [ 2019 ] create a separate neural network to approximate the average strategy in their Deep CFR model . Therefore , a natural idea is to design regret-minimization algorithms whose last strategy converges ( we call this last-iterate convergence ) , ideally at a faster rate than the average iterate . Unfortunately , many regret minimization algorithms such as regret matching , regret matching+ , and hedge , are known not 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to satisfy this property empirically and theoretically even for normal-form games . Although Bowling et al . [ 2015 ] find that in Heads-Up Limit Hold ’ em poker the last strategy of CFR+ is better than the average strategy , and Farina et al . [ 2019b ] observe in some experiments the last-iterate of optimistic OMD and FTRL converge fast , a theoretical understanding of this phenomenon is still absent for EFGs . In this work , inspired by recent results on last-iterate convergence in normal-form games [ Wei et al. , 2021 ] , we greatly extend the theoretical understanding of last-iterate convergence of regretminimization algorithms in two-player zero-sum extensive-form games with perfect recall , and open up many interesting directions both in theory and practice . First , we show that any optimistic online mirror-descent algorithm instantiated with a strongly convex regularizer that is continuously differentiable on the EFG strategy space provably enjoys last-iterate convergence , while CFR with either regret matching or regret matching+ fails to converge . Moreover , for some of the optimistic algorithms , we further show explicit convergence rates . In particular , we prove that optimistic mirror descent instantiated with the 1-strongly-convex dilated entropy regularizer [ Kroer et al. , 2020 ] , which we refer to as Dilated Optimistic Multiplicative Weights Update ( DOMWU ) , has a linear convergence rate under the assumption that there is a unique Nash equilibrium ; we note that this assumption was also made by Daskalakis and Panageas [ 2019 ] , Wei et al . [ 2021 ] in order to achieve similar results for normal-form games . 2 Related Work . Extensive-form Games Here we focus on work related to two-player zero-sum perfect-recall games . Although there are many game-solving techniques such as abstraction [ Kroer and Sandholm , 2014 , Ganzfried and Sandholm , 2014 , Brown et al. , 2015 ] , endgame solving [ Burch et al. , 2014 , Ganzfried and Sandholm , 2015 ] , and subgame solving [ Moravcik et al. , 2016 , Brown and Sandholm , 2019b ] , these methods all rely on scalable methods for computing approximate Nash equilibria . There are several classes of algorithms for computing approximate Nash equilibria , such as double-oracle methods [ McMahan et al. , 2003 ] , fictitious play [ Brown , 1951 , Heinrich and Silver , 2016 ] , first-order methods [ Hoda et al. , 2010 , Kroer et al. , 2020 ] , and CFR methods [ Zinkevich et al. , 2007 , Lanctot et al. , 2009 , Tammelin , 2014 ] . Notably , variants of the CFR approach have achieved significant success in poker games [ Bowling et al. , 2015 , Moravčík et al. , 2017 , Brown and Sandholm , 2018 ] . Underlying the first-order and CFR approaches is the sequence-form representation [ von Stengel , 1996 ] , which allows the problem to be represented as a bilinear saddle-point problem . This leads to algorithms based on smoothing techniques and other first-order methods [ Gilpin et al. , 2008 , Kroer et al. , 2017 , Gao et al. , 2021 ] , and enables CFR via the theorem connecting no-regret guarantees to Nash equilibrium . Online Convex Optimization and Optimistic Regret Minimization Online convex optimization [ Zinkevich , 2003 ] is a framework for repeated decision making where the goal is to minimize regret . When applied to repeated two-player zero-sum games , it is known that the average strategy converges to Nash Equilibria at the rate of O ( 1/ √ T ) when both players apply regret-minimization algorithms whose regret grows on the order of O ( √ T ) [ Freund and Schapire , 1999 , Hart and Mas-Colell , 2000 , Zinkevich et al. , 2007 ] . Moreover , when the players use optimistic regret-minimization algorithms , the convergence rate is improved to O ( 1/T ) [ Rakhlin and Sridharan , 2013 , Syrgkanis et al. , 2015 ] . Recent works have applied optimism ideas to EFGs , such as optimistic algorithms with dilated regularizers [ Kroer et al. , 2020 , Farina et al. , 2019b ] , CFR-like local optimistic algorithms [ Farina et al. , 2019a ] , and optimistic CFR algorithms [ Burch , 2018 , Brown and Sandholm , 2019a , Farina et al. , 2021a ] . However , the theoretical results in all these existing papers consider the average strategy , while we are the first to consider last-iterate convergence in EFGs . Last-iterate Convergence in Saddle-point Optimization As mentioned previously , two-player zero-sum games can be formulated as saddle-point optimization problems . Saddle-point problems have recently gained a lot of attention due to their applications in machine learning , for example in generative adversarial networks [ Goodfellow et al. , 2014 ] . Basic algorithms , including gradient descent ascent and multiplicative weights update , diverge even in simple instances [ Mertikopoulos et al. , 2018 , Bailey and Piliouras , 2018 ] . In contrast , their optimistic versions , optimistic gradient descent ascent ( OGDA ) [ Daskalakis et al. , 2018 , Mertikopoulos et al. , 2019 , Wei et al. , 2021 ] and optimistic multiplicative weights update ( OMWU ) [ Daskalakis and Panageas , 2019 , Lei et al. , 2021 , Wei et al. , 2021 ] have been shown to enjoy attractive last-iterate convergence guarantees . However , almost none of these results apply to the case of EFGs : Wei et al . [ 2021 ] show a result that implies linear convergence of vanilla OGDA in EFGs ( see Corollary 5 ) , but no results are known for vanilla OMWU or more importantly for algorithms instantiated with dilated regularizers which lead to fast iterate updates in EFGs . In this work we extend the existing results on normal-form games to EFGs , including the practically-important dilated regularizers . 3 Problem Setup . We start with some basic notation . For a vector z , we use zi to denote its i-th coordinate and ‖z‖p to denote its p-norm ( with ‖z‖ being a shorthand for ‖z‖2 ) . For a convex function ψ , the associated Bregman divergence is define as Dψ ( u , v ) = ψ ( u ) − ψ ( v ) − 〈∇ψ ( v ) , u − v〉 , and ψ is called κ-strongly convex with respect to the p-norm if Dψ ( u , v ) ≥ κ2 ‖u − v‖ 2 p holds for all u and v in the domain . The Kullback-Leibler divergence , which is the Bregman divergence with respect to the entropy function , is denoted by KL ( · , · ) . Finally , we use ∆P to denote the ( P − 1 ) -dimensional simplex and [ N ] to denote the set { 1 , 2 . . . , N } for some positive integer N . Extensive-form Games as Bilinear Saddle-point Optimization We consider the problem of finding a Nash equilibrium of a two-player zero-sum extensive-form game ( EFG ) with perfect recall . Instead of formally introducing the definition of an EFG ( see Appendix A for an example ) , for the purpose of this work , it suffices to consider an equivalent formulation , which casts the problem as a simple bilinear saddle-point optimization [ von Stengel , 1996 ] : min x∈X max y∈Y x > Gy = max y∈Y min x∈X x > Gy , ( 1 ) whereG ∈ [ −1 , +1 ] M×N is a known matrix , and X ⊂ RM and Y ⊂ RN are two polytopes called treeplexes ( to be defined soon ) . The set of Nash equilibria is then defined as Z∗ = X ∗ × Y∗ , where X ∗ = argminx∈X maxy∈Y x > Gy and Y∗ = argmaxy∈Y minx∈X x > Gy . Our goal is to find a point z ∈ Z = X × Y that is close to the set of Nash equilibria Z∗ , and we use the Bregman divergence ( of some function ψ ) between z and the closest point in Z∗ to measure the closeness , that is , minz∗∈Z∗ Dψ ( z∗ , z ) . For notational convenience , we let P = M +N and F ( z ) = ( Gy , −G > x ) for any z = ( x , y ) ∈ Z ⊂ RP . Without loss of generality , we assume ‖F ( z ) ‖∞ ≤ 1 for all z ∈ Z ( which can always be ensured by normalizing the entries ofG accordingly ) . Treeplexes The structure of the EFG is implicitly captured by the treeplexes X and Y , which are generalizations of simplexes that capture the sequential structure of an EFG . The formal definition is as follows . ( In Appendix A , we provide more details on the connection between treeplexes and the structure of the EFG , as well as concrete examples of treeplexes for better illustrations . ) Definition 1 ( Hoda et al . [ 2010 ] ) . A treeplex is recursively constructed via the following three operations : 1 . Every probability simplex is a treeplex . 2 . Given treeplexes Z1 , . . . , ZK , the Cartesian product Z1 × · · · × ZK is a treeplex . 3 . ( Branching ) Given treeplexes Z1 ⊂ RM and Z2 ⊂ RN , and any i ∈ [ M ] , Z1 i Z2 = { ( u , ui · v ) ∈ RM+N : u ∈ Z1 , v ∈ Z2 } is a treeplex . By definition , a treeplex is a tree-like structure built with simplexes , which intuitively represents the tree-like decision space of a single player , and an element in the treeplex represents a strategy for the player . LetHZ denote the collection of all the simplexes in treeplex Z , which following Definition 1 can be recursively defined as : HZ = { Z } if Z is a simplex ; HZ = ⋃K k=1HZi if Z is a Cartesian product Z1×· · ·×ZK ; andHZ = HZ1 ∪HZ2 if Z = Z1 i Z2 . In EFG terminology , HX andHY are the collections of information sets for player x and player y respectively , which are the decision points for the players , at which they select an action within the simplex . For any h ∈ HZ , we let Ωh denote the set of indices belonging to h , and for any z ∈ Z , we let zh be the slice of z whose indices are in Ωh . For each index i , we also let h ( i ) be the simplex i falls into , that is , i ∈ Ωh ( i ) . In Definition 1 , the last branching operation naturally introduces the concept of a parent variable for each h ∈ HZ , which can again be recursively defined as : if Z is a simplex , then it has no parent ; if Z is a Cartesian product Z1 × · · · × ZK , then the parent of h ∈ HZ is the same as the parent of h in the treeplex Zk that h belongs to ( that is , h ∈ HZk ) ; finally , if Z = { ( u , ui · v ) : u ∈ Z1 , v ∈ Z2 } , then for all h ∈ HZ2 without a parent , their parent in Z is ui , and for all other h , their parents remain the same as in Z1 or Z2 . We denote by σ ( h ) the index of the parent variable of h , and let it be 0 if h has no parent . For convenience , we let z0 = 1 for all z ∈ Z ( so that zσ ( h ) is always well-defined ) . Also defineHi = { h ∈ HZ : σ ( h ) = i } to be the collection of simplexes whose parent index is i . Similarly , for an index i , its parent index is defined as pi = σ ( h ( i ) ) , and i is called a terminal index if it is not a parent index ( that is , i 6= pj for all j ) . Finally , for an element z ∈ Z and an index i , we define qi = zi/zpi . In EFG terminology , qi specifies the probability of selecting action i in the information set h ( i ) according to strategy z .

Summary: This paper studies learning algorithms in zero-sum extensive form games. While there has been a huge amount of interest in recent years in understanding the convergence of learning algorithms to Nash equilibria in continuous or normal form  zero-sum games, this has not extended as much to extensive form games. This paper makes use of prior works that allows such games to be formulated as a bilinear zero-sum games over action sets known as treeplexes that capture the sequential nature of the actions in the game. The authors then analyze the family of Optimistic Online Mirror Descent (OOMD) algorithms in such games due to their strong last-iterate guarantees in normal form games. In particular they first show that counterfactual regret minimization can diverge in extensive form games (using rock-paper-scissors as an example) before showing that OOMD algorithms with strongly convex regularizers enjoy last iterate convergence. The authors again invoke prior works to give convergence rates for vanilla GDA, give new convergence rates for vanilla optimistic multiplicative weights, and then provide a new analysis of the last-iterate convergence for a previously proposed algorithm known as Dilated Optimistic Multiplicative Weight Update which is specialized to the structure of a treeplex (and therefore actually implementable).