On the Rate of Convergence of Regularized Learning in Games: From Bandits and Uncertainty to Optimism and Beyond

In this paper , we examine the convergence rate of a wide range of regularized methods for learning in games . To that end , we propose a unified algorithmic template that we call “ follow the generalized leader ” ( FTGL ) , and which includes as special cases the canonical “ follow the regularized leader ” algorithm , its optimistic variants , extra-gradient schemes , and many others . The proposed framework is also sufficiently flexible to account for several different feedback models – from full information to bandit feedback . In this general setting , we show that FTGL algorithms converge locally to strict Nash equilibria at a rate which does not depend on the level of uncertainty faced by the players , but only on the geometry of the regularizer near the equilibrium . In particular , we show that algorithms based on entropic regularization – like the exponential weights algorithm – enjoy a linear convergence rate , while Euclidean projection methods converge to equilibrium in a finite number of iterations , even with bandit feedback . 1 Introduction . In the presence of uncertainty , the players of a game may not have full knowledge of its structure , “ or the ability and inclination to go through any complex reasoning process to calculate an equilibrium . But the participants are still supposed to adapt by accumulating empirical information on the relative advantages of the various pure strategies at their disposal ” . This aphorism – originally due to Nash [ 34 , p. 21 ] – constitutes the driving principle of game-theoretic learning , and highlights one of the field ’ s most central questions : Does learning with empirical observations lead to a Nash equilibrium ? And , if so , at what rate ? These questions have been at the forefront of game-theoretic research ever since the early days of the field , and they have recently received renewed attention via their connection to multi-agent reinforcement learning [ 43 ] , generative adversarial networks [ 18 ] , auctions [ 44 ] , and many other applications where online decision-making plays a major role . Still , any attempt to provide a positive answer to these questions must wrestle with a major roadblock : the well-known impossibility result of Hart and Mas-Colell [ 19 ] shows that there are no uncoupled dynamics that converge to Nash equilibrium in all games , thus shattering any hope of obtaining a universal convergence result . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , . In view of the above , contemporary research on game-theoretic learning has focused on relaxing the feedback requirements of the players ’ learning processes , and understanding the stability – and instability – properties of different kinds of equilibria under popular learning algorithms . One property that stands out in this regard is the so-called “ folk theorem ” of evolutionary game theory [ 20 ] , which can be stated as follows : Under the replicator dynamics – the continuous-time limit of the multiplicative / exponential weights ( EW ) algorithm [ 2 , 29 , 45 ] – a Nash equilibrium is stable and attracting if and only if it is strict ( i.e. , if and only if each player has a unique best response ) . The replicator dynamics are the most widely studied model for evolution in population games , so the above equivalence essentially delineates what is and what isn ’ t achievable in an evolutionary setting . In the context of online learning ( our paper ’ s main focus ) , a similar equivalence was obtained only recently [ 11 , 15 , 30 ] , but it extends to the entire family of “ follow the regularized leader ” ( FTRL ) dynamics [ 41 , 42 ] , in both continuous [ 11 , 30 ] and discrete time [ 15 ] . In particular , [ 15 ] studied the convergence of discrete-time FTRL models in the presence of uncertainty , and proved a high-probability , stochastic version of this equivalence that holds for several different types of feedback ( full information , bandit , etc. ) . Thus , coupled with the prominence of FTRL in online and game-theoretic learning , strict Nash equilibria emerge as the only stable limit points of regularized learning under uncertainty . Our contributions . One important limitation of the above results is that they are qualitative in nature . Indeed , even though asymptotic stability guarantees that a learning process converges locally to a strict equilibrium , it provides no information about the speed of this convergence . In particular , especially for discrete-time models of regularized learning , asymptotic stability does not provide any guidance on how to tune the algorithm ’ s hyperparameters ( learning rate , mixing , etc . ) , and/or what to expect in terms of the number of iterations required to reach a neighborhood of a Nash equilibrium . Our paper aims to provide quantitative answers to these questions for a wide array of regularized learning methods in the presence of uncertainty and limited information . To do so , we first introduce a flexible algorithmic framework – dubbed “ follow the generalized leader ” ( FTGL ) – that incorporates a broad spectrum of action choice mechanisms and feedback models . In more detail ( and in analogy to FTRL ) , the FTGL template maintains a cumulative estimate for the payoff of each action available to the learner , and then selects a mixed strategy via a suitable “ regularized ” choice map . Specifically : 1 . In terms of regularization , the FTGL template includes as special cases the standard logit choice and Euclidean projection methods ( as well as all other standard regularizers used in practice ) . 2 . In terms of the information used to update the “ aggregate score ” of each pure strategy , FTGL includes “ vanilla ” FTRL , its optimistic variants [ 10 , 38–40 ] , extra-gradient and mirror-prox methods [ 24 , 25 , 35 ] , with either full , oracle-based , or bandit feedback . In this general context , our main result may be summarized as follows . First , we introduce a “ rate function ” q that depends only on the regularizer defining the learning process , and which captures the sensitivity of the induced choice map to external stimuli : for example , q ( G ) = exp ( G ) for entropic / logit choice models , whereas q ( G ) = [ G ] + for methods run with Euclidean projections . We then show that , with probability at least 1 − X , the algorithm ’ s local rate of convergence to a strict equilibrium G∗ is of the form ‖-= − G∗‖ ≤ q ( 3 − 2 ∑= B=1 WB ) , where W= is the method ’ s learning rate and 2 , 3 are constants with 2 > 0 . This result shows that the convergence speed of FTGL methods depends only on the choice of regularizer and learning rate : for example , EW methods run with a constant step size converge to an equilibrium at an exponential rate , whereas Euclidean regularization attains convergence in a finite number of iterations . From a regret-theoretic point of view , this is somewhat surprising because the regret guarantees of entropic FTRL ( the EW algorithm ) are far superior to those of FTRL with Euclidean regularization [ 5 , 41 ] . Equally surprising is the fact that the type of feedback employed does not affect the method ’ s rate of convergence : ceteris paribus , the base sequence of states generated by an FTGL method attains the same rate of convergence to strict Nash equilibria , whether run with full , partial , or bandit / payoff-based feedback . This comes into stark contrast with the corresponding rates of regret minimization , which depend crucially on the type of feedback received [ 6 , 27 ] ; in a certain , precise sense , this robustness in the face of uncertainty shows that regret minimization and convergence to Nash equilibrium are fundamdentally different questions . Related work . The convergence speed of methods based on the FTRL template – “ vanilla ” , optimistic , or otherwise – have been studied extensively in the context of monotone games and variational inequalities ; for a ( highly incomplete ) list of recent references , see [ 9 , 10 , 16 , 17 , 21 , 23 , 28 , 31–33 ] and references therein . In this branch of the literature , there are two distinct threads : results concerning the convergence of the “ time-average ” of the process [ 16 , 24 , 33 , 35 ] , and those focusing on the algorithm ’ s “ last-iterate ” [ 9 , 10 , 17 , 21 , 23 , 28 ] . In the latter case ( which is the one closest to our setting ) , the fastest achievable speed of convergence is exponential when the method is run with a finetuned constant step-size , perfect payoff gradient observations , and the operator defining the problem is strongly monotone and Lipschitz smooth . When run with stochastic gradients , the corresponding min-max optimal rate is O ( 1/ ) ) under the same assumptions ( zeroth-order rates are usually much worse ) . The apparent gulf between the rates of convergence obtained for monotone games and those obtained herein have to do with two crucial factors : first , we are studying finite games , which are typically not monotone ; second , we are examining the algorithm ’ s rate of convergence to strict equilibria , which are corner points of the problem ’ s domain . This means that the geometry of the problem around a strict equilibrium is fundamentally sharper than the geometry around a solution of a generic monotone variational inequality , a fact which in turn explains the qualitatively different nature of the rates we obtain . In the context of finite games , there have been several works examining the speed of convergence to the game ’ s set of coarse correlated equilibria ( CCE ) by leveraging the algorithm ’ s regret minimization properties , cf . [ 3 , 4 , 12 , 13 , 36 , 44 ] and references therein . However , in addition to examining the algorithm ’ s empirical average – as opposed to the induced day-to-day sequence of play – these results focus almost exclusively on CCE , which means that it is not possible to draw any conclusions about convergence to the game ’ s Nash set – qualitatively or quantitatively . To the best of our knowledge , the closest work to our own in the literature is the paper of Cohen et al . [ 8 ] who showed that the EXP3 algorithm with explicit exploration converges at a sub-geometric rate in potential games ; our analysis allows for a wider range of learning rates , so we are able to obtain faster convergence rates than Cohen et al . [ 8 ] . We are not aware of any other comparable results in the literature . 2 Preliminaries . Finite games . Throughout this work we consider # -players finite games in normal form . Formally , each player , indexed by 8 ∈ N = { 1 , . . . , # } , has a finite set of pure strategies U8 ∈ A8 = { 1 , . . . , 8 } , and a payoff function D8 : A→ ℝ , where A B ∏ 8 A8 is the space of all pure strategy profiles . For concision , we will denote such a game as a tuple Γ = Γ ( N , A , D ) . During play , players can also play mixed strategies , i.e. , probability distributions G8 ∈ X8 B Δ ( A8 ) over their pure strategies . In this case , we will write G8U8 for the probability that player 8 ∈ N selects U8 ∈ A8 under G8 , G = ( G1 , . . . , G # ) for the players ’ mixed strategy profile , and X B ∏ 8 X8 for the set thereof . Finally , when focusing on the mixed strategy of a particular player 8 ∈ N , we will use the shorthand ( G8 ; G−8 ) B ( G1 , . . . , G8 , . . . , G # ) – and , similarly , ( U8 ; U−8 ) for pure strategies . Now , the expected payoff of player 8 in a mixed strategy profile G ∈ X is given by D8 ( G ) ≡ D8 ( G8 ; G−8 ) = ∑ U1∈A1 · · · ∑ U # ∈A # D8 ( U1 , . . . , U # ) · G1 , U1 · · · G # , U # ( 1 ) where D8 ( U1 , . . . , U # ) is the payoff of player 8 in the action profile U = ( U1 , . . . , U # ) ∈ A . For posterity , we will also write E8U8 ( G ) = D8 ( U8 ; G−8 ) for the payoff that player 8 would have gotten by playing U8 ∈ A8 against the mixed strategy profile G−8 of all other players . In this way , the mixed payoff vector of the 8-th player can be seen as a vector field E8 : X → Y8 = ℝA8 which can be written in components as E8 ( G ) = ( E8U8 ( G ) ) U8 ∈A8 . ( 2 ) Again , we will write E ( G ) = ( E1 ( G ) , . . . , E # ( G ) ) for the ensemble of payoff vectors of all players and Y = ∏8 Y8 for the space of payoff vectors respectively . Finally , in a slight abuse of notation , we will identify U8 with the mixed strategy that assigns all probability to U8 , and we will write E8 ( U ) = ( D8 ( U8 ; U−8 ) ) U8 ∈A8 . for the corresponding pure payoff vector . Nash equilibrium . The most widely used solution concept in game theory is that of a Nash equilibrium i.e. , a ( possibly ) mixed strategy profile G∗ ∈ X that discourages unilateral deviations ; formally , G∗ ∈ X is said to be a Nash equilibrium of Γ if D8 ( G∗ ) ≥ D8 ( G8 ; G∗−8 ) for all G8 ∈ X8 and all 8 ∈ N . ( NE ) The set of pure strategies supported at the equilibrium component G∗ 8 ∈ X8 of each player will be denoted by supp ( G∗ 8 ) = { U8 ∈ A8 : G∗8U8 > 0 } . In turn , the size of the support of G ∗ leads to the following dichotomy : G∗ is called pure if supp ( G∗ 8 ) ≡ ∏8∈ # supp ( G∗8 ) is a singleton and mixed otherwise . Finally , we will also say that a Nash equilibrium G∗ is strict if ( NE ) holds as a strict inequality whenever G8 ≠ G∗8 ; obviously , strict equilibria are also pure , but the converse need not hold . Strict Nash equilibria play a key role in game theory because any unilateral deviation incurs a strict loss to the deviating player ; put differently , if G∗ is strict , every player has a unique best response . In addition , they are the only equilibria that remain invariant under small generic perturbations of the game [ 14 ] ; these robustness properties of strict equilibria will play a key role in the sequel .

Summary: This paper focused on normal-form games and proposed an algorithm framework called "follow the generalized leader" (FTGL). The framework covers the well-known "follow the regularized leader" methods as well as many other variants. The authors established FTGL's local convergence to strict Nash equilibria and further demonstrated that MWU converged linearly, while Euclidean projection methods converge within finite steps.