On the Rate of Convergence of Regularized Learning in Games: From Bandits and Uncertainty to Optimism and Beyond

In this paper , we examine the convergence rate of a wide range of regularized methods for learning in games . To that end , we propose a unified algorithmic template that we call “ follow the generalized leader ” ( FTGL ) , and which includes as special cases the canonical “ follow the regularized leader ” algorithm , its optimistic variants , extra-gradient schemes , and many others . The proposed framework is also sufficiently flexible to account for several different feedback models – from full information to bandit feedback . In this general setting , we show that FTGL algorithms converge locally to strict Nash equilibria at a rate which does not depend on the level of uncertainty faced by the players , but only on the geometry of the regularizer near the equilibrium . In particular , we show that algorithms based on entropic regularization – like the exponential weights algorithm – enjoy a linear convergence rate , while Euclidean projection methods converge to equilibrium in a finite number of iterations , even with bandit feedback . 1 Introduction . In the presence of uncertainty , the players of a game may not have full knowledge of its structure , “ or the ability and inclination to go through any complex reasoning process to calculate an equilibrium . But the participants are still supposed to adapt by accumulating empirical information on the relative advantages of the various pure strategies at their disposal ” . This aphorism – originally due to Nash [ 34 , p. 21 ] – constitutes the driving principle of game-theoretic learning , and highlights one of the field ’ s most central questions : Does learning with empirical observations lead to a Nash equilibrium ? And , if so , at what rate ? These questions have been at the forefront of game-theoretic research ever since the early days of the field , and they have recently received renewed attention via their connection to multi-agent reinforcement learning [ 43 ] , generative adversarial networks [ 18 ] , auctions [ 44 ] , and many other applications where online decision-making plays a major role . Still , any attempt to provide a positive answer to these questions must wrestle with a major roadblock : the well-known impossibility result of Hart and Mas-Colell [ 19 ] shows that there are no uncoupled dynamics that converge to Nash equilibrium in all games , thus shattering any hope of obtaining a universal convergence result . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , . In view of the above , contemporary research on game-theoretic learning has focused on relaxing the feedback requirements of the players ’ learning processes , and understanding the stability – and instability – properties of different kinds of equilibria under popular learning algorithms . One property that stands out in this regard is the so-called “ folk theorem ” of evolutionary game theory [ 20 ] , which can be stated as follows : Under the replicator dynamics – the continuous-time limit of the multiplicative / exponential weights ( EW ) algorithm [ 2 , 29 , 45 ] – a Nash equilibrium is stable and attracting if and only if it is strict ( i.e. , if and only if each player has a unique best response ) . The replicator dynamics are the most widely studied model for evolution in population games , so the above equivalence essentially delineates what is and what isn ’ t achievable in an evolutionary setting . In the context of online learning ( our paper ’ s main focus ) , a similar equivalence was obtained only recently [ 11 , 15 , 30 ] , but it extends to the entire family of “ follow the regularized leader ” ( FTRL ) dynamics [ 41 , 42 ] , in both continuous [ 11 , 30 ] and discrete time [ 15 ] . In particular , [ 15 ] studied the convergence of discrete-time FTRL models in the presence of uncertainty , and proved a high-probability , stochastic version of this equivalence that holds for several different types of feedback ( full information , bandit , etc. ) . Thus , coupled with the prominence of FTRL in online and game-theoretic learning , strict Nash equilibria emerge as the only stable limit points of regularized learning under uncertainty . Our contributions . One important limitation of the above results is that they are qualitative in nature . Indeed , even though asymptotic stability guarantees that a learning process converges locally to a strict equilibrium , it provides no information about the speed of this convergence . In particular , especially for discrete-time models of regularized learning , asymptotic stability does not provide any guidance on how to tune the algorithm ’ s hyperparameters ( learning rate , mixing , etc . ) , and/or what to expect in terms of the number of iterations required to reach a neighborhood of a Nash equilibrium . Our paper aims to provide quantitative answers to these questions for a wide array of regularized learning methods in the presence of uncertainty and limited information . To do so , we first introduce a flexible algorithmic framework – dubbed “ follow the generalized leader ” ( FTGL ) – that incorporates a broad spectrum of action choice mechanisms and feedback models . In more detail ( and in analogy to FTRL ) , the FTGL template maintains a cumulative estimate for the payoff of each action available to the learner , and then selects a mixed strategy via a suitable “ regularized ” choice map . Specifically : 1 . In terms of regularization , the FTGL template includes as special cases the standard logit choice and Euclidean projection methods ( as well as all other standard regularizers used in practice ) . 2 . In terms of the information used to update the “ aggregate score ” of each pure strategy , FTGL includes “ vanilla ” FTRL , its optimistic variants [ 10 , 38–40 ] , extra-gradient and mirror-prox methods [ 24 , 25 , 35 ] , with either full , oracle-based , or bandit feedback . In this general context , our main result may be summarized as follows . First , we introduce a “ rate function ” q that depends only on the regularizer defining the learning process , and which captures the sensitivity of the induced choice map to external stimuli : for example , q ( G ) = exp ( G ) for entropic / logit choice models , whereas q ( G ) = [ G ] + for methods run with Euclidean projections . We then show that , with probability at least 1 − X , the algorithm ’ s local rate of convergence to a strict equilibrium G∗ is of the form ‖-= − G∗‖ ≤ q ( 3 − 2 ∑= B=1 WB ) , where W= is the method ’ s learning rate and 2 , 3 are constants with 2 > 0 . This result shows that the convergence speed of FTGL methods depends only on the choice of regularizer and learning rate : for example , EW methods run with a constant step size converge to an equilibrium at an exponential rate , whereas Euclidean regularization attains convergence in a finite number of iterations . From a regret-theoretic point of view , this is somewhat surprising because the regret guarantees of entropic FTRL ( the EW algorithm ) are far superior to those of FTRL with Euclidean regularization [ 5 , 41 ] . Equally surprising is the fact that the type of feedback employed does not affect the method ’ s rate of convergence : ceteris paribus , the base sequence of states generated by an FTGL method attains the same rate of convergence to strict Nash equilibria , whether run with full , partial , or bandit / payoff-based feedback . This comes into stark contrast with the corresponding rates of regret minimization , which depend crucially on the type of feedback received [ 6 , 27 ] ; in a certain , precise sense , this robustness in the face of uncertainty shows that regret minimization and convergence to Nash equilibrium are fundamdentally different questions . Related work . The convergence speed of methods based on the FTRL template – “ vanilla ” , optimistic , or otherwise – have been studied extensively in the context of monotone games and variational inequalities ; for a ( highly incomplete ) list of recent references , see [ 9 , 10 , 16 , 17 , 21 , 23 , 28 , 31–33 ] and references therein . In this branch of the literature , there are two distinct threads : results concerning the convergence of the “ time-average ” of the process [ 16 , 24 , 33 , 35 ] , and those focusing on the algorithm ’ s “ last-iterate ” [ 9 , 10 , 17 , 21 , 23 , 28 ] . In the latter case ( which is the one closest to our setting ) , the fastest achievable speed of convergence is exponential when the method is run with a finetuned constant step-size , perfect payoff gradient observations , and the operator defining the problem is strongly monotone and Lipschitz smooth . When run with stochastic gradients , the corresponding min-max optimal rate is O ( 1/ ) ) under the same assumptions ( zeroth-order rates are usually much worse ) . The apparent gulf between the rates of convergence obtained for monotone games and those obtained herein have to do with two crucial factors : first , we are studying finite games , which are typically not monotone ; second , we are examining the algorithm ’ s rate of convergence to strict equilibria , which are corner points of the problem ’ s domain . This means that the geometry of the problem around a strict equilibrium is fundamentally sharper than the geometry around a solution of a generic monotone variational inequality , a fact which in turn explains the qualitatively different nature of the rates we obtain . In the context of finite games , there have been several works examining the speed of convergence to the game ’ s set of coarse correlated equilibria ( CCE ) by leveraging the algorithm ’ s regret minimization properties , cf . [ 3 , 4 , 12 , 13 , 36 , 44 ] and references therein . However , in addition to examining the algorithm ’ s empirical average – as opposed to the induced day-to-day sequence of play – these results focus almost exclusively on CCE , which means that it is not possible to draw any conclusions about convergence to the game ’ s Nash set – qualitatively or quantitatively . To the best of our knowledge , the closest work to our own in the literature is the paper of Cohen et al . [ 8 ] who showed that the EXP3 algorithm with explicit exploration converges at a sub-geometric rate in potential games ; our analysis allows for a wider range of learning rates , so we are able to obtain faster convergence rates than Cohen et al . [ 8 ] . We are not aware of any other comparable results in the literature . 2 Preliminaries . Finite games . Throughout this work we consider # -players finite games in normal form . Formally , each player , indexed by 8 ∈ N = { 1 , . . . , # } , has a finite set of pure strategies U8 ∈ A8 = { 1 , . . . , 8 } , and a payoff function D8 : A→ ℝ , where A B ∏ 8 A8 is the space of all pure strategy profiles . For concision , we will denote such a game as a tuple Γ = Γ ( N , A , D ) . During play , players can also play mixed strategies , i.e. , probability distributions G8 ∈ X8 B Δ ( A8 ) over their pure strategies . In this case , we will write G8U8 for the probability that player 8 ∈ N selects U8 ∈ A8 under G8 , G = ( G1 , . . . , G # ) for the players ’ mixed strategy profile , and X B ∏ 8 X8 for the set thereof . Finally , when focusing on the mixed strategy of a particular player 8 ∈ N , we will use the shorthand ( G8 ; G−8 ) B ( G1 , . . . , G8 , . . . , G # ) – and , similarly , ( U8 ; U−8 ) for pure strategies . Now , the expected payoff of player 8 in a mixed strategy profile G ∈ X is given by D8 ( G ) ≡ D8 ( G8 ; G−8 ) = ∑ U1∈A1 · · · ∑ U # ∈A # D8 ( U1 , . . . , U # ) · G1 , U1 · · · G # , U # ( 1 ) where D8 ( U1 , . . . , U # ) is the payoff of player 8 in the action profile U = ( U1 , . . . , U # ) ∈ A . For posterity , we will also write E8U8 ( G ) = D8 ( U8 ; G−8 ) for the payoff that player 8 would have gotten by playing U8 ∈ A8 against the mixed strategy profile G−8 of all other players . In this way , the mixed payoff vector of the 8-th player can be seen as a vector field E8 : X → Y8 = ℝA8 which can be written in components as E8 ( G ) = ( E8U8 ( G ) ) U8 ∈A8 . ( 2 ) Again , we will write E ( G ) = ( E1 ( G ) , . . . , E # ( G ) ) for the ensemble of payoff vectors of all players and Y = ∏8 Y8 for the space of payoff vectors respectively . Finally , in a slight abuse of notation , we will identify U8 with the mixed strategy that assigns all probability to U8 , and we will write E8 ( U ) = ( D8 ( U8 ; U−8 ) ) U8 ∈A8 . for the corresponding pure payoff vector . Nash equilibrium . The most widely used solution concept in game theory is that of a Nash equilibrium i.e. , a ( possibly ) mixed strategy profile G∗ ∈ X that discourages unilateral deviations ; formally , G∗ ∈ X is said to be a Nash equilibrium of Γ if D8 ( G∗ ) ≥ D8 ( G8 ; G∗−8 ) for all G8 ∈ X8 and all 8 ∈ N . ( NE ) The set of pure strategies supported at the equilibrium component G∗ 8 ∈ X8 of each player will be denoted by supp ( G∗ 8 ) = { U8 ∈ A8 : G∗8U8 > 0 } . In turn , the size of the support of G ∗ leads to the following dichotomy : G∗ is called pure if supp ( G∗ 8 ) ≡ ∏8∈ # supp ( G∗8 ) is a singleton and mixed otherwise . Finally , we will also say that a Nash equilibrium G∗ is strict if ( NE ) holds as a strict inequality whenever G8 ≠ G∗8 ; obviously , strict equilibria are also pure , but the converse need not hold . Strict Nash equilibria play a key role in game theory because any unilateral deviation incurs a strict loss to the deviating player ; put differently , if G∗ is strict , every player has a unique best response . In addition , they are the only equilibria that remain invariant under small generic perturbations of the game [ 14 ] ; these robustness properties of strict equilibria will play a key role in the sequel .

Summary: This paper proposes an abstraction of the online Follow-the-(Blank)-Leader type of methods they call Follow-the-Generalized-Leader (FTGL) that not only incorporates various regularization functions but also different feedback models. The authors provide a general convergence analysis for the family of methods over the problem of finding *strict* Nash equilibria in *finite* games. The main insight of the work is to claim that the convergence rate in this problem only depends on the regularization function used and *not* on the different feedback models, e.g. full information feedback vs bandit feedback.