Exploration-Exploitation in Multi-Agent Competition: Convergence with Bounded Rationality

1 Introduction . Zero-sum games and variants thereof are arguably amongst the most well studied settings in game theory . Indeed much attention has focused on the class of strictly competitive games [ 4 ] , i.e. , two player games such that when both players change their mixed strategies , then either there is no change in the expected payoffs , or one of the two expected payoffs increases and the other decreases.1 According to Aumann [ 4 ] , “ Strictly competitive games constitute one of the few areas in game theory , and indeed in the social sciences , where a fairly sharp , unique prediction is made . '' The unique prediction , of course , refers to the min-max solution and the resulting values guaranteed to both agents due to the classic work of von Neumann [ 53 ] . Unfortunately , when we move away from the safe haven of two-agent strictly competitive games , a lot of these regularities disappear . For example , in multi-agent variants of zero-sum and strictly competitive games , several critical aspects of the min-max theorem collapse [ 12 ] . Critically , Nash Equilibrium ( NE ) payoffs need not be unique . In fact , there can be continua of equilibria with the payoff range of different agents corresponding to positive measure sets . Furthermore , NE strategies need not be exchangeable ( i.e. , mixing-matching strategies from different Nash profiles does not lead to a Nash ) nor max-min . Thus , network competition is not only significantly harder , but poses qualitatively different questions than two-agent competition . Nevertheless , and despite the intense recent interest inspired by Artificial Intelligence ( AI ) and Machine Learning ( ML ) applications such as generative adversarial networks and actor-critic systems to understand learning dynamics in zero-sum games and even network variants thereof [ 19 , 22 , 27 ] , so far , there has been no systematic study of how agents should deal with uncertainty of the resulting 1In fact , as recent work has established these strictly competitive games are formally equivalent to weighted zero-sum games , i.e. , affine transformations of zero-sum games [ 1 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . payoffs in such games . In these settings , the use of purely optimization-driven , regret-minimizing algorithms is no longer equally attractive as in the two-player case . The multiplicity of equilibria and the lack of unique value give rise to a non-trivial problem of equilibrium selection and learning agents face the fundamental dilemma between exploration and exploitation [ 15 , 41 , 11 , 59 ] . These considerations drive our motivating question : Are there exploration-exploitation dynamics that provably converge in networks of strictly competitive games ? How do they behave in settings with multiple , payoff diverse Nash equilibria ? Model and Results . We study a well-known smooth variant of Q-learning [ 55 , 54 ] , with softmax or Boltzmann exploration , one of the most fundamental models of exploration-exploitation in multiagent systems ( MAS ) , termed Boltzmann or smooth Q-learning [ 51 , 48 ] . Informally ( see Section 3 for the rigorous definition ) , each agent k updates their choice distribution x = ( xi ) according to the rule ẋi/xi = ( ri − r̄ ) − Tk ( lnxi − ∑ j xj lnxj ) , where ri , r̄ denote agent k ’ s rewards from action i and average rewards , respectively , given all other agents ’ actions and Tk is agent k ’ s exploration rate . In our main result , we show convergence of Q-learning to Quantal Response Equilibria ( QRE ) , the prototypical extension of Nash equilibria for games with bounded rationality [ 36 ] , in multiagent/network generalizations of strictly competitive games [ 13 , 12 ] . As long as all exploration rates are positive , we prove via a global Lyapunov argument that the Q-learning dynamic converges pointwise to a unique QRE regardless of initial conditions and regardless of the number of the Nash equilibria of the original network game ( Theorem 4.1 ) . Related to the above , we demonstrate how exploration by all agents leads to equilibrium selection . Thus , this long-standing open problem ( [ 32 , 45 , 43 ] ) becomes tractable in practice due to the theoretical guarantees of fast convergence to QRE that we provide in Theorem 4.1 for this class of competitive games . In fact , Theorem 4.1 is in some sense tight , as there exist network competitive settings whose dynamics lead to limit cycles if not all of the agents are performing exploration ( see experiments and discussion in Section 5 ) . Other Related Works . The variant of Q-learning that we study has recently received a lot of attention due to its connection to evolutionary game theory [ 47 , 31 , 57 ] . It has been also extensively studied in the economics and reinforcement learning literature under various names , see e.g. , [ 2 , 46 ] and [ 28 , 39 ] . Recent works demonstrate that it is possible to show convergence of the Q-learning dynamics in multi-agent cooperative settings and to select highly desirable equilibria via complex bifurcation phenomena [ 34 ] . On the other hand , competitive multi-agent systems constitute one of the current frontiers in AI and ML research . Many recent works investigate the complex behavior of competitive game theoretic settings [ 44 , 9 , 8 , 7 , 37 ] , focusing on carefully designed convergent algorithms , e.g. , optimism [ 19 , 20 , 56 ] , extra-gradient methods [ 40 , 3 , 26 ] , regularization [ 43 ] , momentum adapted dynamics [ 23 ] , or symplectic integration schemes [ 6 ] . However , despite the theoretical progress and the impressive results in the empirical front [ 49 , 52 , 33 ] , the literature on equilibrium selection seems to have received little attention so far . Prior works have focused on equilibrium selection in cooperative AI or have theoretically studied competition in 2-agent zero-sum games in which equilibrium selection is irrelevant [ 17 , 18 ] . To our knowledge , we are the first paper to explicitly study both theoretically and experimentally multi-agent competitive settings without uniquely defined values . 2 Game-Theoretic Model . A polymatrix or separable network game , Γ = ( ( V , E ) , ( Sk , wk ) k∈V , ( Akl ) [ k , l ] ∈E ) , consists of a graph ( V , E ) , where V = { 1 , 2 , . . . , n } is the set of players ( or agents ) , and E is a set of pairs , [ k , l ] , of players k ̸= l ∈ V . Each player , k ∈ V , has a finite set of actions ( or strategies ) Sk with generic elements sk ∈ Sk ( depending on the context , sometimes we will also write i or j ∈ Sk ) . Players may also use mixed strategies ( or choice distributions ) xk = ( xki ) i∈Sk ∈ ∆k , where ∆k is the simplex in R|Sk| , i.e. , ∑ i∈Sk xki = 1 , and xki ≥ 0 , for any xk ∈ ∆k . The interior of ∆k is the set of all points xk ∈ ∆k with xki ∈ ( 0 , 1 ) for all i ∈ Sk . All points of ∆k that are not in the interior , are called boundary points . We will write x = ( x1 , x2 , . . . , xn ) or x = ( xk , x−k ) for a mixed strategy profile x ∈ ∆ : = ∏ k∈V ∆k , where x−k ∈ ∆−k = ∏ l ̸=k∈V ∆l is the vector of mixed strategies of all players l ∈ V other than k. Each edge [ k , l ] ∈ E defines a two-player game with payoff matrices Akl ∈ R|Sk|×|Sl| and Alk ∈ R|Sl|×|Sk| . The elements akl ( sk , sl ) of a matrix Akl denote the payoffs of player k when players k and l use pure actions sk ∈ Sk and sl ∈ Sl , respectively . Each player k ∈ V chooses a strategy ( mixed or pure ) and plays that strategy in all games [ k , l ] ∈ E. Thus , the payoff of player k at the pure strategy profile s = ( s1 , . . . , sn ) ∈ S : = ∏ k∈V Sk is uk ( s ) = ∑ [ k , l ] ∈E akl ( sk , sl ) . Similarly , the expected reward of player k in the mixed strategy profile x ∈ ∆ is uk ( x ) : = ∑ [ k , l ] ∈E x⊤k Aklxl = x ⊤ k ( ∑ [ k , l ] ∈E Aklxl ) . ( 1 ) It will be convenient to write rki ( x−k ) : = uk ( i , x−k ) = ∑ [ k , l ] ∈E { Aklxl } i ( where { v } i to denotes the i-th element of a vector v ) , for the reward of pure action i ∈ Sk of player k when all other players use the strategy profile x−k and rk ( x−k ) : = ( rki ( x−k ) ) i∈Sk for the resulting reward vector of all pure actions of agent k , respectively . Using this notation , the expected reward of player k ∈ V at the mixed strategy profile x = ( xk , x−k ) can be compactly expressed as uk ( x ) = x⊤k rk ( x−k ) . Weighted Zero-Sum Polymatrix Games . Γ is called a weighted or rescaled zero-sum polymatrix game [ 12 ] , if there exist positive constants w1 , w2 , . . . , wn > 0 , so that∑ k∈V wkuk ( x ) = 0 , for all x ∈ ∆ . ( 2 ) By summing over the edges in E ( instead of the players in V ) , we can equivalently express the weighted zero-sum property as∑ [ k , l ] ∈E [ wkx ⊤ k Aklxl + wlx ⊤ l Alkxk ] = 0 , for all x ∈ ∆ . ( 3 ) Nash Equilibrium . A strategy profile ( tuple of mixed strategies ) , p = ( pk ) k∈V ∈ ∆ , with pk = ( pki ) i∈Sk ∈ ∆k for each k ∈ V is a Nash equilibrium ( NE ) of Γ if uk ( p ) ≥ uk ( xk , p−k ) , for all xk ∈ ∆k , for all k ∈ V , ( NE ) i.e. , if there exist no profitable unilateral deviations . By linearity , it suffices to verify the condition in equation ( NE ) only for pure actions sk ∈ Sk instead of all xk ∈ ∆k . 3 Joint Learning Model : Q-Learning Dynamics . We next discuss how we can get to the Q-learning dynamics from Q-learning agents when there are multiple learners in the system . Our goal is to identify the dynamics in competitive systems in which multiple agents are playing a rescaled zero-sum polymatrix game repeatedly over time . Q-learning . Q-learning [ 55 , 54 ] is a value-iteration method for solving the optimal strategies in Markov Decision Processes ( MDPs ) . It can be used as a model where users learn about their optimal strategy when facing uncertainties . Specifically , at every time-point n ≥ 0 , each Q-learning agent , k ∈ V , keeps track of the past performance of each of their actions , i ∈ Sk , via a Q-value , denoted by Qik ( n ) . Qki ( n ) is also called the memory of agent k about the performance of action i ∈ Sk up to time step n ≥ 0 . After selecting action i at time n , the corresponding Q-value is updated according to the Q-learning update rule Qki ( n+ 1 ) = Qki ( n ) + αk [ rki ( x−k , n ) −Qki ( n ) ] , i ∈ Sk , ( 4 ) where αk ∈ [ 0 , 1 ] denotes the learning rate or memory decay of agent k [ 48 , 31 ] . Agent k ∈ V updates their actions ( choice distributions ) according to a Boltzmann-type distribution , with xki ( n ) = exp ( Qki ( n ) /Tk ) ∑ j∈Sk exp ( Qkj ( n ) /Tk ) , for each i ∈ Sk , ( 5 ) where Tk ∈ [ 0 , +∞ ) denotes agent k ’ s learning sensitivity or adaptation , i.e. , how much the choice distribution is affected by the past performance . We will refer to Tk as the exploration rate or temperature of player k [ 34 , 51 ] ( see also next paragraph for a discussion ) . Combining equations ( 4 ) and ( 5 ) , one obtains the recursive equation of player k ’ s mixed strategy ( or choice distribution ) xki ( n+ 1 ) = xki ( n ) exp ( ( Qki ( n+ 1 ) −Qki ( n ) ) /Tk ) ∑ j∈Sk xkj ( n ) exp ( ( Qkj ( n+ 1 ) −Qkj ( n ) ) /Tk ) , for each i ∈ Sk . In practice , agents perform large numbers of action updates ( updates of Q-values ) for each choice-distribution update . This motivates to consider a continuous time version of the learning process of each agent which results in the following update rules for both the memories Qki and the selection probabilities xki of each action i ∈ Sk Q̇ki = αk [ rki ( x−k ) −Qki ] , and ẋki = xki ( Q̇ki − ∑ j∈Sk Q̇kj ) /Tk , where we omitted the dependence on the continuous time , t ≥ 0 . Combining these two expressions with equation ( 5 ) under the assumptions that pairs of actions have constant relationship over time and that agents ’ choice distributions are independently distributed , we obtain the Q-Learning Dynamics ẋki = xki [ rki ( x−k ) − x⊤k rk ( x−k ) − Tk ( ln ( xki ) − x⊤k ln ( xk ) ) ] , ( QLD ) for all players k ∈ V and all actions i ∈ Sk . Fixed Points of QLD . Using the convention x lnx : = 0 if x = 0 ( recall that limx→0+ x lnx = 0 ) , we can see from equation ( QLD ) that xki ( t ) = xki ( 0 ) for all t > 0 , for all i ∈ Sk such that xki ( 0 ) ∈ { 0 , 1 } . In other words , the boundary of ∆k is invariant for ( QLD ) , i.e. , if the dynamics start on the boundary , then they will remain there . This implies that all boundary points of the simplex , ∆k , are trivially fixed points of ( QLD ) . The interesting part concerns the fixed-points of the dynamics for interior starting points . In this case , equation ( QLD ) , implies that , whenever Tk > 0 , any resulting fixed point of ( QLD ) can only lie again in the interior of ∆k due to the entropy term ( ln of the choice probabilities ) . Clearly , the rate of change of xki must be equal to zero . Formally , this implies that a strategy profile p = ( pk ) k∈V with pk = ( pki ) i∈Sk for each k ∈ V is an interior fixed point of ( QLD ) if rki ( p−k ) = p ⊤ k rk ( p−k ) + Tk ( ln ( pki ) − p⊤k ln ( pk ) ) , for all k ∈ V. ( 6 ) Exploration Rates . Equation ( QLD ) suggests that higher values of Tk ’ s indicate higher exploration rates , whereas values close to 0 indicate a higher exploitation rates . Specifically , for Tk = 0 , the dynamics in ( QLD ) reduce to the well-known replicator dynamics , and agent k simply selects the action i ∈ Sk with the highest Q-value . On the other hand , as Tk → ∞ , player k essentially ignores the input ( rewards ) from the environment and maximizes the entropy of their current choice distribution ( which leads to uniform randomization over their available actions in Sk ) . From a behavioral perspective , this leads to an equivalent interpretation of the Tk ’ s as the degree of bounded rationality of the agent [ 25 , 24 , 21 ] . Finally , from an algorithmic perspective , Tk and the corresponding entropy term can be viewed as regularizers that prevent the learning dynamics from overfitting , i.e. , reaching the boundary or getting trapped in local optima [ 10 , 16 , 38 ] and [ 29 , 30 ] . Exponential Discounting . In continuous time , the Q-learning updates of equation ( 4 ) can be intuitively interpreted as exponential discounting of payoffs , with payoffs further back in the past being less important . In other words , an agent ’ s Q-value for an action shows how beneficial is the action for that agent when payoffs are discounted . Proposition 3.1 ( Q-Value Updates and Exponential Discounting ) . For an agent k ∈ V , consider the Q-learning updates in equation ( 4 ) and assume that Qi ( 0 ) = 0 for all i ∈ Sk . Then , in continuous time , the Q-value updates are given by Qik ( t ) = α ∫ t 0 e−αksri ( t− s ) ds = αe−αkt ∫ t 0 eαksri ( s ) ds , for any t > 0 . This interpretation also showcases the connection between ( QLD ) and another commonly studied dynamic in behavioral game theory , Experienced Weighted Attraction ( EWA ) [ 14 , 21 ] . Along with the proof of Proposition 3.1 , this is explained in more detail in Appendix A .

Summary: This paper investigates the convergence of a smooth variant of Q-learning dynamics in multiplayer network games called weighted zero-sum polymatrix games. The authors suggest interpreting the exploration rate of Q-learning as the inverse of the lambda parameter of the quantal response model, determining the degree of the agent’s rationality. The main result of this work claims that in weighted zero-sum polymatrix games, the Q-learning dynamics will always converge to the unique quantal response equilibrium associated with strictly positive exploration rates exponentially fast, irrespective of the initial profile of strictly mixed strategies (i.e., no agent adopts a pure strategy). The manuscript is concluded by empirical analysis of convergence of Q-learning in two-player and multiplayer weighted zero-sum games. The results show that with strictly positive exploration rates, the dynamic will indeed converge to the associated quantal response equilibrium and approximate Nash equilibrium as exploration rates approach zero. In the case of the two-player setting, the authors observe the quantal response equilibrium is reached even when only one of the players explore (and the dynamic is cyclic when no one explores), which is turned into a formal proposition in appendix C for two-player games with two actions per each player. For multiplayer games, exploration by only a subset of players will not guarantee to reach equilibrium.