The Evolution of Uncertainty of Learning in Games

1 INTRODUCTION . A primary goal of ML research is to understand the behaviors of learning algorithms in various settings . One standard approach is from each initial condition , we determine whether a learning algorithm converges to a local optimum or stable state . Yet , in the context of online learning in games , and more generally in distributed multi-agent learning , it is natural to consider the evolution of a probability distribution over initial conditions instead . In these settings , each agent forms an initial belief on her own , and she typically does not reveal her belief to other agents or external modelers/analysts . For a modeler to understand the behaviors of the system while being uncertain of the agents ’ beliefs , one natural approach is to infer how the initialization distributes by using data of observations from the past . The modeler then use this inference to predict the likelihoods of different outcomes in the future , either by simulation or by analysis . Also , random initialization can happen due to random external signals , e.g . weather , as well as noisy measurements . For readers who want to know more intuitions and mathematical aspects of such models , see Appendix A . In such cases , it is critical to understand whether the initial probability distribution evolves toward stability , or if its uncertainty gets amplified . Such analysis provides insight into the learning system ’ s robustness against random initialization and environment perturbations . If a coordinator of the system desires stability but the analysis shows the uncertainty gets amplified , she ought to coordinate with the agents to make changes , e.g . encourage them to use other online learning algorithms . To analyze how uncertainty evolves , we need a measure of it . A natural choice is entropy . In his seminal work in 1948 , Claude Shannon formulated an axiomatic foundation of information theory and introduced the seminal definition of Shannon entropy ( SE ) : given a discrete random variable with possible outcomes x1 , . . . , xn which occur with probability p1 , . . . , pn , its SE is − ∑n i=1 pi log pi = E [ log ( 1/pi ) ] . Entropy is the canonical measure of uncertainty : when pi = 1 for some i , the distribution is considered certain and its entropy is zero ; uniform distribution is viewed as the most uncertain , and it attains the maximum possible entropy of log n. For continuous random variables with probability density function g , Shannon proposed an extension of SE called the differential entropy ( DE ) , defined as E [ log ( 1/g ( x ) ) ] = − ∫ X g ( x ) log g ( x ) dx , where X is the support set of the random variable . We will analyze how DE evolves in multi-agent learning . Our Contributions . In our model , a learning-in-game system starts with a distribution over a set of initial conditions in the cumulative payoff space , a coordinate system which is inherent in the implementations of many online learning algorithms . The initial distribution evolves over time depending on the combination of agents ’ learning algorithms and the underlying game . We focus on two popular learning algorithms , Multiplicative Weights Update ( MWU ) and its optimistic variant ( OMWU ) .1 The game settings include the standard two-player matrix games , and one-population games which are fundamental in biological evolution models . We show that the DE of a broad range of learning-in-game systems increases linearly with time ( up to a certain limit ) , formalizing their increased unpredictability . Such systems include MWU in zero-sum games , OMWU in coordination games , and MWU in certain one-population games which reward more to intra-species play than to inter-species play . Our results apply to any smooth initial distribution with bounded support . At this point one may naturally wonder : What level of uncertainty does a linear increase of DE indicate ? What are its implications ? To answer these questions , it is best to compare ourselves against other simple benchmarks . Consider the following simple learning system : in each round , the payoffs for each action are generated from a uniform distribution over [ −1 , 1 ] . In the cumulative payoff space , the distribution at time T converges to a multivariate Gaussian distribution with variance Θ ( T ) , and hence the entropy grows at a rate of O ( log T ) , much slower than linear growth . In Section 4.1 , we present an implication of linear DE growth in learning-in-game systems . Briefly speaking , the DE can not increase substantially for an indefinite amount of time in such systems . Thus , the distribution after a sufficiently long time must concentrate in a region that yields slower DE growth or decline ; such region does not cover any point in the cumulative payoff space that corresponds to an interior Nash equilibrium . We will refer to this phenomenon by “ the grand escape ” . For the implications in information theory , we refer readers to Chapter 8 of Cover & Thomas ( 2006 ) . The central tool in analyzing the changes of DE is the Jacobian matrix of our multi-agent dynamical system . This is also the key notion in volume analysis , which was used in a recent series of papers to demonstrate Lyapunov chaos in learning-in-game systems . ( Cheung & Piliouras ( 2019 ; 2020 ) ; Cheung & Tao ( 2020 ) ) By showing that the determinant of the Jacobian matrix is strictly above 1 in a large domain of the cumulative payoff space , they showed that the volume ( Lebesgue measure ) of a small set of initial conditions will blow up exponentially to become a set of large diameter , thus demonstrating a general Lyapunov chaos phenomenon in learning-in-game . Indeed , the same property of Jacobian matrix guarantees linear increase of DE . Additionally , we present the first Jacobian analysis of online learning in one-population games . Finally , our DE analysis is robust against small perturbations . Consequently , our results extend to the setting where the game in each round can be different , as long as the payoffs are perturbations of each other . Such settings capture many games in the real world , where we know the “ reference values ” of the payoffs , but the accurate payoffs in each round differ slightly due to unknown external ( random or adversarial ) factors . Our model and analysis can be viewed as an a strengthening of volume analysis . Volume analysis focuses on whether it is possible to reach a state or not , whereas in our model we are also concerned with how probable/likely it is to reach a state . More explicitly , a state can be reached but only with a tiny chance . When studying chaos and volume , such states matter , but less so when studying uncertainty as their contributions to entropy will be low . To illuminate this , we simulate MWU ( with step-size 0.01 ) in Matching Pennies game , and present a few plots in Figure 2 . The top-left plot shows the initial set , which is a small square around the unique Nash equilibrium . The top-right plot shows the range of possibility after 40000 steps ( a similar figure was first presented by Cheung & Piliouras ( 2019 ) ) . In our model , we assume the initial distribution is uniform over the small square , and plot the heat-map of probability densities after 40000 steps ( bottom-left ) . We observe that the states in the boundary of the vortex are more probable to occur , while the densities around the Nash equilibrium decline . The bottom-right plot shows the densities that generate the heat-map . Further Related Work . Learning models that explicitly study the effects random initialization have received relatively little attention in game theory where static equilibrium concepts are typically 1As we shall point out in Section 3 , all our results about MWU also extend easily to the broad family of Follow-the-Regularized-Leader ( FTRL ) algorithms , which is a generalization of MWU . For clarity of our presentation , we choose to focus on MWU in this paper . the preferred solution . Lahkar & Seymour ( 2013 ) studies online learning under random initialization in population dynamics , where one population game is a common payoff model . In population dynamics , there exists a large population of agents who have different initial choices of mixed strategies , modeled as a distribution . The dynamics proceed by randomly pairing up agents in each round to play a game . Lahkar & Seymour ( 2014 ) extends this framework to other learning dynamics to describe the evolution of a distribution of mixed strategies in a population playing variants of RockPaper-Scissors games establishing local convergence to the interior equilibrium . Recently , further learning models inspired by Q-learning dynamics have been defined ( Hu et al. , 2019 ) . Very little is formally known about the behavior of such models particularly under instability . For zero-sum games with an interior Nash equilibrium , “ the grand escape ” phenomenon follows from Bailey & Piliouras ( 2018 ) ; Cheung ( 2018 ) , who showed that MWU diverges away from Nash . Our results are more general , since their analysis only works with those zero-sum game dynamics , while our technique applies to many other game dynamics as well . Such instability results in zerosum games are undesirable for ML applications such as GANs thus a lot of effort has been invested in developing algorithms that achieve convergence Daskalakis et al . ( 2017 ) ; Mertikopoulos et al . ( 2019 ) ; Daskalakis & Panageas ( 2018 ) ; Gidel et al . ( 2019 ) ; Mokhtari et al . ( 2020 ) ; Perolat et al . ( 2021 ) . Our research direction here is in some sense orthogonal . We aim to characterize in a more detailed fashion the behavior of learning dynamics despite their instability . The instability of MWU , FTRL and other optimization-driven dynamics in games has attracted a lot of attention in recent years . One formal technique to establish the unpredictability of MWU and other dynamics is based on proving Li-Yorke chaos , which has been established robustly in different families of routing/potential/Cournot games ( Palaiopanos et al . ( 2017 ) ; Chotibut et al . ( 2020 ) ; Bielawski et al . ( 2021 ) ; Cheung et al . ( 2021 ) ) . The techniques in those papers are independent from ours as they require the identification of one dimensional invariant manifolds , and Li-Yorke chaos implies the existence of periodic orbits , which is not possible in our systems . In a very recent series of works , Flokas et al . ( 2020 ) ; Giannou et al . ( 2021 ) established that all ( even partially ) mixed equilibria in all games are not asymptotically stable for any choice for FTRL dynamics both in discrete as well as in the idealized continuous-time limit . In fact , it is possible to construct simple matrix games such that the orbits of the resulting dynamics can be arbitrarily complex Andrade et al . ( 2021 ) . If , inspired by ML applications , we allow for more complex differentiable games it becomes even simpler to establish strong instability results for effectively any type of training algorithms ( Letcher ( 2020 ) ; Balduzzi et al . ( 2020 ) ) . The sweeping nature of these strong negative results showcases the practical interest in uncovering more information about the nature of unstable dynamics of games . Our proposed framework introduces a novel , quantitative methodology for their study . The notion of differential entropy ( DE ) has been used as uncertainty measure in many works across multiple disciplines . We note that while the extended definition of DE seems natural , it is not considered as the perfect generalization of SE since it misses some favorable properties , and therefore , as quoted from the popular information theory text of Cover & Thomas ( 2006 ) , “ there is need for some care in using the concept ” .2 Nevertheless , DE remains as an effective measure of uncertainty3 , which is commonly used in physics , economics , control theory , engineering , biology , medical research and beyond . For a general discussion , we refer readers to the text of Cover & Thomas ( 2006 ) . For background information of evolutionary game theory , we recommend Hofbauer & Sigmund ( 1998 ) ; Sandholm ( 2010 ) ; Vega-Redondo ( 1996 ) .

Summary: This paper extends the existing line of research of the dynamics of multiplicative weights update and similar algorithms for games.  It shows that for zero-sum two-player games and for population games satisfying certain conditions, that the entropy increases linearly as long as the strategies are far from the distribution; this implies that the strategies concentrate near the boundary in the long-run, in a sense that the paper formalizes.  Compared to previous work, this analysis applies to games that are not zero-sum--specifically, population games that satisfy a certain condition on the payoffs.