The Evolution of Uncertainty of Learning in Games

1 INTRODUCTION . A primary goal of ML research is to understand the behaviors of learning algorithms in various settings . One standard approach is from each initial condition , we determine whether a learning algorithm converges to a local optimum or stable state . Yet , in the context of online learning in games , and more generally in distributed multi-agent learning , it is natural to consider the evolution of a probability distribution over initial conditions instead . In these settings , each agent forms an initial belief on her own , and she typically does not reveal her belief to other agents or external modelers/analysts . For a modeler to understand the behaviors of the system while being uncertain of the agents ’ beliefs , one natural approach is to infer how the initialization distributes by using data of observations from the past . The modeler then use this inference to predict the likelihoods of different outcomes in the future , either by simulation or by analysis . Also , random initialization can happen due to random external signals , e.g . weather , as well as noisy measurements . For readers who want to know more intuitions and mathematical aspects of such models , see Appendix A . In such cases , it is critical to understand whether the initial probability distribution evolves toward stability , or if its uncertainty gets amplified . Such analysis provides insight into the learning system ’ s robustness against random initialization and environment perturbations . If a coordinator of the system desires stability but the analysis shows the uncertainty gets amplified , she ought to coordinate with the agents to make changes , e.g . encourage them to use other online learning algorithms . To analyze how uncertainty evolves , we need a measure of it . A natural choice is entropy . In his seminal work in 1948 , Claude Shannon formulated an axiomatic foundation of information theory and introduced the seminal definition of Shannon entropy ( SE ) : given a discrete random variable with possible outcomes x1 , . . . , xn which occur with probability p1 , . . . , pn , its SE is − ∑n i=1 pi log pi = E [ log ( 1/pi ) ] . Entropy is the canonical measure of uncertainty : when pi = 1 for some i , the distribution is considered certain and its entropy is zero ; uniform distribution is viewed as the most uncertain , and it attains the maximum possible entropy of log n. For continuous random variables with probability density function g , Shannon proposed an extension of SE called the differential entropy ( DE ) , defined as E [ log ( 1/g ( x ) ) ] = − ∫ X g ( x ) log g ( x ) dx , where X is the support set of the random variable . We will analyze how DE evolves in multi-agent learning . Our Contributions . In our model , a learning-in-game system starts with a distribution over a set of initial conditions in the cumulative payoff space , a coordinate system which is inherent in the implementations of many online learning algorithms . The initial distribution evolves over time depending on the combination of agents ’ learning algorithms and the underlying game . We focus on two popular learning algorithms , Multiplicative Weights Update ( MWU ) and its optimistic variant ( OMWU ) .1 The game settings include the standard two-player matrix games , and one-population games which are fundamental in biological evolution models . We show that the DE of a broad range of learning-in-game systems increases linearly with time ( up to a certain limit ) , formalizing their increased unpredictability . Such systems include MWU in zero-sum games , OMWU in coordination games , and MWU in certain one-population games which reward more to intra-species play than to inter-species play . Our results apply to any smooth initial distribution with bounded support . At this point one may naturally wonder : What level of uncertainty does a linear increase of DE indicate ? What are its implications ? To answer these questions , it is best to compare ourselves against other simple benchmarks . Consider the following simple learning system : in each round , the payoffs for each action are generated from a uniform distribution over [ −1 , 1 ] . In the cumulative payoff space , the distribution at time T converges to a multivariate Gaussian distribution with variance Θ ( T ) , and hence the entropy grows at a rate of O ( log T ) , much slower than linear growth . In Section 4.1 , we present an implication of linear DE growth in learning-in-game systems . Briefly speaking , the DE can not increase substantially for an indefinite amount of time in such systems . Thus , the distribution after a sufficiently long time must concentrate in a region that yields slower DE growth or decline ; such region does not cover any point in the cumulative payoff space that corresponds to an interior Nash equilibrium . We will refer to this phenomenon by “ the grand escape ” . For the implications in information theory , we refer readers to Chapter 8 of Cover & Thomas ( 2006 ) . The central tool in analyzing the changes of DE is the Jacobian matrix of our multi-agent dynamical system . This is also the key notion in volume analysis , which was used in a recent series of papers to demonstrate Lyapunov chaos in learning-in-game systems . ( Cheung & Piliouras ( 2019 ; 2020 ) ; Cheung & Tao ( 2020 ) ) By showing that the determinant of the Jacobian matrix is strictly above 1 in a large domain of the cumulative payoff space , they showed that the volume ( Lebesgue measure ) of a small set of initial conditions will blow up exponentially to become a set of large diameter , thus demonstrating a general Lyapunov chaos phenomenon in learning-in-game . Indeed , the same property of Jacobian matrix guarantees linear increase of DE . Additionally , we present the first Jacobian analysis of online learning in one-population games . Finally , our DE analysis is robust against small perturbations . Consequently , our results extend to the setting where the game in each round can be different , as long as the payoffs are perturbations of each other . Such settings capture many games in the real world , where we know the “ reference values ” of the payoffs , but the accurate payoffs in each round differ slightly due to unknown external ( random or adversarial ) factors . Our model and analysis can be viewed as an a strengthening of volume analysis . Volume analysis focuses on whether it is possible to reach a state or not , whereas in our model we are also concerned with how probable/likely it is to reach a state . More explicitly , a state can be reached but only with a tiny chance . When studying chaos and volume , such states matter , but less so when studying uncertainty as their contributions to entropy will be low . To illuminate this , we simulate MWU ( with step-size 0.01 ) in Matching Pennies game , and present a few plots in Figure 2 . The top-left plot shows the initial set , which is a small square around the unique Nash equilibrium . The top-right plot shows the range of possibility after 40000 steps ( a similar figure was first presented by Cheung & Piliouras ( 2019 ) ) . In our model , we assume the initial distribution is uniform over the small square , and plot the heat-map of probability densities after 40000 steps ( bottom-left ) . We observe that the states in the boundary of the vortex are more probable to occur , while the densities around the Nash equilibrium decline . The bottom-right plot shows the densities that generate the heat-map . Further Related Work . Learning models that explicitly study the effects random initialization have received relatively little attention in game theory where static equilibrium concepts are typically 1As we shall point out in Section 3 , all our results about MWU also extend easily to the broad family of Follow-the-Regularized-Leader ( FTRL ) algorithms , which is a generalization of MWU . For clarity of our presentation , we choose to focus on MWU in this paper . the preferred solution . Lahkar & Seymour ( 2013 ) studies online learning under random initialization in population dynamics , where one population game is a common payoff model . In population dynamics , there exists a large population of agents who have different initial choices of mixed strategies , modeled as a distribution . The dynamics proceed by randomly pairing up agents in each round to play a game . Lahkar & Seymour ( 2014 ) extends this framework to other learning dynamics to describe the evolution of a distribution of mixed strategies in a population playing variants of RockPaper-Scissors games establishing local convergence to the interior equilibrium . Recently , further learning models inspired by Q-learning dynamics have been defined ( Hu et al. , 2019 ) . Very little is formally known about the behavior of such models particularly under instability . For zero-sum games with an interior Nash equilibrium , “ the grand escape ” phenomenon follows from Bailey & Piliouras ( 2018 ) ; Cheung ( 2018 ) , who showed that MWU diverges away from Nash . Our results are more general , since their analysis only works with those zero-sum game dynamics , while our technique applies to many other game dynamics as well . Such instability results in zerosum games are undesirable for ML applications such as GANs thus a lot of effort has been invested in developing algorithms that achieve convergence Daskalakis et al . ( 2017 ) ; Mertikopoulos et al . ( 2019 ) ; Daskalakis & Panageas ( 2018 ) ; Gidel et al . ( 2019 ) ; Mokhtari et al . ( 2020 ) ; Perolat et al . ( 2021 ) . Our research direction here is in some sense orthogonal . We aim to characterize in a more detailed fashion the behavior of learning dynamics despite their instability . The instability of MWU , FTRL and other optimization-driven dynamics in games has attracted a lot of attention in recent years . One formal technique to establish the unpredictability of MWU and other dynamics is based on proving Li-Yorke chaos , which has been established robustly in different families of routing/potential/Cournot games ( Palaiopanos et al . ( 2017 ) ; Chotibut et al . ( 2020 ) ; Bielawski et al . ( 2021 ) ; Cheung et al . ( 2021 ) ) . The techniques in those papers are independent from ours as they require the identification of one dimensional invariant manifolds , and Li-Yorke chaos implies the existence of periodic orbits , which is not possible in our systems . In a very recent series of works , Flokas et al . ( 2020 ) ; Giannou et al . ( 2021 ) established that all ( even partially ) mixed equilibria in all games are not asymptotically stable for any choice for FTRL dynamics both in discrete as well as in the idealized continuous-time limit . In fact , it is possible to construct simple matrix games such that the orbits of the resulting dynamics can be arbitrarily complex Andrade et al . ( 2021 ) . If , inspired by ML applications , we allow for more complex differentiable games it becomes even simpler to establish strong instability results for effectively any type of training algorithms ( Letcher ( 2020 ) ; Balduzzi et al . ( 2020 ) ) . The sweeping nature of these strong negative results showcases the practical interest in uncovering more information about the nature of unstable dynamics of games . Our proposed framework introduces a novel , quantitative methodology for their study . The notion of differential entropy ( DE ) has been used as uncertainty measure in many works across multiple disciplines . We note that while the extended definition of DE seems natural , it is not considered as the perfect generalization of SE since it misses some favorable properties , and therefore , as quoted from the popular information theory text of Cover & Thomas ( 2006 ) , “ there is need for some care in using the concept ” .2 Nevertheless , DE remains as an effective measure of uncertainty3 , which is commonly used in physics , economics , control theory , engineering , biology , medical research and beyond . For a general discussion , we refer readers to the text of Cover & Thomas ( 2006 ) . For background information of evolutionary game theory , we recommend Hofbauer & Sigmund ( 1998 ) ; Sandholm ( 2010 ) ; Vega-Redondo ( 1996 ) .

Summary: The paper studies the evolution of uncertainty in multi-agent game dynamics. More specifically, it studies how the probability distribution over the players' cumulative payoffs evolves as players use typical online learning algorithms to play the game.  The game uncertainty is quantified by the notion of Differential Entropy (DE) of such distribution, a quantity related to the Jacobian of the game dynamics. Authors show that DE increases linearly with time for a set of games including two-player zero-sum, coordination games, and population games, confirming the negative convergence results obtained in past works.