Multi-agent Performative Prediction: From Global Stability and Optimality to Chaos

1 INTRODUCTION . Performative prediction ( Perdomo et al. , 2020 ) is a recently introduced framework that focuses on a natural but largely unexplored element of supervised learning . In many practical cases the predictive model can affect the very outcome that it is trying to predict . For example , predictions about which part of a road network will have high congestion trigger responses from the drivers which affect the resulting traffic realization leading to a shift of the target distribution . In such settings , ( Perdomo et al. , 2020 ) explored conditions for the existence and approximate optimality of stable equilibria of such processes . One possible way to interpret the performative prediction setting is a single agent “ game ” , where the predictive agent is playing a game against himself . An agent chooses the parameters of his model as his actions but the predictive accuracy/cost of the model depends on his own past actions . Fixed points of this process do not allow for profitable deviations . Once cast in this light , it becomes self-evident that the restriction to a single predictive agent/model is arguably only the first step in capturing more general phenomena where there is a closed loop between predictive models and their environment . This motivates our central question : What is the interplay between stability , optimality in cases where multiple predictive models operate in parallel to each other ? Can the competition between multiple models lead to novel phenomena such as phase transitions from stability to instability and chaos ? A natural setting to consider for example is market competition , where multiple hedge funds are trying to simultaneously predict future prices , volatility of financial instruments . Of course as they act upon their predictions they also move the prices of these commodities in a highly correlated way . As more agents enter the market and the competition becomes increasingly fierce , is it possible that at some point the market flips from quickly discovering accurate stable predictions reflecting the underlying market fundamentals to self-induced unpredictability and chaos ? When it comes to performative prediction can too many cooks spoil the soup ? Our model Standard supervised learning consists of three components : a set of predictive models , a loss function , and a data distribution . The learner ( agent ) observes samples of the distribution , and then decides a predictive model . When predictions are performative , the agent ’ s decision of a predictive model influences the data distribution . Thus , instead of a fixed data distribution , a predictive prediction also has a distribution map : a mapping from the agent ’ s predictive models to data distributions . We further propose multi-agent performative prediction which model the influence of multiple agents ’ decisions on the data distribution , and ask whether these influence leads to convergent or chaotic systems . To understand the framework of multi-agent performative prediction , we study a natural regression problem with multi-agent location-scale distribution maps ( definition 2.2 ) where the agents ’ influence is linear in the agents ’ models . Specifically , the data consist of features and outcomes . For each agent i ∈ [ n ] , his influence on the distribution of outcome is his model weighted by a scalar λi > 0 . When n = 1 , our multi-agent location-scale distribution map is a special case of locationscale family in Miller et al . ( 2021 ) . In the market example , each hedge fund company tries to predict the price ( outcome ) based on macroeconomic data or other information ( features ) . These influence parameters λ1 , . . . , λn can be seen as each hedge fund ’ s capital that can influence the distribution of the future price . Finally , we consider the agents do not know the dependency between their model and the data distribution , and they can only improve their predictive models myopically and iteratively . In this paper , the agents myopically use a reinforcement learning algorithm , exponentiated gradient for linear regression ( Kivinen & Warmuth , 1997 ) , to improve their predictive models in rounds . We study the long-term behavior , and ask if the system converge to the performative stable and optimal point , or behave chaotically . Our results We first study basic properties of our multi-agent preformative prediction with multiagent location-scale distribution map . We show 1 ) the existence of performative stable point in our setting ( proposition 3.1 ) and 2 ) the performative stability and performative optimality are equivalent ( proposition 3.2 ) . This equivalence allows us to focus on the dynamical behavior of the system . In section 4 , we introduce learning dynamics to the multi-agent performative prediction , and study their long-term behavior . We provide a threshold result which depends on the learning rates of exponentiated gradient descent and the collective influence . Theorem 4.1 shows the dynamics of exponential gradient descent converge to the performative stable and optimal point when the learning rate is small enough . Our convergence result in theorem 4.1 holds when the feature space is multi-dimensional , and every learning agent can use different learning rates starting at arbitrarily interior states . Our exact convergence result also works in the single-agent performative prediction setting . Contrarily , previous convergent results in Perdomo et al . ( 2020 ) ; Miller et al . ( 2021 ) ; Mendler-Dünner et al . ( 2020 ) only show that their dynamics converge to a small neighborhood of the performative stable point . On the other hand , section 4.2 shows the dynamics can have chaotic behavior if the collective influence Ln is large enough . Specifically , theorem 4.6 shows that even when the feature space is in R2 these systems provably exhibit Li-Yorke chaos , when the collective influence ∑ i λi is large enough . This implies that there exists an uncountable “ scrambled ” set so that given any two initial conditions in the set , the liminf of the distance between these two dynamics is zero , but the limsup of their distance is positive . ( definition 2.4 ) The chaotic result in theorem 4.6 also holds for the original single-agent performative prediction so long as the agent ’ s influence is large enough , and , thus , complements previous performative prediction works on convergence behavior , which primarily consider that the agent ’ s influence on the data distribution is sufficiently small . Moreover , no matter how small the agents ’ learning rates , theorem 4.6 shows that chaos is inevitable in some performative predictions settings when the number of agents exceeds a carrying capacity . After that the system becomes totally unpredictable with small perturbations exploding exponentially fast . 1 Finally , section 5 provides numerical examples of our dynamics and show convergent and chaotic behavior . Additionally , through simulation , we demonstrate that our convergent and chaotic results also hold when the agents can only access noisy estimation of the gradient and conduct stochastic exponentiated gradient descent . Related work Data distribution shift is not a new topic in ML , but earlier works focused primarily on exogenous changes to the data generating distribution . Performativity is machine learning was introduced by Perdomo et al . ( 2020 ) . The original work and several follow-ups study the discrepancy between performative stability and performative optimality and prove approximated conver- 1We highlight our revision in blue color . gence of learning dynamics , e.g. , stochastic gradient descent , or iterative empirical risk minimization . ( Mendler-Dünner et al. , 2020 ; Drusvyatskiy & Xiao , 2020 ; Izzo et al. , 2021 ) Performativity of prediction is also related to several applications : strategic classification ( Hardt et al. , 2016 ) , retraining Bartlett ( 1992 ) ; Kuh et al . ( 1990 ) . Inspired by the instability of training algorithms in ML applications such as Generative Adversarial Networks ( GANs ) , there has been a lot of recent interest in understanding conditions ( particularly in multi-agent systems ) where learning behavior may be non-equilibrating/unstable ( Cheung & Tao , 2020 ; Balduzzi et al. , 2020 ; Flokas et al. , 2020 ; Andrade et al. , 2021 ; Letcher , 2021 ; Giannou et al. , 2021 ) . The ( in ) stability and performance of exponentiated gradient descent in particular ( also referred to as Multiplicative Weights Updates ) and other closely related dynamics has attracted a lot of attention ( Cohen et al. , 2017 ; Bailey & Piliouras , 2018 ; Cheung , 2018 ; Panageas et al. , 2019 ; Cheung & Piliouras , 2020 ; Vadori et al. , 2021 ) . The technique of Li-Yorke chaos has recently found applications across several different domains such as routing games , Cournot games and blockchain protocols ( Palaiopanos et al. , 2017 ; Chotibut et al. , 2020 ; Bielawski et al. , 2021 ; Cheung et al. , 2021 ; Leonardos et al. , 2021 ) . To our knowledge , this is the first time where such formal chaotic results are established in settings related to performative prediction and supervised learning more generally . Another line of related works is learning in games . Specifically , our dynamics can be seen as special cases multiplicative weight update ( Hedge algorithms ) on congestion games . Previous works on Hedge algorithms only show exact convergence when the learning rate is decreasing Kleinberg et al . ( 2009 ) ; Cohen et al . ( 2017 ) , and , to our best knowledge , our results are the first that shows exact convergence of Hedge algorithms with small constant learning rates . Our results also complement the exact convergence result of the linear variant of multiplicative weight update by Palaiopanos et al . ( 2017 ) . 2 PRELIMINARY . 2.1 MULTI-AGENT PERFORMATIVE PREDICTION . A multi-agent performative prediction comprises n agents deploying their predictive models fθ1 , . . . , fθn with parameters θ1 , θ2 , . . . , θn ∈ Θ that collectively influence the future data distribution . We formalize such dependency via a distribution map D ( · ) which outputs a distribution on the data , D ( ~θ ) , given a models profile ~θ : = ( θ1 , . . . , θn ) ∈ Θn . A loss function ` ( z , θ′ ) measures the loss of a model fθ′ on a data point z ∈ Z , and the expected loss on a distribution D is Ez∼D [ ` ( z , θ′ ) ] . For performative prediction , we further define the decoupled performative loss on a distribution mapping D as ` ( ~θ , θ′ ) : = Ez∼D ( ~θ ) ` ( z , θ ′ ) where θ′ ∈ Θ denotes a predictive model , while ~θ ∈ Θn denotes a deployed model profile . Given the set of model Θ , the loss function ` , and the distribution mapping D , each agent in a multi-agent performative prediction ( Θ , ` , D ) pursues minimal loss on the distribution that they collectively induce . We consider two solution concepts performative optimality and performative stability which generalizes the original ones in Perdomo et al . ( 2020 ) . Definition 2.1 . Given ( Θ , ` , D ) , a models profile ~θ∗∈ Θn is performatively optimal if the total loss is minimized , ~θ∗ ∈ arg min ~θ∈Θn ∑ i ` ( ~θ , θi ) . Another desirable property of a model profile is that , given all agents deploy their models , their models are also simultaneously optimal for distribution that their model induces . Formally , ~θ∗ is performatively stable if for all i ∈ [ n ] ( θ∗ ) i ∈ arg min θi∈Θ ` ( ~θ∗ , θi ) . The performative optimality does not implies the performative stable point . For performative optimal point , the variable for minimization ~θ affects both the first and the second argument , but only affect the second one for performative stable point . Now we introduce our model in this paper . Location-scale distribution map In this paper , we study the family of multi-agent location-scale map for regression problem where a data point consists of d-dimensional feature and scalar outcome , z = ( x , y ) . These are natural classes of distribution maps in which performative effects enter through an additive or multiplicative factor that is linear in ~θ . Definition 2.2 . Given d ∈ N and d ≥ 2 , and Θn ⊆ Rd , a distribution map D : Θn → Rd × R is a multi-agent location-scale distribution map on n parties if there exists a static distribution DX on Rd+1 , θ0 ∈ Rd , and n linear functions Λ1 , . . . , Λn from Rd to Rd so that the distribution of ( x , y ) ∼ D ( ~θ ) has the following form : The feature is x ∈ Rd and noise x0 ∈ R is jointly sampled from DX . Given feature x ∈ Rd and noise x0 ∈ R , and the the outcome is y = 〈 θ0 − n∑ i=1 Λi ( θ i ) , x 〉 + x0 . In this paper , we consider the scaling maps Λi ( θ ) = λiθ for all θ ∈ Θ with scalar λi > 0 for all i ∈ [ d ] . We call λ : = ( λ1 , . . . , λn ) the influence parameters , and Ln : = ∑n i=1 λi collective influence . Furthermore , we let A : = E [ xx > ] ∈ Rd×d be the covariance matrix of the feature , and b : = Aθ0 + E [ x0x ] ∈ Rd . We will specify the multi agent location-scale distribution map with parameters n , d , λ , A , and b . When n = 1 , our multi-agent location-scale distribution map is a special case of location-scale family in Miller et al . ( 2021 ) where the model θ may both the outcome y as well as the feature x. Predictive Models and Loss Function We consider linear predictive model with constraint where fθ′ ( x ) = 〈θ′ , x〉 , and the collection of parameter is the d-simplex , Θ = { θ : ∑d k=1 θk = 1 , θk ≥ 0 } . We use mean squared error to measure a predictive model θ′ ∈ Θ on a distribution map with a deployed model profile ~θ ∈ Θn , ` ( ~θ , θ′ ) = E ( x , y ) ∼D ( ~θ ) [ ( y−fθ′ ( x ) ) 2 ] = E ( x , y ) ∼D ( ~θ ) [ ( y−θ ′ ·x ) 2 ] . Given a deployed model profile ~θ and a predictive model θ′ , the gradient of the decoupled loss is g ( ~θ , θ′ ) : = ∇θ′ ` ( ~θ , θ′ ) , and ifD is a location-scale distribution map , g ( ~θ , θ′ ) = E ( x , y ) ∼D ( ~θ ) [ 2 ( θ ′ · x− y ) x ] . Furthermore , withA and b defined in definition 2.2 , the gradient can be written as g ( ~θ , θ′ ) = ∇θ′ ` ( ~θ , θ′ ) = 2A ( θ′ + ∑ i λiθ i ) − 2b . ( 1 ) Additionally , given a deployed model profile ~θ and a predictive model profile ~θ′ , we define the gradient of agent i ’ s decoupled loss as gi ( ~θ , ~θ′ ) : = g ( ~θ , ( θ′ ) i ) , and gi ( ~θ ) : = gi ( ~θ , ~θ ) when the deployed model profile is identical to the predictive model profile . We denote the gradient of agent i ’ s average loss as ḡi ( ~θ ) : = ∑ l θ i lg i l ( ~θ ) for all ~θ ∈ Θn . Finally , we define ~ξ ( ~θ ) = ( ξ1 , . . . , ξn ) with ξi ∈ Rd so that for all i ∈ [ n ] and k ∈ [ d ] ξik ( ~θ ) : = θik ( ∑ θilg i l − gik ) . For brevity , we omit ~θ and define ~g : = ~g ( ~θ ) and ~ξ : = ~ξ ( ~θ ) when there is no ambiguity .

Summary: This paper studied the different behaviours of using exponentiated gradient descent (Def. 2.3) in linear regression with different learning rates. The setting is called performative prediction, which can be viewed as a special case of reinforcement learning (after the model makes a prediction, the environment returns a feedback by changing the data distribution).  The main results consist of two parts, for small enough and large learning rates:  (i) with small enough learning rate, the authors proved that the exponentiated gradient descent is "stable" (asymptotically converges to some global minimizer). The proof is based on standard analysis of studying a potential (surrogate/Lyapunov) function for the mean squared error, and arguing that the gradient flow vanishes only if the potential approaches $0$, which implies that the convergence of the flow. Then the discrete gradient update is approximating the flow well if the stepsize is small enough.  (ii) if the learning rate is larger than some threshold value, the authors then showed that the exponentiated gradient descent became Li-Yorke chaotic, where it can periodically oscillate for infinitely many times. The main idea is to use Eq. (7) to characterize the dynamics, and show that under some conditions Eq. (7) induces a sequence of { x_i }_{ i >= 1}, with some recurrent behaviours as shown in Lemma 4.7.  The authors then verified the theoretical findings using simulations on simple examples.