Finding General Equilibria in Many-Agent Economic Simulations using Deep Reinforcement Learning

1 INTRODUCTION . Real-world economies can be seen as a general-sum sequential imperfect-information game with many heterogeneous strategic agents , including consumer-workers , firms , and governments . Dynamic general equilibrium models ( DGE ) describe the economic incentives , activity , interactions , and constraints of ( a large collection of individual ) agents to model and predict macro-economic outcomes , such as productivity , equality , and growth ( Heer & Maussner , 2009 ) . Following micro-economic theory ( Mas-Colell et al. , 1995 ) , the behavior of each agent is driven by incentives ( “ rewards ” ) , e.g. , consumers maximize utility ( “ happiness ” ) . For economic analysis , it is key to find the strategic equilibria . e.g. , at an -Nash equilibrium , no agent would unilaterally change its behavior , assuming , for example , that all agents act rationally and optimally ( Nisan et al. , 2007 ) .1 Existing analytical and computational methods often struggle to find explicit DGE equilibria . First , the nonlinear structure of DGE models makes it challenging to find optimal agent policies explicitly . For example , an often-used analytical approach is to solve a linearized version of the DGE dynamics , akin to a Taylor expansion . However , a linearized DGE model may have fewer or different equilibria compared to the full DGE model , and solving the linearized model may choose one equilibrium arbitrarily when many are possible . Furthermore , they may only be good approximations around 1This could be expanded to more closely mimic human behavior through “ boundedly rational ” agents that may act suboptimally or whose affordances or objectives encode cognitive biases or limits ( Kahneman , 2003 ) . steady-state equilibria and not be valid in the presence of shocks , e.g. , when agents change their behavior dramatically in time ( Boneva et al. , 2016 ; Atolia et al. , 2010 ) . Second , formal solutions methods , e.g. , backwards induction and related dynamic programming methods , often can not be solved explicitly and only characterize optimal policies implicitly ( Stokey et al. , 1989 ) . For example , analytical work in economics has studied implicit characterizations of the impact of taxes . One can derive analytical expressions for the distortion in consumer savings or capital due to a taxation policy or the asymptotic behavior of taxes in the far future , the tax rate on the highest-skill agent , or the structure of incentives ( Golosov et al. , 2011 ; Acemoglu et al. , 2010 ) . However , these methods generally can not find the full tax policy explicitly or only describe part of it . Third , computational methods can suffer from the curse of dimensionality , e.g. , when there is a large number of agents . In particular , agent incentives are often misaligned and pose a complex general-sum game . However , enumerating and selecting equilibria in general-sum games is an unsolved challenge ( Bai et al. , 2021 ) . Multi-agent deep reinforcement learning ( RL ) ( Sutton & Barto , 2018 ) is a powerful framework to model agent behavior and has several key advantages for economic analysis . Deep neural network policies can learn nonlinear predictive patterns over high-dimensional state-action spaces , while deep value functions can model complex reward function landscapes over long time horizons . Notably , deep RL has achieved superhuman performance in high-dimensional , sequential games ( Silver et al. , 2017 ; Vinyals et al. , 2019 ; OpenAI , 2018 ) , suggesting it can learn ( approximate ) strategic equilibria in complex settings . Moreover , one does not have to approximate nonlinear world dynamics ( e.g. , as in the DGE model ) ; RL has been successful in highly nonlinear environments ( Tassa et al. , 2018 ) . However , deep RL is still not commonly used in economics , although recent work has started to explore its use in economics ( Zheng et al. , 2020 ; Trott et al. , 2021 ; Hill et al. , 2021 ; Chen et al. , 2021 ) . In particular , deep MARL poses several challenges in the context of economic simulations . Our work is to our knowledge the first empirical application of multi-agent deep RL to general equilibrium economic simulation where all agent types learn . First , a general MARL challenge is that each agent faces a non-stationary environment when multiple agents learn simultaneously ; each agent ’ s experienced reward may depend on what other agents do . For example , a consumer may not be able to buy a desired good if there is not enough supply generated by the corresponding firm . Second , economic agents typically have private information that is not available to other agents , e.g. , a worker ’ s skill level and behavioral policy ( parameters ) are not observable by firms . This means that learning has to be ( partially ) decentralized , as in Dec-POMDPs ( Oliehoek & Amato , 2016 ) . Moreover , a key feature of economic models is that the actions of one economic agent may change the shape of the reward function of another agent , e.g. , a consumer ’ s expendable income changes when firms change prices or governments change taxes . Hence , agents change the optimization problem for other agents during training . In MARL , especially deep MARL , joint learning can be unstable , and naive approaches are challenging to get to work . In the context of our economic models , individually trained policies easily get stuck in trivial equilibria where no labor or production occurs . Contributions To address these challenges , we develop MARL techniques that enable finding equilibria in DGE models with many strategic agents . We analyze DGE models with hundreds of heterogeneous RL agents , a scale that has not been studied before with strategic agents . Given the large number of agents in the simulation , we run both simulation and RL on a GPU , following the WarpDrive framework ( Lan et al. , 2021 ) , so training converges in hours ( rather than days ) . WarpDrive uses deep RL to train neural network policy models that are stored on the GPU , using simulation data that is generated on that same GPU . This system design accelerates training significantly by avoiding copying data unnecessarily and placing both simulation and agent models on the same device . From an algorithmic perspective , we generalize the curriculum approach from Zheng et al . ( 2020 ) to structured learning curricula for simulations with multiple agent types . This approach yields nontrivial solutions more stably compared to using agent-independent RL . We also show our approach is sound , e.g. , by showing that an RL government can improve social welfare vs fixed baselines . From a modeling perspective , our approach is more flexible and can find stable solutions in a variation of real-business-cycle ( RBC ) models ( Pierre Danthine & Donaldson , 1993 ) , a family of DGE models . We find equilibria in closed and open RBC economies , the latter has a price-taking export market . From a game theory perspective , we show that our MARL approach can explictly find -Nash equilibria for the meta-game over agent types , without approximating the DGE model . Here , the meta-game equilibrium is a set of agent policies such that no agent type can unilaterally improve its reward by more than ( its best-response ) . Previous solutions only could do so implicitly or for approximations of the DGE dynamics . Furthermore , we learn a spectrum of equilibria in open RBC economies , i.e. , explicitly learn the equilibrium behavioral policies . More general types of equilibria may exist in DGE models but can be hard to validate . For instance , there may be asymmetric Stackelberg-like equilibria , where a leader ( e.g. , the government ) acts first ( e.g. , sets taxes ) , and the followers ( e.g. , consumers and firms ) respond . This differs from symmetric Nash equilibria where all agents act simultaneously . However , analyzing the equilibria in Stackelberglike , general-sum games is an open challenge ( Bai et al. , 2021 ) . For instance , with 2 or more ( heterogeneous ) followers , finding the Stackelberg best-response to a fixed leader requires finding multi-agent equilibria for the followers . This is computationally expensive and there is no known provably convergent algorithm for multi-level Stackelberg games . Instead , we use MARL to converge to a stable solution and analyze best-responses to evaluate to what extent it is an “ equilibrium ” . Moreover , which equilibrium agents converge to ( “ equilibrium selection ” ) may depend on , e.g. , the 1 ) world dynamics , 2 ) initial conditions and policies , 3 ) learning algorithm , and 4 ) policy model class . Our framework allows us to study how these factors relate to which equilibria can be found . 2 RELATED WORK . We highlight some related work ; for a more thorough discussion , see Appendix A . The AI Economist uses RL to design optimal economic policies in simulation environments ( Zheng et al. , 2020 ; Trott et al. , 2021 ) . Other works also use deep RL for economic modeling , but they typically only allow one agent type to learn , and make assumptions like market-clearing ( Hill et al. , 2021 ; Chen et al. , 2021 ) . Perhaps most similar to our work is the agent-based model of Sinitskaya & Tesfatsion ( 2015 ) , which uses tabular Q-learning in a discretized environment for small numbers of identical consumer-workers and firms ( but no government ) , with looser market-clearing constraints . Discussed in the Appendix are other works related to the use of RL for games with leader-follower or Stackelberg structure , and the use of computational methods for approximating equlibria . 3 CASE STUDY : REAL-BUSINESS-CYCLE MODELS . We show how RL can find equilibria in real-business-cycle models ( RBC ) , a representative DGE model in which consumers earn income from labor and buy goods , firms produce goods using capital and labor , and the government taxes income and profits ( Pierre Danthine & Donaldson , 1993 ) , see Figure 1 . RBC models are stylized and may not fully describe reality ( Summers et al. , 1986 ) . However , RBC models are a suitable environment to validate the use of RL , as they feature heterogenous agents with nonlinear economic interactions , making it challenging to find equilibria . Below , we describe the RBC dynamics ; for low-level details , see Appendix D. At a high level , our model includes worker-consumers , price-and-wage setting firms , and a government who sets tax rates and redistributes . A key point about our model is that we do not assume as part of the environment that prices and wages are set so that markets clear at each time step – that is , goods may be overdemanded and firms are free to set prices and wages lower or higher than would balance supply and demand . These assumptions are an essential part of the techniques used to derive analytic solutions – avoiding this modeling choice requires using other techniques . Agent Types . Formally , our RBC model can be seen as a Markov Game ( Littman , 1994 ) with multiple agent types and partial observability . The simulations proceeds in finite-length episodes with T timesteps . Each timestep represents a quarter of 3 months . At each timestep t , we simulate • firms and their goods who use the labor of the worker-consumers to each produce a different good , indexed by i ∈ I , • consumers , indexed by j ∈ J , who work and consume goods , and • a government who sets a tax rate on income from labor and on revenue from selling goods . Each agent iteratively receives an observation oi , t of the world state st , executes an action ai , t sampled from its policy πi , and receives a reward ri , t . The environment state s consists formally of all agent states and a general world state ( see Appendix D for details ) . At each timestep t , all agent simultaneously execute actions . However , some actions only apply to the next timestep t+ 1 . For example , the government sets taxes that will apply at the next timestep t + 1 . These tax rates are observable by the firms and consumer-workers at timestep t ; hence , they can condition their behavioral policy in response to the government policy . Similarly , at timestep t , firms set prices and wages that will be part of the global state at the next t + 1 . This setup is akin to , though not exactly corresponding to , a Stackelberg leader-follower structure , where , for example , the government ( leader ) moves first , and the firms and consumer ( followers ) and move second ( von Stackelberg et al. , 2010 ) . This typically gives a strategic advantage to the followers : they have more information to condition their policy on . Consumer-Workers . Individual people both consume and work ; we will refer to them as consumerworkers . At each timestep t , person j works lj , t hours and consumes ci , j , t units of good i . Each person chooses to work for a specific firm i at each timestep . Consumer-workers also ( attempt to ) consume ĉi , j , t . Each firm ’ s good has a price pi ( set by the firms , described below ) , and the government also sets an income tax rate . However , consumers can not borrow or outspend their budget : if the cost of attempted consumption exceeds the budget , then we scale consumption so that∑ i piĉt , i , j = Bj . Moreover , the realized consumption depends on the available inventory of goods ( yi , t , described below ) . The total demand for good i is ĉi , t = ∑ j ĉi , j , t . If there is not enough supply , we ration goods proportionally : ci , j , t = min ( 1 , yi , t ĉi , t ) ĉi , j , t . ( 1 ) Consuming and working change a consumer ’ s budget Bj , t . Consumer j has labor income zj , t =∑ i li , j , twi , t ; each firm pays a wage wi , t . The cost of consumption is ∑ i pi , t · ci , j , t Moreover , with tax rate τt , workers pay income tax τt · zj , t ; the total tax revenue Rt ( which also includes taxes on the firms , described below ) is redistributed evenly back to workers . In all , consumer budgets change as : Bt+1 , j = Bj , t + ( 1− τt ) zj , t + Rt |J | − ∑ i pi , t · ci , j , t . ( 2 ) Each consumer optimizes its behavioral policy to maximize utility : max πj Ec , l∼πj [ ∑ t γtc ∑ i u ( ci , j , t , li , j , t , θj ) ] , u ( c , l , θ ) = ( c+ 1 ) 1−η − 1 1− η − θ 2 lt , ( 3 ) where γc is the consumer ’ s discount factor . The utility function is a sum of isoelastic utility over consumption and a linear disutility of work with coefficient θj that can vary between workers . Firms . At each timestep t , a firm receives labor from workers , produces goods , sells goods , and may invest in capital . At each timestep t , it sets a price pt+1 , i for its good and chooses a wage wt+1 , i to pay , both effective at the next timestep t+ 1 . If a firm invests ∆ki , t in capital , its capital increases as ki , t+1 = ki , t + ∆ki , t . Using its capital ki , t and total labor Li , t = ∑ j li , j , t ( hours worked ) , a firm i produces Y units of good i , modeled using the production function : Y i , t = Aik 1−α i , t L α i , t , ( 4 ) where 0 ≤ α ≤ 1 sets the importance of capital relative to labor . Also , consumers buy Ci , t =∑ j ci , j , t units of good i . Accordingly , inventories change as yt+1 , i = yi , t + Y i , t −Ci , t . Inventories are always positive , as only actually produced goods can be consumed . The firms receive a profit ( or loss ) P , pay taxes on their profit , and experience a change in their budget B : P i , t = pi , tCi , t − wi , tLi , t −∆ki , t , Bt+1 , i = Bi , t + ( 1− σt ) P i , t , ( 5 ) where σ is the corporate tax rate . The government receives σtP i , t . Firms may borrow and temporarily be in debt ( negative budget ) , but should have non-negative budget at the end of an episode ( no-Ponzi condition ) . This may allow firms to invest more , which may lead to higher future economic growth . Each firm optimizes its behavioral policy to maximize profits as in Equation 5 : max πi Ep , w , ∆k∼πi [ ∑ t γtfP ( pj , t , wj , t , ∆kj , t ) ] , ( 6 ) where γf is the firm ’ s discount factor . The firm gets a negative penalty if it violates no-Ponzi . Government . The government , or social planner , indexed by p , sets corporate and income tax rates , and receives total tax revenue Rt = σt ∑ j zj , t + τt ∑ i P i , t . As a modeling choice , this revenue is redistributed evenly across consumer-workers ; as such , the government ’ s budget is always 0 . The government optimizes its policy πp to maximize social welfare : max πp Eτ , σ∼πp [ ∑ t γtpswf ( τt , σt , st ) ] , ( 7 ) where swf ( st ) is the social welfare at timestep t , and γp is the government ’ s discount factor . In this work , we consider two definitions of social welfare ( although many other definitions are possible ) : 1 ) total consumer utility ( after redistribution ) , or 2 ) total consumer utility and total firm profit . Open and Closed Economies via Export Markets . We consider both open and closed economies . In the open economy , firms can also sell goods to an external market which acts as a price-taker : their demand does not depend on the price of a good . Operationally , export happens after workerconsumers consume . The export market has a minimum price pexport and a cap qexport . If the price of good i is greater than the minimum price ( pi , t > pexport ) then the additional export consumption is ct , export = min ( qexport , yi , t − Ci , t ) , at price pi , t , i.e. , the exported quantity is insensitive to the price . From a learning perspective , the export market prevents firms from seeing extremely low total demand for their good , e.g. , when prices are exorbitantly high and consumers do not want or can not consume the good . In such cases , an on-policy learner that represents a firm may get stuck in a suboptimal solution with extremely high prices and no production as consumers cease to consume in response .

Summary: This paper proposes a multi-agent deep reinforcement learning (DRL) method to compute general equilibria in economics. The main contribution is hence algorithmic. The method uses a combination of DRL, structured learning curricula, and suitable annealing of action space and of some penalty coefficients (which are useful to make some problems easier to solve, but distort the computed solution). The authors apply their method to an example of real-business-cycle model, which involve three types of agents: firms, consumers and a government. The problem is of Stackelberg (or leader-follower) type, with the government acting as a leader. Computing equilibria for such problems is generally very challenging.