Finding General Equilibria in Many-Agent Economic Simulations using Deep Reinforcement Learning

1 INTRODUCTION . Real-world economies can be seen as a general-sum sequential imperfect-information game with many heterogeneous strategic agents , including consumer-workers , firms , and governments . Dynamic general equilibrium models ( DGE ) describe the economic incentives , activity , interactions , and constraints of ( a large collection of individual ) agents to model and predict macro-economic outcomes , such as productivity , equality , and growth ( Heer & Maussner , 2009 ) . Following micro-economic theory ( Mas-Colell et al. , 1995 ) , the behavior of each agent is driven by incentives ( “ rewards ” ) , e.g. , consumers maximize utility ( “ happiness ” ) . For economic analysis , it is key to find the strategic equilibria . e.g. , at an -Nash equilibrium , no agent would unilaterally change its behavior , assuming , for example , that all agents act rationally and optimally ( Nisan et al. , 2007 ) .1 Existing analytical and computational methods often struggle to find explicit DGE equilibria . First , the nonlinear structure of DGE models makes it challenging to find optimal agent policies explicitly . For example , an often-used analytical approach is to solve a linearized version of the DGE dynamics , akin to a Taylor expansion . However , a linearized DGE model may have fewer or different equilibria compared to the full DGE model , and solving the linearized model may choose one equilibrium arbitrarily when many are possible . Furthermore , they may only be good approximations around 1This could be expanded to more closely mimic human behavior through “ boundedly rational ” agents that may act suboptimally or whose affordances or objectives encode cognitive biases or limits ( Kahneman , 2003 ) . steady-state equilibria and not be valid in the presence of shocks , e.g. , when agents change their behavior dramatically in time ( Boneva et al. , 2016 ; Atolia et al. , 2010 ) . Second , formal solutions methods , e.g. , backwards induction and related dynamic programming methods , often can not be solved explicitly and only characterize optimal policies implicitly ( Stokey et al. , 1989 ) . For example , analytical work in economics has studied implicit characterizations of the impact of taxes . One can derive analytical expressions for the distortion in consumer savings or capital due to a taxation policy or the asymptotic behavior of taxes in the far future , the tax rate on the highest-skill agent , or the structure of incentives ( Golosov et al. , 2011 ; Acemoglu et al. , 2010 ) . However , these methods generally can not find the full tax policy explicitly or only describe part of it . Third , computational methods can suffer from the curse of dimensionality , e.g. , when there is a large number of agents . In particular , agent incentives are often misaligned and pose a complex general-sum game . However , enumerating and selecting equilibria in general-sum games is an unsolved challenge ( Bai et al. , 2021 ) . Multi-agent deep reinforcement learning ( RL ) ( Sutton & Barto , 2018 ) is a powerful framework to model agent behavior and has several key advantages for economic analysis . Deep neural network policies can learn nonlinear predictive patterns over high-dimensional state-action spaces , while deep value functions can model complex reward function landscapes over long time horizons . Notably , deep RL has achieved superhuman performance in high-dimensional , sequential games ( Silver et al. , 2017 ; Vinyals et al. , 2019 ; OpenAI , 2018 ) , suggesting it can learn ( approximate ) strategic equilibria in complex settings . Moreover , one does not have to approximate nonlinear world dynamics ( e.g. , as in the DGE model ) ; RL has been successful in highly nonlinear environments ( Tassa et al. , 2018 ) . However , deep RL is still not commonly used in economics , although recent work has started to explore its use in economics ( Zheng et al. , 2020 ; Trott et al. , 2021 ; Hill et al. , 2021 ; Chen et al. , 2021 ) . In particular , deep MARL poses several challenges in the context of economic simulations . Our work is to our knowledge the first empirical application of multi-agent deep RL to general equilibrium economic simulation where all agent types learn . First , a general MARL challenge is that each agent faces a non-stationary environment when multiple agents learn simultaneously ; each agent ’ s experienced reward may depend on what other agents do . For example , a consumer may not be able to buy a desired good if there is not enough supply generated by the corresponding firm . Second , economic agents typically have private information that is not available to other agents , e.g. , a worker ’ s skill level and behavioral policy ( parameters ) are not observable by firms . This means that learning has to be ( partially ) decentralized , as in Dec-POMDPs ( Oliehoek & Amato , 2016 ) . Moreover , a key feature of economic models is that the actions of one economic agent may change the shape of the reward function of another agent , e.g. , a consumer ’ s expendable income changes when firms change prices or governments change taxes . Hence , agents change the optimization problem for other agents during training . In MARL , especially deep MARL , joint learning can be unstable , and naive approaches are challenging to get to work . In the context of our economic models , individually trained policies easily get stuck in trivial equilibria where no labor or production occurs . Contributions To address these challenges , we develop MARL techniques that enable finding equilibria in DGE models with many strategic agents . We analyze DGE models with hundreds of heterogeneous RL agents , a scale that has not been studied before with strategic agents . Given the large number of agents in the simulation , we run both simulation and RL on a GPU , following the WarpDrive framework ( Lan et al. , 2021 ) , so training converges in hours ( rather than days ) . WarpDrive uses deep RL to train neural network policy models that are stored on the GPU , using simulation data that is generated on that same GPU . This system design accelerates training significantly by avoiding copying data unnecessarily and placing both simulation and agent models on the same device . From an algorithmic perspective , we generalize the curriculum approach from Zheng et al . ( 2020 ) to structured learning curricula for simulations with multiple agent types . This approach yields nontrivial solutions more stably compared to using agent-independent RL . We also show our approach is sound , e.g. , by showing that an RL government can improve social welfare vs fixed baselines . From a modeling perspective , our approach is more flexible and can find stable solutions in a variation of real-business-cycle ( RBC ) models ( Pierre Danthine & Donaldson , 1993 ) , a family of DGE models . We find equilibria in closed and open RBC economies , the latter has a price-taking export market . From a game theory perspective , we show that our MARL approach can explictly find -Nash equilibria for the meta-game over agent types , without approximating the DGE model . Here , the meta-game equilibrium is a set of agent policies such that no agent type can unilaterally improve its reward by more than ( its best-response ) . Previous solutions only could do so implicitly or for approximations of the DGE dynamics . Furthermore , we learn a spectrum of equilibria in open RBC economies , i.e. , explicitly learn the equilibrium behavioral policies . More general types of equilibria may exist in DGE models but can be hard to validate . For instance , there may be asymmetric Stackelberg-like equilibria , where a leader ( e.g. , the government ) acts first ( e.g. , sets taxes ) , and the followers ( e.g. , consumers and firms ) respond . This differs from symmetric Nash equilibria where all agents act simultaneously . However , analyzing the equilibria in Stackelberglike , general-sum games is an open challenge ( Bai et al. , 2021 ) . For instance , with 2 or more ( heterogeneous ) followers , finding the Stackelberg best-response to a fixed leader requires finding multi-agent equilibria for the followers . This is computationally expensive and there is no known provably convergent algorithm for multi-level Stackelberg games . Instead , we use MARL to converge to a stable solution and analyze best-responses to evaluate to what extent it is an “ equilibrium ” . Moreover , which equilibrium agents converge to ( “ equilibrium selection ” ) may depend on , e.g. , the 1 ) world dynamics , 2 ) initial conditions and policies , 3 ) learning algorithm , and 4 ) policy model class . Our framework allows us to study how these factors relate to which equilibria can be found . 2 RELATED WORK . We highlight some related work ; for a more thorough discussion , see Appendix A . The AI Economist uses RL to design optimal economic policies in simulation environments ( Zheng et al. , 2020 ; Trott et al. , 2021 ) . Other works also use deep RL for economic modeling , but they typically only allow one agent type to learn , and make assumptions like market-clearing ( Hill et al. , 2021 ; Chen et al. , 2021 ) . Perhaps most similar to our work is the agent-based model of Sinitskaya & Tesfatsion ( 2015 ) , which uses tabular Q-learning in a discretized environment for small numbers of identical consumer-workers and firms ( but no government ) , with looser market-clearing constraints . Discussed in the Appendix are other works related to the use of RL for games with leader-follower or Stackelberg structure , and the use of computational methods for approximating equlibria . 3 CASE STUDY : REAL-BUSINESS-CYCLE MODELS . We show how RL can find equilibria in real-business-cycle models ( RBC ) , a representative DGE model in which consumers earn income from labor and buy goods , firms produce goods using capital and labor , and the government taxes income and profits ( Pierre Danthine & Donaldson , 1993 ) , see Figure 1 . RBC models are stylized and may not fully describe reality ( Summers et al. , 1986 ) . However , RBC models are a suitable environment to validate the use of RL , as they feature heterogenous agents with nonlinear economic interactions , making it challenging to find equilibria . Below , we describe the RBC dynamics ; for low-level details , see Appendix D. At a high level , our model includes worker-consumers , price-and-wage setting firms , and a government who sets tax rates and redistributes . A key point about our model is that we do not assume as part of the environment that prices and wages are set so that markets clear at each time step – that is , goods may be overdemanded and firms are free to set prices and wages lower or higher than would balance supply and demand . These assumptions are an essential part of the techniques used to derive analytic solutions – avoiding this modeling choice requires using other techniques . Agent Types . Formally , our RBC model can be seen as a Markov Game ( Littman , 1994 ) with multiple agent types and partial observability . The simulations proceeds in finite-length episodes with T timesteps . Each timestep represents a quarter of 3 months . At each timestep t , we simulate • firms and their goods who use the labor of the worker-consumers to each produce a different good , indexed by i ∈ I , • consumers , indexed by j ∈ J , who work and consume goods , and • a government who sets a tax rate on income from labor and on revenue from selling goods . Each agent iteratively receives an observation oi , t of the world state st , executes an action ai , t sampled from its policy πi , and receives a reward ri , t . The environment state s consists formally of all agent states and a general world state ( see Appendix D for details ) . At each timestep t , all agent simultaneously execute actions . However , some actions only apply to the next timestep t+ 1 . For example , the government sets taxes that will apply at the next timestep t + 1 . These tax rates are observable by the firms and consumer-workers at timestep t ; hence , they can condition their behavioral policy in response to the government policy . Similarly , at timestep t , firms set prices and wages that will be part of the global state at the next t + 1 . This setup is akin to , though not exactly corresponding to , a Stackelberg leader-follower structure , where , for example , the government ( leader ) moves first , and the firms and consumer ( followers ) and move second ( von Stackelberg et al. , 2010 ) . This typically gives a strategic advantage to the followers : they have more information to condition their policy on . Consumer-Workers . Individual people both consume and work ; we will refer to them as consumerworkers . At each timestep t , person j works lj , t hours and consumes ci , j , t units of good i . Each person chooses to work for a specific firm i at each timestep . Consumer-workers also ( attempt to ) consume ĉi , j , t . Each firm ’ s good has a price pi ( set by the firms , described below ) , and the government also sets an income tax rate . However , consumers can not borrow or outspend their budget : if the cost of attempted consumption exceeds the budget , then we scale consumption so that∑ i piĉt , i , j = Bj . Moreover , the realized consumption depends on the available inventory of goods ( yi , t , described below ) . The total demand for good i is ĉi , t = ∑ j ĉi , j , t . If there is not enough supply , we ration goods proportionally : ci , j , t = min ( 1 , yi , t ĉi , t ) ĉi , j , t . ( 1 ) Consuming and working change a consumer ’ s budget Bj , t . Consumer j has labor income zj , t =∑ i li , j , twi , t ; each firm pays a wage wi , t . The cost of consumption is ∑ i pi , t · ci , j , t Moreover , with tax rate τt , workers pay income tax τt · zj , t ; the total tax revenue Rt ( which also includes taxes on the firms , described below ) is redistributed evenly back to workers . In all , consumer budgets change as : Bt+1 , j = Bj , t + ( 1− τt ) zj , t + Rt |J | − ∑ i pi , t · ci , j , t . ( 2 ) Each consumer optimizes its behavioral policy to maximize utility : max πj Ec , l∼πj [ ∑ t γtc ∑ i u ( ci , j , t , li , j , t , θj ) ] , u ( c , l , θ ) = ( c+ 1 ) 1−η − 1 1− η − θ 2 lt , ( 3 ) where γc is the consumer ’ s discount factor . The utility function is a sum of isoelastic utility over consumption and a linear disutility of work with coefficient θj that can vary between workers . Firms . At each timestep t , a firm receives labor from workers , produces goods , sells goods , and may invest in capital . At each timestep t , it sets a price pt+1 , i for its good and chooses a wage wt+1 , i to pay , both effective at the next timestep t+ 1 . If a firm invests ∆ki , t in capital , its capital increases as ki , t+1 = ki , t + ∆ki , t . Using its capital ki , t and total labor Li , t = ∑ j li , j , t ( hours worked ) , a firm i produces Y units of good i , modeled using the production function : Y i , t = Aik 1−α i , t L α i , t , ( 4 ) where 0 ≤ α ≤ 1 sets the importance of capital relative to labor . Also , consumers buy Ci , t =∑ j ci , j , t units of good i . Accordingly , inventories change as yt+1 , i = yi , t + Y i , t −Ci , t . Inventories are always positive , as only actually produced goods can be consumed . The firms receive a profit ( or loss ) P , pay taxes on their profit , and experience a change in their budget B : P i , t = pi , tCi , t − wi , tLi , t −∆ki , t , Bt+1 , i = Bi , t + ( 1− σt ) P i , t , ( 5 ) where σ is the corporate tax rate . The government receives σtP i , t . Firms may borrow and temporarily be in debt ( negative budget ) , but should have non-negative budget at the end of an episode ( no-Ponzi condition ) . This may allow firms to invest more , which may lead to higher future economic growth . Each firm optimizes its behavioral policy to maximize profits as in Equation 5 : max πi Ep , w , ∆k∼πi [ ∑ t γtfP ( pj , t , wj , t , ∆kj , t ) ] , ( 6 ) where γf is the firm ’ s discount factor . The firm gets a negative penalty if it violates no-Ponzi . Government . The government , or social planner , indexed by p , sets corporate and income tax rates , and receives total tax revenue Rt = σt ∑ j zj , t + τt ∑ i P i , t . As a modeling choice , this revenue is redistributed evenly across consumer-workers ; as such , the government ’ s budget is always 0 . The government optimizes its policy πp to maximize social welfare : max πp Eτ , σ∼πp [ ∑ t γtpswf ( τt , σt , st ) ] , ( 7 ) where swf ( st ) is the social welfare at timestep t , and γp is the government ’ s discount factor . In this work , we consider two definitions of social welfare ( although many other definitions are possible ) : 1 ) total consumer utility ( after redistribution ) , or 2 ) total consumer utility and total firm profit . Open and Closed Economies via Export Markets . We consider both open and closed economies . In the open economy , firms can also sell goods to an external market which acts as a price-taker : their demand does not depend on the price of a good . Operationally , export happens after workerconsumers consume . The export market has a minimum price pexport and a cap qexport . If the price of good i is greater than the minimum price ( pi , t > pexport ) then the additional export consumption is ct , export = min ( qexport , yi , t − Ci , t ) , at price pi , t , i.e. , the exported quantity is insensitive to the price . From a learning perspective , the export market prevents firms from seeing extremely low total demand for their good , e.g. , when prices are exorbitantly high and consumers do not want or can not consume the good . In such cases , an on-policy learner that represents a firm may get stuck in a suboptimal solution with extremely high prices and no production as consumers cease to consume in response .

Summary: This work investigates the use of Deep MARL in order to study economies. They propose and implement an RBC (economic market model) and propose a reward shaping schedule to bias agents into learning non-degenerate joint-strategies. In their setting, low-welfare equilria and provide a brief quantitative and qualitative analysis of discovered equilriba in open- and closed-economies.