Finding General Equilibria in Many-Agent Economic Simulations using Deep Reinforcement Learning
Authors: Michael Curry
Alexander R Trott
Soham Phade
Yu Bai
Stephan Zheng
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
economics
simulation
multi-agent RL
equilibrium

Abstract: Real economies can be seen as a sequential imperfect-information game with many heterogeneous, interacting strategic agents of various agent types, such as consumers, firms, and governments. Dynamic general equilibrium models are common economic tools to model the economic activity, interactions, and outcomes in such systems. However, existing analytical and computational methods struggle to find explicit equilibria when all agents are strategic and interact, while joint learning is unstable and challenging. Amongst others, a key reason is that the actions of one economic agent may change the reward function of another agent, e.g., a consumer's expendable income changes when firms change prices or governments change taxes. We show that multi-agent deep reinforcement learning (RL) can discover stable solutions that are $\epsilon$-Nash equilibria for a meta-game over agent types, in economic simulations with many agents, through the use of structured learning curricula and efficient GPU-only simulation and training.Conceptually, our approach is more flexible and does not need unrealistic assumptions, e.g., market clearing, that are commonly used for analytical tractability. Our GPU implementation enables training and analyzing economies with a large number of agents within reasonable time frames, e.g., training completes within a day. We demonstrate our approach in real-business-cycle models, a representative family of DGE models, with 100 worker-consumers, 10 firms, and a government who taxes and redistributes. We validate the learned meta-game $\epsilon$-Nash equilibria through approximate best-response analyses, show that RL policies align with economic intuitions, and that our approach is constructive, e.g., by explicitly learning a spectrum of meta-game $\epsilon$-Nash equilibria in open economic models.

Under review as a conference paper at ICLR 2022
FINDING GENERAL EQUILIBRIA IN MANY-AGENT ECONOMIC SIMULATIONS USING DEEP REINFORCE- MENT LEARNING
Anonymous authors Paper under double-bling blind review
ABSTRACT
Real economies can be seen a sequential imperfecttinnfommaionn game with many heterogeneous, interacting strategic agents of various agent types, such as con- sumers, firms, and governments. Dynamic general equilibrium models are common economic tools to model the economic activity, interactions, and outcomes in such systems. However, existing analytical and computational methods struggle to find explicit equilibria when all agents strategic and interact, while joint learning is unstable and challenging. Amongst others, key reason is that the actions one economic agent may change the reward function of another agent, e.g., consumer's 's expendable income changes when firms change prices or governments change taxes. We show that multi-agent deep reinforcement learning (RL) discover stable solutions that are e-Nash equilibria for a meta-game over agent types, in economic simulations with many agents, through the use of structured learning curricula and efficient GPU-only simulation and training. Conceptually, our approach is more flexible and does not need unrealistic assumptions, e.g., market clearing, that are commonly used for analytical tractability. Our GPU implementation enables training and analyzing economies with a large number of agents within reasonable time frames, e.g., training completes within day. We demonstrate our approach in -eal-business--cclee models, a representative family DGE models, with 100 worker-consumers, 10 firms, and government who taxes and redistributes. We validate the learned meta-game gaaggaame e-Nash equilibria through approximate best-response analyses, show that RL policies align with economic intuitions, and that our approach is constructive, e.g., by explicitly learning a spectrum of meta-game e-Nash equilibria in open RBC models.
INTRODUCTION
Real-world economies be many heterogeneous strategic agents, including consumer- workers,, firms, and govermments. Dynamic general equilibrium models (DGE) describe the economic incentives, activity, interactions, and constraints of (a large collection of individual) agents to model and predict macro-economic outcomes, such productivity, equality, and growth (Heer & Maussner, 2009). Following micro-economic theory (Mas-Car ColllC et al.. 1995). the behavior of each agent is driven by incentives ("rewards"), c.g.. consumers maximize utility ("happiness"). For economic analysis, equilibria. e.g., at an c-Nash equilibrium, agent would unilaterally change its behavior, assuming, for example, that all agents act rationally and optimally (Nisan et al., 2007).
general-sum sequential imperfect-information game with
key to find the strategic
Existing analytical and computational methods often struggle to find explicit DGE equilibria. First, the nonlinear structure DGE models makes it challenging to find optimal agent policies explicitly. For example, an often-used analytical approach is to solve linearized version of the DGE dynamics, akin to Taylor expansion. However, linearized DGE model may have fewer or different equilibria compared to the full DGE model, and solving the linearized model may choose equilibrium arbitrarily when many are possible. Furthermore, they may only be good approximations around
'This could be expanded to more closely mimic human hehavior through "boundedly rational" agents that may act suboptimally whose affordances objectives encode cognitive biases limits (Kahneman, 2003).

---

Under review as a conference paper at ICLR 2022
steady-state equilibria and not be valid in the presence of shocks, e.g., when agents change their behavior dramatically in time (Boneva et al., 2016; Atolia et al., 2010). Second, formal solutions methods, e.g., backwards induction and related dynamic programming methods, often cannot be solved explicitly and only characterize optimal policies implicitly (Stokey et al., 1989). For example, analytical work in economics has studied implicit characterizations of the impact of taxes. One can derive analytical expressions for the distortion in consumer savings or capital due to taxation policy or the asymptotic behavior of taxes in the far future, the tax rate on the highest-skill agent, or the structure of incentives (Golosov et al., 201 I; Acemoglu et al., 2010). However, these methods generally cannot find the full tax policy explicitly or only describe part of it. Third, computational methods can suffer from the curse of dimensionality, e.g., when there is a large number of agents. In particular, agent incentives are often misaligned and pose a complex general-sum game. However, enumerating and selecting equilibria in general-sum games is an unsolved challenge (Bai et al., 2021).
aueti- deep reinforcement learning (RL) (Sutton Barto, 2018) is a powerful framework to model agent behavior and has several key advantages for economic analysis. Deep neural network policies can learn nonlinear predictive patterns over high-dimensional state-action spaces, while deep value functions
model complex reward function landscapes over long time horizons. Notably, deep RL. has achieved superhuman performance in high-dimensional sequential games (Silver et al., 2017: Vinyals et al., 2019; OpenAI, 2018), suggesting can learn (approximate) strategic equilibria in complex settings. Moreover, one does not have to approximate nonlinear world dynamics (e.g.. as in the DGE model); RL has been successful in highly nonlinear environments (Tassa et al., 2018).
However, deep RL is still not commonly used in economics, although recent work has started to explore its use in economics (Zheng et al., 2020; Trout et al., 2021: Hill al., 2021; Chen al., 2021). In particular, deep MARL poses several challenges in the context of economic simulations. Our work to our knowledge the first empirical application of multi multi-agent deep RL to general equilibrium economic simulation where all agent types learn.
First, general MARL challenge is that each agent faces non-stationary environment when multiple agents learn simultaneously; each agent's s experienced reward may depend on what other agents do. For example, consumer may not be able to buy desired good if there is not enough supply generated by the corresponding firm. Second, economic agents typically have private information that is not available to other agents, e.g., a worker's s skill level and behavioral policy (parameters) are not observable by firms. This means that learning has to be (partially) decentralized, in Dec-POMDDP (Olichock & Amato, 2016). Moreover, a key feature of economic models is that the actions of one economic agent may change the shape of the reward function of another agent, e.g., a consumer's expendable income changes when firms change prices or governments change taxes. Hence, agents change the optimization problem for other agents during training.
In MARL, especially deep MARL, joint learning can be unstable, and naive approaches are challeng- ing to get to work. In the context of our economic models, individually trained policies easily get stuck in trivial equilibria where no labor or production occurs.
Contributions To address these challenges, we develop MARL techniques that enable finding equilibria in DGE models with many strategic agents. We analyze DGE models with hundreds of heterogeneous RL agents, a scale that has not been studied before with strategic agents. Given the large number of agents in the simulation, we run both simulation and RL on a GPU, following the WarpDrive framework (Lan et al., 2021). so training converges in hours (rather than days). WarpDrive uses deep RL to train neural network policy models that are stored on the GPU, using simulation data that is generated on that same GPU. This system design accelerates training significantly by avoiding copying data unnecessarily and placing both simulation and agent models on the same device.
From algorithmic perspective, we generalize the curriculum approach from Zheng et al. (2020) to structured learning curricula for simulations with multiple agent types. This approach yields non- trivial solutions more stably compared to using agent-independent RL. We also show our approach is sound, e.g., by showing that an RL government can improve social welfare vs fixed baselines.
From modeling perspective, our approach is more flexible and can find stable solutions in variation of real- real-business-on business-cycle (RBC) models (Pierre Danthine & Donaldson, 1993). a family of DGE models. We find equilibria in closed and open RBC economies, the latter has price-taking export market.
2

---

Under review as a conference paper at ICLR 2022
Corporate Tax
Income Tax
Wages
Government
Consumer-Worker
Firms
Subsidy
Consumption
Optimize Social Welfare
Maximize Utility
Maximize Profit
Set Income Tax Rates
Consume
Set prices
Export Market
Set Corporate Tax Rates
Work
Set wages
Figure 1: RBC model with consumers, firms, and governments. Arrows represent money flow, Consumer-workers carn wages through work and consume goods from firms. They also strategically choose which firm to work for and which basker of goods to buy, but this not explicitly visualized. Firms produce goods, pay wages, and set a price for their goods. They also invest fixed fraction of profits to increase capital. The government taxes labor income and firm profits, and redistribute the tax revenue through subsidies to the consumer-workent Firms can also sell goods to external export market, which acts as price-taker that willing to consume goods at any price.
From game theory perspective, we show that our MARL approach can explictly find e-Nash equilibria for the meta-game over agent types, without approximating the DGE model. Here, the meta-game equilibrium is a set of agent policies such that no agent type can unilaterally improve its reward by more than c (its best-response). Previous solutions only could do so implicitly for approximations of the DGE dynamics. Furthermore, we learn a spectrun of equilibria in open RBC economies, i.e., explicitly learn the equilibrium behavioral policies.
More general types of equilibria may exist in DGE models but can be hard to validate. For instance, there may asymmetric Stackellber- equilibria, where leader (e.g., the government) acts first (e.g., sets taxes), and the followers (c.g., consumers and firms) respond. This differs from symmetric Nash equilibria where agents act simultancously. However, analyzing the equilibria in Stackelberg- like, general-sum games (heterogeneous) followers, finding the Stackelberg best-response to a fixed leader requires finding multi-agent equilibria for the followers. This computationally expensive and there no known provably convergent algorithm for multi-level Stackelberg games. Instead, we use MARL to converge stable solution and analyze best-responses evaluate to what extent an "equilibrium".
open challenge (Bai al., 2021). For instance, with 2 or more
Moreover, which equilibrium agents converge to ('equilibrium selection") may depend on, c.g., the world dynamics, 2) initial conditions and policies, 3) learning algorithm, and policy model class. Our framework allows us to study how these factors relate to which equilibria can be found.
2 RELATED WORK
We highlight some related work; for more thorough discussion, see Appendix A. The Al Economist uses RL to design optimal economic policies in simulation environments (Zheng et al., 2020; Trott et al., 2021). Other works also use deep RL for economic modeling, but they typically only allow one agent type to learn, and make assumptions like market-clearing (Hill al.. 2021; Chen et al., 2021). Perhaps most similar to our work is the agent-based model of Sinitskaya & Tesfatsion (2015), which uses tabular Q-learning a discretized environment for small numbers of identical consumer-workers and firms (but no government), with looser market-clearing constraints. Discussed in the Appendix are other works related to the use of RL for games with leader-follower or Stackelberg structure, and the use of computational methods for approximating equlibria.
CASE STUDY:
MODELS
We show how RL can find equilibria in real-business-cycle models (RBC), representative DGE model in which consumers earn income from labor and buy goods, firms produce goods using capital and labor, and the government taxes income and profits (Pierre Danthine & Donaldson, 1993), see Figure RBC models are stylized and may not fully describe reality (Summers et al., 1986). However, RBC models suitable environment to validate the use of RI, they feature
3

---

Under review as a conference paper at ICLR 2022
heterogenous agents with nonlinear economic interactions, making it challenging to find equilibria. Below, we describe the RBC dynamics; for low-level details, see Appendix D. At a high level. our model includes and redistributes. A key point about our model is that we do not assume as part of the environment that prices and wages set so that markets clear each time step that is, goods may be overdemanded and firms are free to set prices and wages lower or higher than would balance supply and demand. These assumptions are an essential part of the techniques used derive analytic solutions avoiding this modeling choice requires using other techniques.
price-and-wage setting firms, and a government who sets tax rates
Agent Types. Formally, our RBC model can be seen as a Markov Game (Littman. 1994) with multiple agent types and partial observability. The simulations proceeds in finite-length episodes with timesteps. Each timestep represents a quarter of 3 months. At each timestep 6, we simulate
firms and their goods who use the labor of the worker-consumers to each produce a different good, indexed by E J,
consumers, indexed by j E J, who work and consume goods, and
government who sets a tax rate on income from labor and on revenue from selling goods.
Each agent iteratively receives an observation o of the world state 8t, executes an action as sampled from its policy т;. and receives a reward Tt.t- The enviromment state s consists formally of all agent states and a general world state (see Appendix D for details). each timestep 4, all agent simultaneously execute actions. However, some actions only apply to the next timestep t + 1. For example, the government sets taxes that will apply at the next timestep + 1. These tax rates observable by the firms and consumer-worker at timestep t; hence, they can condition their behavioral policy in response to the government policy, Similarly, at timestep f, firms set prices and wages that will be part of the global state at the next t 1. This setup is akin to, though not exactly corresponding to, a Stackelberg (leader) moves first, and the firms and consumer (followers) and move second (von Stackelberg et al., 2010). This typically gives a strategic advantage to the followers: they have more information to condition their policy on.
structure, where, for example, the government
Consumer- Workers. Individual people both consume and work; we will refer to them as consumer- workers. At each timestep f, person j works 1,,e hours and consumes C,j,t units of good Each person chooses to work for a specific firm i at each timestep. Consumer workers also (attempt to) consume Gj.j.t- Each firm's good has a price p: (set by the firms, described below), and the government also sets an income tax rate. However, consumers cannot borrow or outspend their budget: if the cost of attempted consumption exceeds the budget, then we scale consumption so that E, P. Bj-
Konsumer-workerss
Moreover, the realized consumption depends on the available inventory of goods (Vi,z, described below). The total demand for good i is c G. If there is not enough supply, we ration goods proportionally:
C.j.t min (1. Vi.t 25,t
(1,
Consuming and working change a consumer's budget BjN,, Consumer j has labor income Zj,² E, Wij. each firm pays wage Wi,s The cost of consumption is Pi. Pi.t Cija Moreover, with tax rate Tú, workers pay income tax T. Zj.ii the total tax revenue Rg (which also includes taxes on the firms, described below) is redistributed evenly back to workers. In all, consumer budgets change as:
Bi+1 BJ, (1 2zj,
Re
Σ Pla Cig.tt
[J
Each consumer optimizes its behavioral policy maximize utility:
Σ 1 hess Eusleessthst 0,)
u(c, (c+ 1)2-1 -1
0
max ITy
2
where Toe is the consumer's 's discount factor. The utility function is a sum of isoelastic utility over consumption and a linear disutility of work with coefficient 0g that can vary between workers.
4

---

Under review as a conference paper at ICLR 2022
Firms. At each timestep firm receives labor from workers, produces goods, sells goods, and may invest in capital. At each timestep t, it sets a price P1+1, for its good and chooses a wage w,-1, to pay, both effective at the next timestep + 1. If a firm invests Akis in capital, its capital increases as kis,+1 = k + Ak Using its capital ki, and total labor Li, = Ej lig,a (hours worked), a firm i produces Y units of good i, modeled using the production function:
Y A, L
where O S a < sets the importance of capital relative to labor. Also, consumers buy Ca. = Ej Ci.ja units of good i. Accordingly, inventories change as yt+1, = yi, + Y., Ct,,- Inventories are always positive, only actually produced goods can be consumed. The firms receive profit (or loss) P. pay taxes on their profit, and experience change in their budget B:
Ps. = Cs, - wi, Aki,, B = B,,, + (1
(5)
where a is the corporate tax rate. The government receives o, oit. Firms may borrow and temporarily be in debt (negative budget), but should have non-negative budget at the end of an episode (no-Ponzi condition). This may allow firms to invest more, which may lead to higher future economic growth. Each firm optimizes its behavioral policy to maximize profits as in Equation
Σ (Ps. W Akz,)
max Bp,., AKNUT 31
(6)
where yy is the firm's discount factor. The firm gets a negative penalty if it violates no-Ponzi.
Government. The government, or social planner, indexed by p, sets corporate and income tax rates, and receives total tax revenue R₂ or E, Zj, + Ti E, Pi.e- As a modeling choice, this revenue is redistributed evenly across consumer-workers; as such, the government's budget is always O. The government optimizes its policy Tp to maximize social welfare:
Enjswe (Te, 01,
max E Tp
B
where swf (s:) is the social welfare at timestep I, and Tip is the government's 's discount factor. In this work, we consider two definitions social welfare (although many other definitions are possible): 1) total consumer utility (after redistribution), or 2) total consumer utility and total firm profit.
Open and Closed Economies via Export Markets. We consider both open and closed economies. In the open economy, firms can also sell goods to external market which acts price takee: their demand does not depend on the price of good. Operationally, export happens after worker- consumers consume. The export market has minimum price Pexport and a cap Jexpent If the price of good is greater than the minimum price (p:,e Pexport) then the additional export consumption is Ct.export min (gexpont V4. Cr at price Pi,r, i.e., the exported quantity is insensitive the price.
From a learning perspective, the export market prevents firms from seeing extremely low total demand for their good, e.g., when prices are exorbitantly high and consumers do not want or cannot consume the good. In such cases, an on-policy learner that represents a firm may get stuck in suboptimal solution with extremely high prices and no production as consumers cease to consume response.
MULTI-AGENT RL AND STRUCTURED CURRICULA
key idea of our approach is the use of structured multi-agent curricula to stabilize learning, where each individual agent uses standard RL. These curricula consist of staged training, annealing of allowed actions, and annealing of penalty coefficients, see Figure 2.
Base RL. Firms and governments use 3-layer fully-connected neural network policies (a(s), each layer using 128-dim features, that map states to action distributions. Consumer policies are similar, using separate heads for each action type, i.e., the joint action distribution is factorized and depends on a shared neural network feature ye (sx):: T(a), az.... (s) = न ( (yp, s) s) (ppl s) ... (omitting t and s, for clarity). Any correlation between actions is modeled implicitly through you. There is
5

---

Under review as a conference paper at ICLR 2022
Consumer- workers are training
Consumer-workers entropy annealing
disutility magnitude
Available firm actions (prices, wages)
Firms are training
Firm entropy annealing
Available govermment actions (tax
Government is training
Government entropy annealing
Training progress
Figure 2: Structured learning curricula. Colored bars show how annealing activates over time. Consumers are always training, use a decaying entropy regularization, and increasingly experience their labor disutility. Firms start with fixed prices and wages, and are slowly allowed to change these. Once the full price and wage range is available, firms start to train. Similarly, tax rates start at and are slowly allowed to change. Once the full tax range is available, the government starts to train.
single policy network for each agent type, shared across the many agents of that type. To distinguish between agents when selecting actions, agent-specific features (parameters like the disutility of work, production parameters, and for firms, simply a one-hot representation of the firm) are included as part of the policy input state. Thus, despite shared policy for each agent type, we model some degree of heterogeneity among agents. We also learn a value function V (ff) for variance reduction purposes. We compare policies trained using policy gradients (Williams, 1992) or PPO Schulman et al. (2017). See Appendix E. for further training details. Both simulation and RL. ran on a GPU with WarpDrive (Lan et al., 2021), see Appendix G.
Intuition for Structured Curricula. We define curricula based on these observations about our multi-agent learning problem: 1) during exploration, many actions may reduce utility, while few actions increase utility. 2) high prices tax rates can eliminate all gains in utility, even though the consumer-we worker did not change its policy, 3) for stable learning, agents should not adapt their policy too quickly when experiencing large negative (changes in) utility, and 4) in Stackelberg game, the followers (e.g., consumers, firms) should get enough time to learn their best response to the leader's s policy (e.g., the government). We now operationalize these intuitions below.
Staged Learning and Action Space Annealing. All policies are randomly initialized. We first allow consumers train, without updating the policies of other agents. Initially, firm and govermment actions are completely fixed; prices and wages start at non-zero levels. We then anneal the range of firm actions without training the randomly initialized policy. This allows consumers learn to best respond to the full scope of prices and wages, without firms strategically responding. Once firm action annealing is complete, we allow the firm train jointly with the consumers. We then perform the same process, gradually allowing the government to increase its corporate and income tax rates, so that firms and consumers can react to a wide range of tax rates. Once the annealing process complete, we allow the government to train to maximize welfare.
Penalty Coefficient Annealing. In addition to the action annealing, we anneal two penalty co- efficients. First, we slowly increase the consumers' disutility of work over time, which avoids disincentivizing work early in the training process. Second, as each agent starts training, we assign a high value (0.5) for the entropy coefficient in their policy gradient loss, and gradually anneal it down over time to a minimum of O. This ensures that when the firm or government policies start training, their "opponent" policies able to learn against them without being too quickly exploited.
Many local equilibria We expect that there are many possible equilibria in our game that our approach may converge Moreover, convergence may be only local, since optimizing our policy parameters. Our training curriculum is designed to avoid trivial equilibria where the economy shuts down, but may also bias which non-trivial equilibria are reached. However,
not globally

---

Under review as a conference paper at ICLR 2022
Consumer Reward
Fimm
Average Price
Average Wage ($/hour)
2400
20
2200
18
100K
2000
50K
1800
3-
1600
0K
1400
5KK
100KK
1000
50K Triningg episoda
50K
200K
50K
100K
150K
Training episode
Training episode
Training episode
Figure 3: Structured Curricula and Training, Open Economy, 3 runs. In each plot, the blue (black) vertical line shows when the firms (government) start training, see Figure 2. Left two plots: Consumer and firm rewards during training. All runs converged to approximate equilibrium, as confirmed by a best-response analysis. The outcomes were chosen to be qualitatively similar across runs. Once firms start training, their reward significantly increases. When the government starts training, firms get even higher reward, as the social welfare definition includes the firm's 's objective (profits). Right two plots: Average wages and prices (over firms) during training. Firms increase prices rapidly and lower wages once they start training.
Agent
Consumer
Firm
Government
Reward improvement under best response
<3%
<10%
<1%
Table 1: Reward improvement under best response at the end of training as a fraction of the reward improvement during training, over 10 random seeds. We observed that besides a few anomalies, the improvements were in fact less than 0.2% for consumers, 5% for firms, and 0. 0.1% % for the government.
observe spectrum of different outcomes, than are possible with simplified and linearized models.
our approach way to explore equilibria
5 LEARNING RBC EQUILIBRIA AND METHOD VALIDATION
We show that our approach sound and can explicitly find spectra of approximate RBC equilibria. We study variations of our RBC model with 100 consumers and 10 firms. For all simulation parameter settings, see Table and Appendix D. We repeat all experiments with random seeds.
Structured Curricula during Training. Figure 3 shows a representative result of training using our structured curricula. Although all RL agents aim to improve their objective, some agents may see decrease as the system converges towards an equilibrium, e.g., the consumers Figure Empirically, we found that training collapses to trivial solutions much more often without curricula.
Best Responses and Local Equilibrium Analysis We abstract the RBC as a normal- meta- game between three players representing the agent types, i.e., the consumers, firms, and government. We test whether a set of learned policies is an e-Nash equilibrium for the meta-game by evaluating whether or not they are approximate best responses. Recall that agents of the same type share policy weights (e.g., consumers), but use agent-specific inputs to their policy, To find an approximate best- response, we train each agent type separately for significant number of RL iterations, holding other agent types fixed, and see if this improves their reward by more than If so, the policies are not an equilibrium; if not, this suggests we found least a local equilibrium. In general, we find empirically that best-responses improve rewards much more at the start compared to the end of training. (See Table 1.) Throughout, we report results where the policies have found an approximate equilibrium at the end of training. Note that the meta-game over agent types is different from the game over individual agents. In particular, a meta-game best-response (where all agents of the same type can best-respond) may not be individual's best-response. A meta game best-response may feature cumpetitive-coordintt dynamics between agents of the same type, e.g., consumers, such that their collective behavior decreases the reward for their agent type. However, this may feature adverse
7

---

Under review as a conference paper at ICLR 2022
150K
Consumer
Consumer
145K
Government
12-
Arm
140K
Mean Rewards
130K
125K
115K
20
60
80
Consumer Rowards
Fized tax rate percentage/11, tac rate
Figure 4: In each plot, the agent types (consumer, firm, and government, resp.) refer to cases when only that agent type is training. Left: Best-Responses during Training, The stacked bar chart shows the improvement in the mean rewards of an agent type. For firms and governments, the mean rewards are measured in units of 10 and 10s, respectively. We compare the best response improvement in the middle and at the end of training. The improvement from best response is significant in the middle and much less the end, indicating that training is closer to equilibrium at the end. Middle: Outcomes under Best-Response. After training, typically neither consumer firm rewards change significantly under best-response analysis. This holds generally for the approximate equilibria reported in this work. Here, rewards are on an absolute scale. In the figure, we display only those which change by more than 1%. Right: Comparing Fixed and RL Government Policies. Mcan rewards for all agent types under fixed tax rates. Again, for firms and governments, the mean rewards are measured in units of 10a and 10², respectively. The mean rewards for the consumers increase with tax rates, whereas for the firms they decrease. An RL government can improve the mean reward for both types by appropriately adapting tax rates. RL policies increase the social welfare by almost 15% over the best baseline policy with fixed tax rate.
behaviors for some individual agents, e.g., there may be free-riders that benefit from the collective performance while not putting in effort themselves. As we consider settings with large number of agents (c.g.. 100 consumers), we limit our analysis to meta-game approximate best-responses.
Comparing with Baseline Government Policies. To show that our approach sound, we show that RL tax policies lead to improved social welfare (here, weighted sum of firm and consumer rewards, see appendix C) compared with several different manually defined fixed tax rate policies. As observed in Figure 5, RL policies generate social welfare ranging from 3000 4000, depending on the equilibrium (as noted above, multiple equilibria could be reached). Compared to that, the best social welfare achieved using fixed tax was 3160. Figure 4 shows the social welfare achieved under different various fixed tax rates. We note that social welfare improves by almost 15% for the best equilibrium under RL tax policy over the best equilibrium with fixed rates. This shows that the RL policy can adjust taxes across different rounds to improve average social welfare.
Exploring RBC Equilbria. Figure 5 visualizes and describes the learned RBC equilibria in open and closed RBC economies (with and without an export market). In both cases, we find multiple equilibria, although learning mostly converges to low-welfare equilibria in the closed economy. The open RBC model admits a wide spectrum of approximate equilibria with distinct outcomes (e.g.. consumer utility, hours worked, prices, and taxes), and trends that align with economic intuitions. To study the differences between equilibria, Figure shows a detailed comparison between two rollouts at two different equilibria. This reveals distinct equilibrium strategies, e.g., firms can profit by either focusing on consumers or the export market.
6 DISCUSSION
Daskalakis et al. (2009) showed that computing equilibria for general-sum games is hard in terms of computational complexity, framework can find all equilibria in sequential economic games. However, our best-response analysis suggests our framework does discover local equilibria. Hence, current framework is at least
for simple matrix games. There is theoretical guarantee that our
8

---

Under review as a conference paper at ICLR 2022
102
Hear Grvernment Reward
(Mean Firm Rowwrdd
Moa Offered Wage
23.0
Conzumption
0.30
120
12.5
0.23
38.0
6,20
7.5
0.35
0.30
20.
2.5
0.05
0.5
3.0
1:5
2.0
255
53
0-4
0.6
100
155
200
250
0,00
1200
1400
3000
1000
2000
Mean Consumer Reward
Meen Incore Rate (S)
Mean Hours
Mean Price
-22 5.06, pe0.45
10.3
5.25, pwo
Nean Government Rewera Reward
22 0.19 p.0.15
>2 0.73, pwo..o
Moan Roownd
Mean offireed Wage
9.0.
3600
85
03
80
5200
1205
3000
2.50
Moan Consumer Roward
Moan Income Tax Rate (N)
Mcan Hours Worked
Moan Price
Figure 5: Learned Equilbria in Open (Bottom) and Closed (Top) RBC Models. Outcomes convergence for the same experiments in Figure 4. Each point represents an approximate equilibrium, verified by an approximate best-response analysis. Points of the same color and shape correspond to the same run. Top: In the closed economy, training often converges to equilibria with low consumer reward and litrle production. particular, social welfare (government reward) does not increase with higher tax rates, average labor does not change with wages, and consumption is unchanged with price. An exception is an equilibrium with significantly higher social welfare, labor, and consumption. This suggests multiple equilibria do exist, but non-trivial equilibria are harder to learn in the closed economy. Bottom: We learn multiple distinct, non-trivial equilibria in an open economy. Blue lines show linear regressions to the data. Consumer and firm rewards positively correlated (r² 0.25), e.g., if consumers earn more, they can consume more, yielding higher profits. Higher prices decrease mean consumption (y 0.73), lower wages decrease mean hours worked (r2 0.19), and there is no strong signal that higher taxes improve social welfare (r² = 0.06).
Achast
Figure 6: Open Economy Rollouts, Two Different Equilibria. We show actions and states for two representative runs at two qualitatively different equilibria. Heatmaps for the same metric use same range of intensities, e.g., for prices the top and bottom row. We observe that firms have different strategies: some prices high and rely exporting goods (for example, firm 3); others set prices lower and also sell to consumers (for example, firm 0). Consumers respond sensibly, only consuming when prices are low and only working when wages are not O. The differences in initial firm endowments as well as evolution during the episode lead to different final capital levels.
an exploratory tool for finding qualitatively different equilibria, e.g., by varying initial conditions, environment parameters, conditioning sampling, and provides exciting avenues for future research.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Daron Acemoglu, Mikhail Golosov, and Alch Tsyvinski. Dynamic mirrlees taxation under political
economy constraints. The Review of Economic Studies, 773)):41-8811 2010.
An, Milind Tambe, and Arunesh Sinha. Stackelberg security games (ssg): Basics and application overview. Improving Homeland Security Decisions, pp. 485, 2017.
Manoj Atolia, Santanu Chatterjee, and Stephen J Turnovsky. How misleading is linearization? evaluating the dynamics of the neoclassical growth model. Journal of Economic Dynamics and Control, 34(9):1550-1577 2010.
Yu Bai, Chi Jin, Huan Wang, and Caiming Xiong. Sample-efficient learning of stackelberg equilibria in general-sum games. arXiv preprini urXiv:2102. 1494, 2021.
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. The Mechanics of n-Player Differentiable Games, arXiv: 05642 /Cs], February 2018. URL arXiv: 1802.05642.
http:///rxivee http://aa.//////..///
Eric Bonabeau. Agent-basede modeling: Methods and techniques for simulating human systems. Proceedings of the National Academy of Sciences, (9(pppl 3):7280-7287, 2002. ISSN 0027-8424. doi: 10.10733pnas.00200080 7280.
URL
Lena Mareen Boneva, R. Anton Braun, and Yuichiro Waki. Some unpleasant properties of loglin- earized solutions when the nominal rate is zero. Journal of Monetary Economics, 84:216-232, 2016. ISSN 0304-3932. doi: https://oi..rr//10 // //wp/sccennedrrett.coo httpt//cccccciecccceeeee wwtsssciiccceeeeeeeeeeeeeecceeee wateencee/ariiiii http:///aaaaaae////030333322223/
URL https:
Michael Bowling and Manuela Veloso. Multiagent learning using variable learning rate. Artificial Intelligence, 136(2):215-250, 2002.
Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418-4244 2018.
Mingli Chen, Andreas Joseph, Michael Kumhof, Xinlei Pan, Rui Shi, and Xuan Zhou. Deep reinforcement learning in a monetary model. arXiv preprint arXiv:2104.09366 2021.
Panayiotis Danassis, Aris MatssRRassi,ss, and Boi Faltings. Achieving diverse objectives with ai- driven prices in deep reinforcement learning multi-agent markets. arXiv preprint arXiv:2106.06060, 2021.
Constantinos Daskalakis, Paul W Goldberg, and Christos H Papadimitriou. The complexity of computing a nash equilibrium. SIAM Journal on Computing, 39(1):195-25) 25999 2009.
Jakob N. Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with Opponent-Leariing Awareness. arXiv:1709 1709.04366 [Cs], September 2017. URL http:// /ataaaaairgaaaaa////./aa//a/aaee abs www.aa///....... /1709.04326.
arXiv: 1709.04326.
Mikhail Golosov, Maxim Troshkin, and Aleh Tsyvinski. Optimal Dynamic Taxes. Working Paper 17642, National Bureau of Economic Research, December 2011. URL papers/ w1742..
Andrew G Haldane and Arthur Turrell. Drawing on different disciplines: macroeconomic agent- based models. Journal of Evolutionary Economics, 29(1):39 66, 2019.
Burkhard Heer and Alfred Maussner. Dynamic general equilibrium modeling: computational methods and applications. Springer Science & Business Media, 2009.
Edward Hill, Marco Bardoscia, and Arthur Turrell. Solving heterogeneous general equilibrium economic models with deep reinforcement learning. arXiv preprint arXiv:2103. /6977, 2021.
Junling Hu and Michael P Wellman. Nash lelaaning for general-sum stochastic games. Journal of machine learning research, 1(N9v):1039-1066) 2003.
10

---

Under review as a conference paper at ICLR 2022
Daniel Kahneman. Maps of bounded rationality: Psychology for behavioral economics, American economic review, 93(5):
2003.
Nitin Kamra,
Umang Gupta, Fei Fang, Yan Liu, and Milind Tambe. Policy learning for continuous space security games using neural networks. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Andreas Klöckner, Nicolas Pinto, Yunsup Lee, Bryan Catanzaro, Paul Ivanov, and Ahmed Fasih. Pycuda and pyopenel: scripting- based approach gpu run-time code generation. Parallel Computing, 38(3):157-174, 2012.
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https://
p.thuu..cmm/kottttiooo///ogooo. - pa2ccpp--ccktrr----- acktr-gai1,200
a2c
gaii, 2018.
Tian Lan, Sunil Srinivasa, Huan Wang, Caiming Xiong, Silvio Savarese, and Stephan Zheng. Warp- drive: Extremely fast end-to-end deep multi-agent reinforcement learning on a gpu, 2021.
Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent Reinforcement Learning in Sequential Social Dilemmas. arXiv: 3tt1::/702.03000 [Cs]. February 2017.
URL http: /ttp::///aaxvie http://wwwwiv....e/.o.//// /bt.s//...3333777..
arXiv: 1702.03037.
Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In William W. Cohen and Haym Hirsh (eds.), Machine Learning Proceedings 1994, pp. 157-163. Morgan Kaufmann, San Francisco (CA), January 1994. ISBN 978-1 978-1-55860-335- 6. doi:
URL http://ww..ceeneeiicccccoo.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-Agent Actor- Critic for Mixed Cooperative-Coompetiivee Environments. 02ti::7706.0222 [Cs], June 2017. arXiv: 1706.02275.
URL http:://aaa.w.oorr////////.....
Andreu Mas-Colele Michael D. Whinston, and Jerry R. Green. Microeconomic Theory. Oxford University Press, Oxford, New York, June 1995. ISBN 11880-19-50734--9
N. Nisan, T. Roughgarden, E. Tardos, and V.V. Vazirani. Algorithmic Game Theory. Cam- bridge University Press, 2007. ISBN 9780521872829, URL https: httt::::/booos..ooglee.oooo
Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs. Springer Publishing Company, Incorporated, 1st edition, 2016. ISBN 978-3-319-2899774
/ot opwnoo-eniii- five/, 2018.
OpenAI. Openai five. https: http//loogpppaa..co.m. wttp://coooopeiiiiiiiieee
Jean Pierre Danthine and John B. Donaldson. Methodological and empirical issues in real business cycle theory. European Economic Review, 37(1):1-35, 1993. ISSN 0014-2921. doi: https://doi.c org/
00010016/0044-22211199900666 article/pii//e 000/0001122222 9391 9390068T.
URL https: http///ww//tttco enced wwwt..oo//o/oooot scwc/e//eeee//////////o
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
Ekaterina Sinitskaya and Leigh Tesfatsion. Macroeconomies as constructively rational games. Journal of Economic Dynamics and Control, 61:152-182. 2015.
Frank Smets and Rafael Wouters. Shocks and frictions in us business cycles: bayesian dsge approach. American economic review, 97(3):5866606, 2007.
Sriram Srinivasan. Mare Lanctot, Vinicius Zambaldi, Julien Pérolat, Karl Tuyls, Rémi Munos, and Michael Bowling. Actor-critic policy optimization in partially observable multiagent environments. arXiv preprint arXiv: 18 1810,0902, 2018
11

---

Under review as a conference paper at ICLR 2022
Nancy L. Stokey, Robert E. Lucas, and Edward C. Prescott. Recursive Methods in Economic Dynamics.
Harvard University Press, 1989. ISBN 9780674750968. URL http:/ http://wwwwttto../....//
stable/j.ctvjaat775
Lawrence H. Summers et al. Some skeptical observations real business cycle theory. Harvard Institute of Economic Research, 1986.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, October 2018. ISBN 778-0-262-35270-- Google-Books-ID. uWVODwAAQBAL
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:/801.000 2018.
Kristal K Trejo, Julio B Clempner, and Alexander S Poznyak. Adapting strategies to dynamic environments in controllable stackelberg security games. 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 5484 5489. IEEE, 2016.
Alexander Trott, Sunil Srinivasa, Douwe van der Wal. Sebastien Haneuse, and Stephan Zheng. Building a foundation for data-driven, interpretable, and robust policy design using the economist. arXiv preprint arXiv:2 029000 2021.
Karl Tuyls, Julien Perolat, Marc Lanctot, Joel 7. Leibo, and Thore Graepel. A generalised method for empirical game theoretic analysis, 2018.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)350-354, 2019.
H. von Stackelberg, D. Bazin, R. Hill, and L. Urch. Market Structure and Equilibrium. Springer Berlin Heidelberg, 2010. ISBN 9783642125850. URL https: wtt.:ook...ogge...o...oo books Pid= MaaDAAACAAJ.
Yufei Wang, Zheyuan Ryan Shi, Lantao Yu, Yi Wu, Rohit Singh, Lucas Joppa, and Fei Fang. Deep reinforcement learning for green security games with real-time information. Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 1401-140S 2019.
Michael P Wellman. Methods for empirical gaee-the analysis. 2006.
Ronald J Williams. Simple statistical follownn-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229 250-2222566, 1992.
Kaiqing Zhang. Zhuoran Yang, and Tamer Başar. Policy optimizationn provably converges to nash equilibria in zero-sum linear quadratic games. arXiv preprint D.XXv::9906000299, 2019.
Stephan Zheng, Alexander Trott, Sunil Srinivasa, Nikhil Naik, Melvin Gruesbeck, David C Parkes, and Richard Socher. The ai economist: Improving equality and productivity with ai-driven tax policies. arXiv preprint a3X1v:2004.13332, 2020.
A RELATED WORK
Computational Approaches to Find Equilibria. Leibo et al. (2017) studied meta-game equilbria in sequential social dilemmas, showing that RL policies can categorized into high-level strategy types akin cooperation or defection in the classic social dilemma. Counterfactual regret minimiza- tion has yielded superhuman poker bots (Brown & Sandholm, 2018), an imperfect-informale however, it can be computationally inefficient. Empirical thme-tthorettc analysis studies equilibria through agent simulations (Wellman, 2006), but limited to games with interchangable agents with identical affordances (Tuyls al., 2018) and do not easily scale settings with heterogeneous agents.
game;
Higher-order gradients (Foerster et al., 2017) and first-order gradient adjustments (Balduzzi et al., 2018) have sought promote convergence to non-trivial equilibria. However, these methods make strong assumptions on what agents know, e.g., that agents can see the policy weights of other agents and/or know the full reward function, and may deform the original equilibria.
12

---

Under review as a conference paper at ICLR 2022
RL for Economic Simulation. Agent-based modeling (ABM) studies economic simulations with individual agents. Early work mostly used rule-based agents or RL in finite state ABMs to study emergent phenomena (Bonabeau, 2002). A recent economics survey of ABMs (Haldane & Turrell, 2019) outlined the potential for RI. agents, following its success in MI..
Several recent works have explored this idea. The AI Economist used RL to design optimal taxes to improve social welfare in high-dimensional simulations, where both agents and governments use RL policies (Zheng et al., 2020). RL also yields interpretable economic and public health policies in pandemic simulations (Trott et al., 2021). Danassis et al. (2021) takes a similar approach: they explore multi-agent environment with harvesters participating in a common fishery, and a centralized price setter, and demonstrate that the price setter can outperform prices set by market equilibrium.
Other contemporary work using deep RL includes Chen al. (2021), which studies monetary policy with single representative household RL. agent, and Hill et al. (2021), which learns the value function of consumers in RBC models. These works are more limited, compared to ours. First, they use RL for one agent type, while the other agents (e.g., firms) use simple and fixed policies. Second, they assume markets always clear at each time step, i.e., prices are set to ensure supply and demand are balanced. However, this is an unrealistic assumption and causes slow simulations, requiring solving a nonlinear optimization problem at each timestep. Perhaps most similar to our work is Sinitskaya & Tesfatsion (2015), which deals with an economy with non-heterogeneous consumer-workers and price- and wage-setting firms (but no government). They to some extent enforce market clearing, by matching supply and demand for labor and goods in double auction, although they do not enforce via constraints that there should be enough labor to produce demanded goods. Small numbers of consumers and firms are allowed learn using tabular Q-learning. Even their setting, they also observe that learning dynamics can collapse to trivial equilibria with no production or consumption.
In our work, we do not enforce market clearing constraints all, simply rationing goods there are not enough. Moreover, consumers have different parameters, and each firm has a different production function and produces distinct good.
RL and Stackelberg Games. In a Nash equilibrium, every player is playing best response against every other player, so that no player can improve their outcome by unilaterally deviating. By contrast, a Stackelberg game, one player is leader who has the opportunity to move first; followers observe that move and choose their best response. As discussed in Zheng et al. (2020), there are aspects of Stackelberg games in economic simulations: a government who sets taxes or a firm setting wages and prices must make choice anticipating a response by other agents. One popular setting for Stackelberg equilibria is in security games (An et al., 2017). In such games, the leader represents a defender who chooses where to deploy patrols; the attacker wishes evade the patrols and knows the strategy chosen by the leader. Several works have studied the use of RL techniques in such settings (Wang et al., 2019; Trejo al., 2016; Kamra et al., 2018; Bai al., 2021).
Theoretical guarantees of convergence Guarantees of convergence for model-free policy opti- mization are difficult to come by. Some theorems only apply to tabular policies (as in Srinivasan al. (2018), and where convergence to Nash not even guaranteed due bound). Other cases also apply only to very limited families of games, as in Zhang et al. (2019), which deals only with zero-sum linear-quadratic games. We deal with a general sum game with large numbers of agents and with neural network policies all three of these properties mean that theoretical convergence guarantees are beyond the reach of theory.
non-sublinear regret
B REPRODUCIBILITY STATEMENT
We discuss specific training details for all experiments such as hyperparameters in the Appendix.
Upon acceptance, all code for this project will be made open source and publicly available for reproducibility purposes and further research. Upon request, we will share our code with reviewers and area chairs for review.
13

---

Under review as a conference paper at ICLR 2022
C ETHICS STATEMENT
Our work proposes a framework to model economies using Multi-Agent Reinforcement Learning and thus may be used to draw implications about the real world. Our findings and used simulations are purely for research purposes and should not be used make decisions in real-world systems. Furthermore, our framework should not be used to explore methods to increase discrimination or unfairness in real-world systems.
Assumptions, limitations, and ethical implications of using ML for economics. All choices in the economic simulation model, RL algorithms, reward functions, etc, play an important but difficulttto-undesstat role in equilibria selection and policy design. As in all ML applications, there are assumptions and limitations in the methodology. This has ethical implications for their use in future policy design applications.
Mitigation strategies and interdisciplinary research. Economic simulation enables studying wide range of cconomic incentives and their consequences, including models of stakeholder capitalism. However, the version of the simulation used in this work is not an actual tool that should be used for policy making.
Many design choices influence the eventual policy recommendations. For example, the designer is free to set the social welfare objective that the government optimizes for. As such, it is crucial that these choices are debated and made in a socially acceptable fashion by all stakeholders, and made transparent and accessible to all.
More generally, mitigate ethical risk, further mitigation strategies may include performing what-if analysis over worst-outcomes, opening research results to domain experts (social scientists, ethics experts, etc), and open-sourcing the research results, amongst others. all, the design and use of ML for policy recommendations will require robust, multilateral discussion and careful consideration of ethical risk, potential harm, and which trade-offs are being made.
We now detail some assumptions, limitations, and potential ethical risk among different dimensions of using ML for economics. We stress that there can be more (unknown) aspects that we do not address here. As such, we see this discussion as a starting point of discussion for the ML and economics community.
Economic simulation and data. While the current version of the economic simulation provides only limited representation of the real world, we recognize that future, large-scale iterations can still contain biases and unrealistic assumptions. Furthermore, non-representative simulation environments may result in biased policy recommendations. For instance, the under-representttion of communities and segments of the work-force in training data might lead to bias in simulations that build on those and lead to biased AI policies. As such, collecting more representative data is a key challenge for future research in using ML for economic policy recommendations.
Our RBC model is a stylized model of real economies, RBC models are commonly used class of economic models (see e.g. Smets & Wouters (2007)). However, as any model, it contains assumptions and stylizations. Future simulations may miss (un)known features that pertain, e.g., to equity and equality in the economy. Therefore, using simulations that are not representative or well-calibrated, can exacerbate or create new socio-economic issues.
We list few salient features and assumptions below, although we cannot exhaustively enumerate all features that may be relevant in future research.
Our RBC model features consumers that differ in skill and perform different amounts of work. However, we do not model more fine-grained distributional features, such as educational attainment, wealth, inheritance, geography, or others.
Similarly, firms produce a single good only and can invest and pay wages. Any worker work for any firm. We do not model hiring practices, the geographic location of firms, non mon-mone incentives or benefits (c.g.. health insurance). To accurately model inequity in the real world, including such features may be necessary.
14

---

Under review as a conference paper at ICLR 2022
On the government and societal level, model tax policies and simple redistribution of tax revenue. We do not model targeted redistribution, tax credits, application-spccfiie subsidies (e.g., education support). We do not model trading, inflation, debt, and other macrocconomic features that may impact social groups disparately.
Our RBC is more general than commonly used models: we do not enforce market clearing, for instance. Market clearing is an unrealistic assumption that supply always meets demand. Economic theory uses such constraints to make analysis tractable. In contrast, our learning approach is flexible and does not require such simplifying assumptions. We also assume that agents can observe the wages offered by all the firms. However, in the real world not all agents have equal access to information and this is a feature that can be studied by future research. As such, we view the flcxibility of our learning approach as a positive, in that our framework may allow for studying more representative models.
Choice of economic incentives and rewards. Agents optimize their behavior given economic incentives, as modeled by their reward function. As such, future economic Al policies should clearly describe for which reward function they were optimized. Furthermore, more research is needed to understand how the choice of reward function influences the resulting policies, and how social and ethical values can be transparently encoded in reward functions. is also an open question which ethical/social norms and values can or cannot be quantified, and how to encode trade-off's between conflicting values.
For example, the planner optimizes its policy maximize "social welfare", a standard economics concept. However, the definition of social welfare heavily influences the resulting policy and social outcomes. For example, Zheng et al. (2020) used equality times productivity as their objective and showed the resulting AI income taxes can improve equality over classic tax models. Standard economic works often use the utilitarian objective (sum of all agent rewards). An alternative the Rawlsian objective (social welfare is the reward of the lowest-income agent). We emphasize the choice of social welfare is flexible and a choice made by the designer(s) and users of the framework.
Another key example is the discount factor used to weight rewards over time. Whether to emphasize short-term vs long-term rewards is a social choice that has ethical implications. For example, firms may emphasize short-term profits over long-term health issues, which may disparately impact different social groups
Choice of agent model. The behavior of agents is determined by the policy model, e.g.. the neural networks used in our work. Neural networks are universal function approximators, given enough width (or depth) in their layers. However, in practice, neural networks may still encode structural biases and only parameterize a particular subspace of all theoretically possible policy models. For our networks, a particular concern might be architectural constraints: our policy networks are not recurrent (so only consider the current state) and sometimes don't 't allow correlated actions. With enough parameters these networks are still capable of representing a wide range of policies, but these architectural constraints represent implicit priors which conceivably might not reflect human decision-making As such, more research needed on what the limits are of neural networks terms of emulating human behaviors, and to what extent more and diverse datasets can help alleviate such concerns.
Choice of algorithms and learning strategies. The RBC model is an economic "game" that has multiple equilibria. It is not well understood theoretically to which equilibria given RL algorithm converges. Indeed, previous work has studied MARL beyond independent learners, including Nash-Q (Hu & Wellman, 2003), WoLF (Bowling & Veloso, 2002), and MADDPG (Lowe et al., 2017). This extends to our use of structured curricula, reward shaping, and other forms of multi-agent learning algorithms or strategies. These methodological choices can all impact the equilibria one finds (or doesn't find) using ML.
This is important because different equilibria can have different levels of social welfare and granular social outcomes (e.g., equality, type of work performed, unemployment level). From an ethical point of view, it is therefore possible that certain choices of algorithms, etc, may bias policy recommenda- tions and simulation outcomes to socially or ethically undesirable situations. For instance, certain social groups in the simulation may be disparately impacted by policy recommendations. Therefore,
15

---

Under review as a conference paper at ICLR 2022
it is important for future research to analyze how different RL algorithms may selectively converge to certain equilibria, and how one might enumerate all possible equilibria. This is still a significant theoretical and empirical challenge.
Defining and justifying the objectives for the social planner and other methodological choices is a complex discussion, and requires more in-depth understanding of the functioning of MI that is beyond the scope of this work. This requires multilateral, interdisciplinary discussion on, for example, what the preferred social choice is with respect to the definition of social welfare and constraints.
Choice of hyperparameters. RL algorithms may converge to different solutions depending on the chosen hyperparameters, e.g., learning rate, entropy regularization, or discount factor. For instance, level of entropy regularization regulates the exploration-exploiattio trade-off in actor-critic methods, form of on-policy RL as used in work. It is known that actor-critic methods may get stuck in suboptimal local maxima. This issue may be exacerbated in the multi-agent setting, where there multiple equilibria, and it is unknown how algorithms converge towards different equilibria. As such, is possible that certain choices of hyperparameter can encode structural biases towards certain outcomes in the simulation. These potential limitations are an area for future research.
Robustness Deep RL. A key question how robust learned policies are to perturbations in the simulation (parameters). This has ethical implications: policies that do well in simulation, may not do well the real world if, e.g., income distributions differ between sim and real. As such, simulations that not representative (enough) may lead policy recommendations that disadvantage underrepresented social groups in the real world. More generally, it is well-known that deep learning models and RL policies can be very brittle and may not generalize well unseen environments. As such, more robustness analysis should be done on any policy recommendation that based deep RL and related methods.
Explainability and Simplicity of AI policies. Even though Al policies may be effective, they may use intricate, unexplainable patterns in their input data to achieve high performance. Moreover, their behavior may vary wildly between different inputs. As a hypothetical example, an RL agent may make significantly different tax rate recommendations for people with slightly different income education levels. Such behaviors can disproportionally affect underprivileged social groups, and have unintended short/long-tee is an open question on what level/amount of data, or specific policy constraints, could mitigate such potential risk and harm. We also note that if one wanted to restrict the class of policies ones that are sufficiently explainable, the same model-free policy optimization scheme could still be applied. Indeed, this is a big potential advantage of the RL approach.
economic consequences, especially if models
well-calibrated. It
D MORE DETAILS ON THE RBC
Settings.
Table 2 lists all the simulation parameters. Table lists all the training hyperparameters.
Parameter
Symbol
Values
Labor disutility
0
0.01
Pareto quantile function scale parameter
4.0
Initial firm endowment
B
2200000
Export market minimum price
500
Export market maximum quantity
100
Production function values
0.2 to 0.8, increments of 0.2
Initial capital
K
5000 or 10000
First round wages
0
First round prices
p 1000
Initial inventory
y 0
Table 2: Simulation Parameters.
More implementation details follow:
16

---

Under review as a conference paper at ICLR 2022
Parameter
Values
Learning Rate
0.001
Learning Rate (Government)
0.0005
Optimizer
Adam
Initial entropy
0.5
Minimum entropy annealing coefficient
0.1
Entropy annealing decay rate
10000
Batch Size
128
Max gradient norm
2.0
PPO clipping parameter
0.1 or 0.2
PPO updates
or 4
Consumer reward scaling factor
Firm reward scaling factor Government reward scaling factor
30000
1000
Table 3: Training Hyperparameters.
For consumers, consumption choices range from 0 to 10 units for each good and work choices from 0 to 1040 hours in increments of 260.
Consumers have CRRA utility function with parameter 0.1, and disutility of work of 0.01.
For firms, price choices range from 0 2500 in units of 500; wage choices from 0 to 44 in units of 11.
The 10 firms split into two groups, receiving either 5000 or 10000 units of capital. Within these groups, firms receive a production exponent ranging from 0.2 to 0.8 in increments of 0.2. Thus each firm has different production "lechnology".
Firms invest 10% of their available budget (if positive) in each round to increase their capital stock.
Government taxation choices range from 0 to 100% in units of 20%, for both income tax and corporate tax rates.
The government can either value only consumers when calculating its welfare ("consumer- only") or value welfare both consumers and firms ("total"), with firm welfare down- weighted by factor of 0.0025 (to be commensurate with consumers).
We set the minimum price at which firms are willing to export to be either 500 or 1000, and the quota for each firm's 's good to variety of values: 10, 50, 100, 1000.
For consumers, consumption choices range from 0 to units for each good and work choices from 0 to 1040 hours in increments of 260.
The environment state s consists formally all agent states and general world state. Each agent observes observe their own information and the global state:
Observation.
8 Sglobal = (t. (yr,); {pt.ch (wrahh {o..),)
(8)
Here yi, is the available supply of good Pi, c is the price, wi,r is the wage. The extra information og, includes whether good was overdemanded at the previous timestep and tax information.
In addition, consumer-worker observe private information about their own state: (Bi,t 0) A firm i also observes its private information: (B ki,t, (O....,I, 1,..., 0), .a), including a one-hot vector encoding its identity and its production function shape parameter a. The government only sees the global state.
Heterogeneity We observe that although there is fixed policy per agent type, agents exhibit heterogeneity through agent-specific input features, such as the disutility of work or production parameters.
17

---

Under review as a conference paper at ICLR 2022
E TRAINING DETAILS
For all experiments, we clip gradient & norms to a maximum of 2.0, use a Huber loss for the value function head, and center and standardize estimated advantages. For PPO, use the clipping based surrogate loss approach rather than a KL penalty, closely following the baseline implementation of Kostrikov (2018). We show pseudocode for single simulation and training step in Algorithm below.
Algorithm 1 A single training step at time step
Tes Ue consumer policy and value network
Tf: + firm policy and value network, prices and wages masked according to & Tgs Ug f- masked government policy and value network, tax rates masked according to
((t): disutility of work parameter, annealed over training steps
wt)):: entropy parameter, annealed over training steps according to max(exp( docay rate ), 0.1)
Oc; Te: fe..8f.,1
89, ag. +- EnvironmentSimmlatefr. Tf, Tg. 0())
Tes Ue -PPOUpdate(T Uos Se; Oes Te, (())
if L > (stant firm then
Ty. + P vy, a,, yy wit start frm)
end if'
if t 1strt government then
gg. Ug < PPOUpdate(TT. Ugg 8g. Ogg g: w(t ttan government))
end
F IMPLEMENTATION DETAILS
Budget Constraints. We implement budget constraints on the consumers by proportionally scaling down the resulting consumption of all goods to within consumer's s budget. Thus, the consumer actions really represent attempted consumption if the budget is small or stock is limited, the actual consumption enjoyed by the consumer may lower. Firm budgets are allowed to go negative (horrowing money). However, because the firm's goal is to maximize profit, they are incentivized to take actions will will be profitable, increasing their budget.
Scaling of Observables. The scales of rewards and state variables vary widely in our simulation, even within time steps in a single episode. If the scales of loss functions or input features are very large small, learning becomes difficult. We directly scale rewards and some state features by constant factors. For certain state features which have very large ranges (item stocks and budgets) we encode each digit of the input as separate dimension of the state vector.
G GPU IMPLEMENTATION
We followed the WarpDrive framework (Lan et al., 2021) to simulate our DGE and do RL on single GPU. We implemented the environment dynamics of our model as a CUDA kernel in order to increase the speed at which we can collect samples. We assigned one thread per agent (consumer, firm, or government); the threads communicate and share data using block-level shared memory. Multiple environment replicas run in multiple blocks, allowing us to reduce variance by training on large mini-batches of rollout data. We use PyCUDA (Klöckner et al., 2012) to manage CUDA memory and compile kernels. The policy network weights and rollout data (states, actions, and rewards) are stored in Py Torch tensors: the CUDA kernel reads and modifies these tensors using pointers to the GPU memory, thereby working with a single source of data and avoiding slow data copying.
18


Reviews:
[{"review_id": "bPtidYciUM", "reviewer": "Reviewer_AWjh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As pointed out by the authors, there are relatively few works on reinforcement learning for economics. This direction seems promising. However, I have a few questions:\n\n1. In the abstract and the introduction, could you please clarify what you mean by finding optimal agent policies “explicitly” versus “implicitly” (top of page 2)?\n\n2. In section 3, several ideas are described but there is no clear description of the algorithm. Since the problem considered here is rather complex, I recommend adding a description (e.g., pseudo-code) of the MARL method. It would be helpful for the reader to see how the ingredients described in this section are combined.\n\n3. In section 4, at the bottom of page 6, how do you check the local equilibrium property for the leader? The text seems to indicate that you fix all the followers and train the leader alone. However, I thought that the leader's reward depends the followers' reactions. Are the followers' policies functions of the leader's policy? Here again, I recommend adding a clear description of the quantities that are computed for the plots.\n\n4. Figure 4, left plot: Even at the end, there is still a significant improvement for the firms (roughly equal to the improvement at the middle of the training). How can you explain that?", "summary_of_the_paper": "This paper proposes a multi-agent deep reinforcement learning (DRL) method to compute general equilibria in economics. The main contribution is hence algorithmic. The method uses a combination of DRL, structured learning curricula, and suitable annealing of action space and of some penalty coefficients (which are useful to make some problems easier to solve, but distort the computed solution). The authors apply their method to an example of real-business-cycle model, which involve three types of agents: firms, consumers and a government. The problem is of Stackelberg (or leader-follower) type, with the government acting as a leader. Computing equilibria for such problems is generally very challenging.\n", "summary_of_the_review": "The application of multi-agent reinforcement learning for economic problems seems very promising, particularly given the high complexity of these problems. The main contribution is algorithmic, but without a precise description of the algorithm, I find it hard to really understand the method and to assess the impact this work could have.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R6b2jH3Ol_S", "reviewer": "Reviewer_pWNT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The most interesting point in this paper is that the proposed scheme breaks the curse of many agents. In the numerical example, the total number of agents is more than 100. As in economics models, a single agent usually represents a large population of homogeneous agents, 100 is a sufficiently large number from an application point of view. The success of the deep RL algorithm studied in this paper highly relies on WarpDrive, a recently developed framework that supports running both simulation and RL on a GPU.\n\nYet, as far as I'm concerned, running the algorithm successfully is just a first step for studying this fundamental problem. Although the authors reported their empirical results, they failed to provide enough theoretical guarantees for supporting the results. \n\n(1) Can the proposed algorithm converge to the ($\\epsilon$-)Markov equilibrium almost surely? If so, can you provide sufficient conditions for the convergence?\n\nIn this paper, the convergence to the equilibrium is checked empirically (p6-best responses and local equilibrium analysis). Yet, a specific case could not provide the convergence guarantee for the general setting. This specific case is relatively special: the 100 consumer-works and the 10 firms are homogeneous, respectively. Therefore, it is similar to (but not the same as) a potential Markov game, which is not significantly different from a single-agent MDP. In the study of dynamic general equilibrium, the heterogeneity of agents has received more and more attention. The convergence of the algorithm in the heterogeneous setting (e.g., the 100 consumer-works are different from one to another) could be more difficult. \n\n(2) Can the proposed algorithm learns a real Stackelberg strategy?\n\nIn the proposed model, the government has a first-mover advantage as in a Stackelberg game. In the optimization literature, it is well known that finding the Stackelberg equilibrium is an NP-hard problem. When both the leader and the followers become MDP agents, this problem would definitely become more complicated. Yet, in this paper, no theoretical or empirical results could support that a (locally) Markov Stackelberg equilibrium is found. The only numerical result for this issue is provided in p7-comparing with baseline government policies: it is reported that the RL government policy improves the social welfare by 15% compared with the case with fixed tax rates. However, it doesn't mean that it is really a Stackelberg strategy unless sufficient results could support that the government has fully realized its first-mover advantage.\n\nMeanwhile, the algorithm itself is not described clearly in this paper. For example, the trick for running both simulation and RL on a GPU is from WarpDrive, yet the details of the WarpDrive framework are not even introduced in this paper. The structured learning curricula and the action space annealing procedure, as another example, are introduced using textual description completely without providing any equations and technical details. ", "summary_of_the_paper": "This paper proposed a deep reinforcement learning framework for finding dynamic general equilibrium, which is one of the most fundamental problems in economics. As the dynamic general equilibrium is a special case of the Markov game, this problem has also been recognized as a significant topic in machine learning. The proposed scheme is tested in a real-business-cycle model with 100 worker-consumers, 10 firms, and a government, the scale of which is much larger compared with the numerical examples in most related works.", "summary_of_the_review": "The paper studied a fundamental problem and tested an efficient algorithm for solving it. However, I do not think the paper can be accepted in the present form because (1) the success of the algorithm highly relies on a trick proposed in another paper; (2) besides running the algorithm empirically, no theoretical analysis of the proposed scheme is provided in the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DDUhZrklnEa", "reviewer": "Reviewer_JN4A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Using a multi-agent reinforcement learning algorithm to solve the complicated game considered in the paper is natural and reasonable. The structured learning curricula runs in multiple stages and could potentially ease the problem of instability in the training process. Also, it is interesting to see that the GPU-only framework is able to significantly accelarate the training process. The experiment results look convincing and interesting. \n\nThere are a number of notations that are used before they are defined (for example, p_i, J, \\tau), making the paper hard to follow. I have to read back and forth, and keep guessing the meaning of these symbols. The readability can be significantly improved.", "summary_of_the_paper": "This paper uses a multi-agent reinforcement learning algorithm to simulate an economic environment and solve the general equilibrium of the induced game. The authors propose to use a structured learning curriculum that runs only on GPUs. The authors conduct experiments to show that their algorithm converges fast and that the solution represents an epsilon Nash equilibrium.", "summary_of_the_review": "In general, I find the paper interesting. But the readability can still be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1uNQ0ghPci9", "reviewer": "Reviewer_yaWE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Review**\n\nWIth recent advancements in Deep MARL, the door has opened to allow it to be brought out and applied to real world scenarios. In this submission, the authors look at economics based on an Real-Business-Cycles (RBC) model. I find this to be an exciting direction for research as it promises to help inform and improve economic policy design. That being said, the RBC model presented here is not motivated nor verified (why should it capture the important economic mechanisms of interest?). However, this may be because I am not adequately familiar with this literature (as I would expect the same of the average ICLR reader). The core contribution of this work is a reward shaping schedule to prevent discovery of \"uninteresting\" equilibria. Very little is said about the implications of the different shaping choices and there are no ablations or analysis of their component effects. As the conclusions, and thus take-aways for economic governance, are critically impacted by the bias imposed from this reward shaping I would have preferred to have it treated more rigoursly. Finally, I found the author's use of empirical game theoretic analysis to be essentially absent despite it being labelled as underpinning the results. In total, I am excited to see interest in this area; however, I do not think this particular unit of research is ready for publication at this time.\n\n**Major Comments**\n\n- Reward shaping is used in this work to bias equilibrium selection towards \"non-trivial\" solutions. Why are these non-trivial solutions not worthy of discussion if they are afforded by the mechanism? How is an equilibrium identified as trivial, and which qualities of solutions where the authors biasing against through their reward shaping?\n- The ICLR community is not economics focused, and as a result it would be beneficial to include additional motivations and explanations when bringing in outside ideas. In particular, a large portion of this paper describes the implementation of a particular RBC model. It would be helpful to either punt a justification of its design to related work that uses the same mechanism. Or, if this is a novel mechanism, compare and contrast it with a pre-existing model include motivation and justification for deviations. \n- As a cross-disciplinary work, I would like to see additional connections back to what is known about RBCs from econonmics. Are there known equilibria for this model? How do the results compare with economics solutions? This would allow an appreciation of the results for non-experts.\n- Empirical game theoretic analysis is a method for solving games by constructing meta-games covering restricted strategy sets of the underlying-game, and solving the meta-game as an approximation of the underlying-game. In this work, convergence is measured by calculating the regret against the opponent's policy by freezing it and using Deep MARL to learn an approximate best-response. If all player's have no regret, then an e-NE is found. While this is all technically true, the quanlity of the analysis is weak, and as a result its likely that e is extremely large. Notably, the meta-game being analyzed in this setting is an incomplete 2x2 (abstracting to 2 players) NFG. The top-left cell being the payoffs being the pre-frozen joint-strategy performance, and the off-diagonal being the the approximate best-response performance against fixed policies. Could the authors provide evidence why such as imple empirical game is sufficent to ensure that e is small, and thus the meta-game's solution is at all relevant to the underlying game?  \n- The ethics statement does not seriously consider the ramifications of this work. Please discuss the biases and assumptions baked into this work and how this may lead to downstream issues instead of making blanket staements \"should not be used to make deicsions in real-world systems\" --- if this statement is true, then what's the point of this work?  It is important to think of the unknown-knowns that readers or future practitioners may overlook in the future. \n- Sec 5, Par 1, \"Empirical game-theoretic analysis .... with heterogeneous agents.\" This is not true. A meta-game is a choice of the practioner and through that choice they add restriction such as role-based meta-games often to gain performance benefits. EGTA is not restricted in analysis to any class of games, and can be analyzed through the lens of any game -- where certain choices, ala NFG, scale poorly (although this is a limitation of game theory that is simply inherited by empirical game theory).\n\n**Minor Comments**\n\n- Consumer reward has very large error bars in Fig 3, is there a particular explanation?\n- The authors note that their method finds only low-welfare equilibria. What does the discovery of these equilibria in RBC suggest to policy-makers? Do the authors suspect this is an artifical resultant of reward shaping?\n\n", "summary_of_the_paper": "This work investigates the use of Deep MARL in order to study economies. They propose and implement an RBC (economic market model) and propose a reward shaping schedule to bias agents into learning non-degenerate joint-strategies. In their setting, low-welfare equilria and provide a brief quantitative and qualitative analysis of discovered equilriba in open- and closed-economies. ", "summary_of_the_review": "I recommend rejecting this paper in a large part because (a) the main contributions (RBC model/reward shaping) have unclear/unspecified novelty, (b) weaknesses in the analysis described in the main review, and (c) need for contextualizing this work in both the MARL and economics literature (help the MARL community understand that this work is important!). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a deep multi-agent RL framework to compute equilibria in a economics problem. Several reviewers raised issues with the presentation, as well as issues with evaluating the impact of the work, partly because the novelty of the approach is made insufficiently clear. While the authors have resolved some of the confusions arising from the presentation in their rebuttal, resulting in 2 out of the 4 reviewers to increase their score, the concerns regarding novelty mostly remain. For these reasons, I don’t think this work is ready for publication at ICLR at the moment and recommend rejection.
Decision: Reject